2024-02-19T18:38:13.874 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-19T18:38:13.874 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566627 --verbose --name pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi --owner scheduled_pdonnell@teuthology --description fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 0a 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566627/orig.config.yaml 2024-02-19T18:38:13.923 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T18:38:14.008 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566627 branch: wip-batrick-testing-20240215.160715 description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 0a 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: pdonnell@redhat.com first_in_suite: false job_id: '7566627' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/v18.2.1 ' name: pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240215.160715 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: f78a58c0ffd401d1493058a1022c35f011d65275 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: f78a58c0ffd401d1493058a1022c35f011d65275 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: i64440 sha1: 1ce7ea8877c709fa862eac608550374bd96e166e owner: scheduled_pdonnell@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 2975 sha1: f78a58c0ffd401d1493058a1022c35f011d65275 sleep_before_teardown: 0 suite: fs:upgrade:mds_upgrade_sequence suite_branch: i64440 suite_path: /home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: 1ce7ea8877c709fa862eac608550374bd96e166e targets: smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPl8kXpGXbVfdtRkjLq6EX56iauEUR6aqba3MpgdRhe6RRcJUzXa4M6z8LW9rOsHFyxNvbNWlz330XuCkDwraMw= smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN3HqTMVQLehp+LPhy0ilDziUjDNwqBIKb3kelgvF3ztYyHGtT/3snZ/CGkU1mJg/z7RZyvJQQhT073dhTF6rKQ= tasks: - install: exclude_packages: - ceph-volume tag: v18.2.1 - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.io/ceph/ceph:v18.2.1 roleless: true - print: '**** done end installing v18.2.1 cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - full_sequential_finally: - sleep: duration: 3600 - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /cephfs/home/pdonnell/src/github.com_batrick_ceph_i64440/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-19_18:28:45 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-19T18:38:14.008 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa; will attempt to use it 2024-02-19T18:38:14.009 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks 2024-02-19T18:38:14.009 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-19T18:38:14.011 INFO:teuthology.task.internal:Checking packages... 2024-02-19T18:38:14.033 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'f78a58c0ffd401d1493058a1022c35f011d65275' 2024-02-19T18:38:14.033 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T18:38:14.033 INFO:teuthology.packaging:ref: None 2024-02-19T18:38:14.034 INFO:teuthology.packaging:tag: None 2024-02-19T18:38:14.034 INFO:teuthology.packaging:branch: wip-batrick-testing-20240215.160715 2024-02-19T18:38:14.034 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:38:14.034 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240215.160715 2024-02-19T18:38:14.175 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1450.gf78a58c0 2024-02-19T18:38:14.177 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-19T18:38:14.184 INFO:teuthology.task.internal:no buildpackages task found 2024-02-19T18:38:14.184 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-19T18:38:14.191 INFO:teuthology.task.internal:Saving configuration 2024-02-19T18:38:14.210 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-19T18:38:14.216 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T18:38:14.241 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566627', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 18:29:52.269874', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPl8kXpGXbVfdtRkjLq6EX56iauEUR6aqba3MpgdRhe6RRcJUzXa4M6z8LW9rOsHFyxNvbNWlz330XuCkDwraMw='} 2024-02-19T18:38:14.264 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566627', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 18:29:52.271432', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN3HqTMVQLehp+LPhy0ilDziUjDNwqBIKb3kelgvF3ztYyHGtT/3snZ/CGkU1mJg/z7RZyvJQQhT073dhTF6rKQ='} 2024-02-19T18:38:14.264 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-19T18:38:14.270 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-19T18:38:14.270 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-19T18:38:14.270 INFO:teuthology.run_tasks:Running task console_log... 2024-02-19T18:38:14.342 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f28c984c0d0>, signals=[15]) 2024-02-19T18:38:14.342 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-19T18:38:14.348 INFO:teuthology.task.internal:Opening connections... 2024-02-19T18:38:14.348 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-02-19T18:38:14.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:38:14.438 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-02-19T18:38:14.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:38:14.525 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-19T18:38:14.540 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-02-19T18:38:14.557 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-02-19T18:38:14.558 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-02-19T18:38:14.614 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-02-19T18:38:14.614 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-02-19T18:38:14.614 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-02-19T18:38:14.615 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-02-19T18:38:14.615 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-02-19T18:38:14.615 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-02-19T18:38:14.615 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-19T18:38:14.615 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-02-19T18:38:14.615 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-02-19T18:38:14.615 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-19T18:38:14.615 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-02-19T18:38:14.615 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T18:38:14.615 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-19T18:38:14.615 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T18:38:14.616 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-02-19T18:38:14.638 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-02-19T18:38:14.655 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-02-19T18:38:14.655 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-02-19T18:38:14.712 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream" 2024-02-19T18:38:14.712 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="9" 2024-02-19T18:38:14.712 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos" 2024-02-19T18:38:14.713 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora" 2024-02-19T18:38:14.713 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="9" 2024-02-19T18:38:14.713 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el9" 2024-02-19T18:38:14.713 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-19T18:38:14.713 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31" 2024-02-19T18:38:14.713 INFO:teuthology.orchestra.run.smithi174.stdout:LOGO="fedora-logo-icon" 2024-02-19T18:38:14.713 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-19T18:38:14.713 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/" 2024-02-19T18:38:14.713 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T18:38:14.713 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-19T18:38:14.713 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T18:38:14.713 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-02-19T18:38:14.737 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-19T18:38:14.747 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-19T18:38:14.752 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-19T18:38:14.752 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-02-19T18:38:14.754 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-02-19T18:38:14.770 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-19T18:38:14.776 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-19T18:38:14.776 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-02-19T18:38:14.811 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-02-19T18:38:14.917 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-19T18:38:14.970 INFO:teuthology.run_tasks:Running task kernel... 2024-02-19T18:38:14.984 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-19T18:38:14.984 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-19T18:38:14.984 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T18:38:14.985 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T18:38:15.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T18:38:15.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T18:38:15.003 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-02-19T18:38:15.003 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-02-19T18:38:15.061 INFO:teuthology.orchestra.run.smithi174.stdout:5.14.0-361.el9.x86_64 2024-02-19T18:38:15.062 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-361.el9.x86_64 2024-02-19T18:38:15.062 INFO:teuthology.task.kernel:Running kernel on smithi174: 5.14.0-361.el9.x86_64 2024-02-19T18:38:15.062 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum install -y kernel 2024-02-19T18:38:15.063 INFO:teuthology.task.kernel:Running kernel on smithi114: 5.14.0-361.el9.x86_64 2024-02-19T18:38:15.063 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2024-02-19T18:38:17.461 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - AppStream 9.4 MB/s | 6.7 MB 00:00 2024-02-19T18:38:17.509 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - AppStream 8.8 MB/s | 6.7 MB 00:00 2024-02-19T18:38:19.070 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-02-19T18:38:19.318 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - BaseOS 2.9 MB/s | 2.0 MB 00:00 2024-02-19T18:38:19.910 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-02-19T18:38:21.036 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.0 MB 00:01 2024-02-19T18:38:36.350 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-02-19T18:38:42.433 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 6.1 MB 00:02 2024-02-19T18:38:44.059 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2024-02-19T18:38:58.132 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 527 kB/s | 19 MB 00:37 2024-02-19T18:39:03.492 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.1 MB 00:01 2024-02-19T18:39:05.009 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 15 kB 00:00 2024-02-19T18:39:08.593 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 5.9 MB/s | 20 MB 00:03 2024-02-19T18:39:12.227 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 744 kB/s | 20 MB 00:28 2024-02-19T18:39:13.551 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-02-19T18:39:15.825 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-19T18:39:15.825 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-19T18:39:15.922 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:39:15.932 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:39:15.932 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-19T18:39:15.932 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:39:15.932 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-19T18:39:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 5.14.0-421.el9 CentOS-BaseOS 6.1 M 2024-02-19T18:39:15.933 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-19T18:39:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core x86_64 5.14.0-421.el9 CentOS-BaseOS 21 M 2024-02-19T18:39:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules x86_64 5.14.0-421.el9 CentOS-BaseOS 39 M 2024-02-19T18:39:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core x86_64 5.14.0-421.el9 CentOS-BaseOS 33 M 2024-02-19T18:39:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:39:15.933 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-19T18:39:15.933 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:39:15.933 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-02-19T18:39:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:39:15.934 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 100 M 2024-02-19T18:39:15.934 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 123 M 2024-02-19T18:39:15.934 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-19T18:39:17.232 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 5.9 kB/s | 1.7 kB 00:00 2024-02-19T18:39:17.652 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): kernel-5.14.0-421.el9.x86_64.rpm 3.5 MB/s | 6.1 MB 00:01 2024-02-19T18:39:19.136 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): kernel-core-5.14.0-421.el9.x86_64.rpm 6.5 MB/s | 21 MB 00:03 2024-02-19T18:39:19.489 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-19T18:39:19.489 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-19T18:39:19.595 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:39:19.600 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:39:19.601 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-02-19T18:39:19.601 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:39:19.601 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-02-19T18:39:19.601 INFO:teuthology.orchestra.run.smithi174.stdout: kernel x86_64 5.14.0-421.el9 CentOS-BaseOS 6.1 M 2024-02-19T18:39:19.601 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-02-19T18:39:19.601 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-core x86_64 5.14.0-421.el9 CentOS-BaseOS 21 M 2024-02-19T18:39:19.601 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules x86_64 5.14.0-421.el9 CentOS-BaseOS 39 M 2024-02-19T18:39:19.601 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-core x86_64 5.14.0-421.el9 CentOS-BaseOS 33 M 2024-02-19T18:39:19.601 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:39:19.601 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-02-19T18:39:19.602 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:39:19.602 INFO:teuthology.orchestra.run.smithi174.stdout:Install 4 Packages 2024-02-19T18:39:19.602 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:39:19.602 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 100 M 2024-02-19T18:39:19.602 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 123 M 2024-02-19T18:39:19.602 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-02-19T18:39:19.644 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): kernel-modules-core-5.14.0-421.el9.x86_6 17 MB/s | 33 MB 00:01 2024-02-19T18:39:20.117 INFO:teuthology.orchestra.run.smithi174.stdout:(1/4): kernel-5.14.0-421.el9.x86_64.rpm 12 MB/s | 6.1 MB 00:00 2024-02-19T18:39:20.395 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): kernel-modules-5.14.0-421.el9.x86_64.rpm 8.8 MB/s | 39 MB 00:04 2024-02-19T18:39:20.398 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:39:20.398 INFO:teuthology.orchestra.run.smithi114.stdout:Total 22 MB/s | 100 MB 00:04 2024-02-19T18:39:20.398 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-19T18:39:20.765 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-19T18:39:20.765 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-19T18:39:20.817 INFO:teuthology.orchestra.run.smithi174.stdout:(2/4): kernel-core-5.14.0-421.el9.x86_64.rpm 17 MB/s | 21 MB 00:01 2024-02-19T18:39:21.806 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-19T18:39:21.807 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-19T18:39:22.400 INFO:teuthology.orchestra.run.smithi174.stdout:(3/4): kernel-modules-5.14.0-421.el9.x86_64.rpm 14 MB/s | 39 MB 00:02 2024-02-19T18:39:22.700 INFO:teuthology.orchestra.run.smithi174.stdout:(4/4): kernel-modules-core-5.14.0-421.el9.x86_6 13 MB/s | 33 MB 00:02 2024-02-19T18:39:22.703 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:39:22.704 INFO:teuthology.orchestra.run.smithi174.stdout:Total 32 MB/s | 100 MB 00:03 2024-02-19T18:39:22.704 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-02-19T18:39:23.090 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-02-19T18:39:23.091 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-02-19T18:39:24.122 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-02-19T18:39:24.122 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-02-19T18:39:24.214 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-19T18:39:24.944 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-core-5.14.0-421.el9.x86_64 1/4 2024-02-19T18:39:25.078 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:39:26.528 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-02-19T18:39:27.190 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-modules-core-5.14.0-421.el9.x86_64 1/4 2024-02-19T18:39:27.319 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:39:27.816 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:39:27.996 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:39:30.054 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:39:30.252 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:39:32.683 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:39:32.801 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:39:35.173 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:39:35.298 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:39:37.557 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:39:40.012 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:40:06.703 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:40:06.703 INFO:teuthology.orchestra.run.smithi114.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-19T18:40:06.704 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:40:06.773 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:40:07.174 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:40:07.174 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-5.14.0-421.el9.x86_64 1/4 2024-02-19T18:40:07.175 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:40:07.175 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:40:07.579 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:40:07.580 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:40:07.580 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-19T18:40:07.580 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-5.14.0-421.el9.x86_64 2024-02-19T18:40:07.580 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core-5.14.0-421.el9.x86_64 2024-02-19T18:40:07.580 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-5.14.0-421.el9.x86_64 2024-02-19T18:40:07.580 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core-5.14.0-421.el9.x86_64 2024-02-19T18:40:07.580 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:40:07.580 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:40:08.088 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T18:40:08.226 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-19T18:40:08.226 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-19T18:40:08.226 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-421.el9.x86_64 2024-02-19T18:40:08.226 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T18:40:08.226 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T18:40:08.227 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-19T18:40:08.227 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T18:40:08.227 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T18:40:08.335 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-19T18:40:08.335 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-19T18:40:08.336 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2024-02-19T18:40:08.369 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2024-02-19T18:40:08.370 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2024-02-19T18:40:08.370 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.06 2024-02-19T18:40:08.370 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 68.el9 2024-02-19T18:40:08.370 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2024-02-19T18:40:08.370 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-19T18:40:08.370 INFO:teuthology.orchestra.run.smithi114.stdout:Group : Unspecified 2024-02-19T18:40:08.370 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 8264254 2024-02-19T18:40:08.370 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2024-02-19T18:40:08.370 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-19T18:40:08.370 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-19T18:40:08.370 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-19T18:40:08.371 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-19T18:40:08.371 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : builder@centos.org 2024-02-19T18:40:08.371 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2024-02-19T18:40:08.371 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T18:40:08.371 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2024-02-19T18:40:08.371 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2024-02-19T18:40:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:40:08.371 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T18:40:08.371 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T18:40:08.371 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T18:40:08.371 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2024-02-19T18:40:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:40:08.371 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T18:40:08.372 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T18:40:08.372 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 5.14.0-421.el9.x86_64 2024-02-19T18:40:08.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T18:40:09.006 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-02-19T18:40:09.052 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:40:09.052 INFO:teuthology.orchestra.run.smithi174.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-19T18:40:09.052 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:40:09.122 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:40:09.608 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:40:09.609 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-5.14.0-421.el9.x86_64 1/4 2024-02-19T18:40:09.609 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:40:09.609 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:40:10.069 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:40:10.069 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:40:10.069 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-02-19T18:40:10.069 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-5.14.0-421.el9.x86_64 2024-02-19T18:40:10.069 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-core-5.14.0-421.el9.x86_64 2024-02-19T18:40:10.069 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-5.14.0-421.el9.x86_64 2024-02-19T18:40:10.069 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-core-5.14.0-421.el9.x86_64 2024-02-19T18:40:10.069 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:40:10.069 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:40:10.327 INFO:teuthology.orchestra.run.smithi114.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-19T18:40:10.363 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-02-19T18:40:10.365 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-02-19T18:40:10.380 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.v7Ewg68ICQ 2024-02-19T18:40:10.380 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.v7Ewg68ICQ 2024-02-19T18:40:10.449 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.v7Ewg68ICQ 2024-02-19T18:40:10.564 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T18:40:10.597 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.v7Ewg68ICQ is 6KB 2024-02-19T18:40:10.648 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.v7Ewg68ICQ 2024-02-19T18:40:10.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T18:40:10.690 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-19T18:40:10.690 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-19T18:40:10.690 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-421.el9.x86_64 2024-02-19T18:40:10.690 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T18:40:10.691 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T18:40:10.691 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-19T18:40:10.691 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi174.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T18:40:10.691 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T18:40:10.727 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-19T18:40:10.727 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-19T18:40:10.728 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-19T18:40:10.728 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64.conf 2024-02-19T18:40:10.729 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64 2024-02-19T18:40:10.800 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-19T18:40:10.801 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-19T18:40:10.801 DEBUG:teuthology.orchestra.run.smithi174:> sudo rpm -qi grub2-tools 2024-02-19T18:40:10.840 INFO:teuthology.orchestra.run.smithi174.stdout:Name : grub2-tools 2024-02-19T18:40:10.840 INFO:teuthology.orchestra.run.smithi174.stdout:Epoch : 1 2024-02-19T18:40:10.840 INFO:teuthology.orchestra.run.smithi174.stdout:Version : 2.06 2024-02-19T18:40:10.841 INFO:teuthology.orchestra.run.smithi174.stdout:Release : 68.el9 2024-02-19T18:40:10.841 INFO:teuthology.orchestra.run.smithi174.stdout:Architecture: x86_64 2024-02-19T18:40:10.841 INFO:teuthology.orchestra.run.smithi174.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-19T18:40:10.841 INFO:teuthology.orchestra.run.smithi174.stdout:Group : Unspecified 2024-02-19T18:40:10.841 INFO:teuthology.orchestra.run.smithi174.stdout:Size : 8264254 2024-02-19T18:40:10.841 INFO:teuthology.orchestra.run.smithi174.stdout:License : GPLv3+ 2024-02-19T18:40:10.841 INFO:teuthology.orchestra.run.smithi174.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-19T18:40:10.841 INFO:teuthology.orchestra.run.smithi174.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-19T18:40:10.841 INFO:teuthology.orchestra.run.smithi174.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-19T18:40:10.841 INFO:teuthology.orchestra.run.smithi174.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-19T18:40:10.841 INFO:teuthology.orchestra.run.smithi174.stdout:Packager : builder@centos.org 2024-02-19T18:40:10.841 INFO:teuthology.orchestra.run.smithi174.stdout:Vendor : CentOS 2024-02-19T18:40:10.841 INFO:teuthology.orchestra.run.smithi174.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T18:40:10.841 INFO:teuthology.orchestra.run.smithi174.stdout:Summary : Support tools for GRUB. 2024-02-19T18:40:10.841 INFO:teuthology.orchestra.run.smithi174.stdout:Description : 2024-02-19T18:40:10.842 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:40:10.842 INFO:teuthology.orchestra.run.smithi174.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T18:40:10.842 INFO:teuthology.orchestra.run.smithi174.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T18:40:10.842 INFO:teuthology.orchestra.run.smithi174.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T18:40:10.842 INFO:teuthology.orchestra.run.smithi174.stdout:hardware devices. 2024-02-19T18:40:10.842 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:40:10.842 INFO:teuthology.orchestra.run.smithi174.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T18:40:10.843 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T18:40:10.843 INFO:teuthology.task.kernel:Updating grub on smithi174 to boot 5.14.0-421.el9.x86_64 2024-02-19T18:40:10.843 DEBUG:teuthology.orchestra.run.smithi174:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T18:40:10.871 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-02-19T18:40:11.481 INFO:teuthology.orchestra.run.smithi174.stderr:Generating grub configuration file ... 2024-02-19T18:40:12.928 INFO:teuthology.orchestra.run.smithi174.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-19T18:40:12.949 INFO:teuthology.orchestra.run.smithi174.stderr:done 2024-02-19T18:40:12.951 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2024-02-19T18:40:12.966 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.JnCLDgvzo0 2024-02-19T18:40:12.966 DEBUG:teuthology.orchestra.run.smithi174:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.JnCLDgvzo0 2024-02-19T18:40:13.052 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.JnCLDgvzo0 2024-02-19T18:40:13.146 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.JnCLDgvzo0 is 6KB 2024-02-19T18:40:13.198 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.JnCLDgvzo0 2024-02-19T18:40:13.215 DEBUG:teuthology.orchestra.run.smithi174:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T18:40:13.282 INFO:teuthology.orchestra.run.smithi174.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-19T18:40:13.283 INFO:teuthology.orchestra.run.smithi174.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-19T18:40:13.283 INFO:teuthology.orchestra.run.smithi174.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-19T18:40:13.283 INFO:teuthology.orchestra.run.smithi174.stdout:15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64.conf 2024-02-19T18:40:13.285 DEBUG:teuthology.orchestra.run.smithi174:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64 2024-02-19T18:40:13.440 DEBUG:teuthology.orchestra.run.smithi174:> sudo shutdown -r now 2024-02-19T18:40:40.900 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-19T18:40:40.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-02-19T18:40:40.902 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:40:43.444 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-19T18:40:43.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T18:40:43.445 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:41:15.581 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-02-19T18:41:24.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-02-19T18:41:24.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:41:42.973 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-02-19T18:41:43.448 DEBUG:teuthology.orchestra.remote:timed out 2024-02-19T18:41:52.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T18:41:52.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:41:52.812 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-02-19T18:41:53.285 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T18:41:53.285 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-421.el9.x86_64"... 2024-02-19T18:41:53.285 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-02-19T18:41:53.303 INFO:teuthology.orchestra.run.smithi174.stdout:5.14.0-421.el9.x86_64 2024-02-19T18:41:53.304 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-421.el9.x86_64 vs 5.14.0-421.el9.x86_64 2024-02-19T18:41:53.304 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T18:41:53.304 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T18:41:54.305 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-19T18:41:54.305 DEBUG:teuthology.orchestra.run.smithi174:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T18:41:54.385 INFO:teuthology.orchestra.run.smithi174.stdout:ttyS1 2024-02-19T18:41:54.434 DEBUG:teuthology.parallel:result is None 2024-02-19T18:41:54.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-02-19T18:41:54.974 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:41:55.359 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-02-19T18:41:55.759 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-02-19T18:41:55.760 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-421.el9.x86_64"... 2024-02-19T18:41:55.760 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-02-19T18:41:55.774 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-421.el9.x86_64 2024-02-19T18:41:55.775 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-421.el9.x86_64 vs 5.14.0-421.el9.x86_64 2024-02-19T18:41:55.775 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T18:41:55.775 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T18:41:56.780 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-19T18:41:56.780 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T18:41:56.875 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2024-02-19T18:41:56.921 DEBUG:teuthology.parallel:result is None 2024-02-19T18:41:56.921 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-19T18:41:56.930 INFO:teuthology.task.internal:Creating test directory... 2024-02-19T18:41:56.931 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T18:41:56.933 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T18:41:56.950 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-19T18:41:56.958 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-19T18:41:56.973 INFO:teuthology.task.internal:Creating archive directory... 2024-02-19T18:41:56.973 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T18:41:56.992 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T18:41:57.013 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-19T18:41:57.025 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-19T18:41:57.025 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T18:41:57.053 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T18:41:57.091 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T18:41:57.100 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T18:41:57.113 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T18:41:57.128 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T18:41:57.128 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-19T18:41:57.136 INFO:teuthology.task.internal:Configuring sudo... 2024-02-19T18:41:57.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T18:41:57.154 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T18:41:57.183 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-19T18:41:57.220 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-19T18:41:57.220 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T18:41:57.233 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T18:41:57.247 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T18:41:57.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T18:41:57.386 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T18:41:57.471 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T18:41:57.517 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T18:41:57.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T18:41:57.590 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T18:41:57.631 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T18:41:57.677 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T18:41:57.773 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T18:41:57.803 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-02-19T18:41:57.804 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T18:41:57.867 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-02-19T18:41:57.871 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-02-19T18:41:57.936 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T18:41:57.952 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T18:41:58.255 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-19T18:41:58.273 INFO:teuthology.task.internal:Starting timer... 2024-02-19T18:41:58.273 INFO:teuthology.run_tasks:Running task pcp... 2024-02-19T18:41:58.323 INFO:teuthology.run_tasks:Running task selinux... 2024-02-19T18:41:58.472 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-19T18:41:58.472 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-02-19T18:41:58.551 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-02-19T18:41:58.556 DEBUG:teuthology.orchestra.run.smithi174:> sudo service auditd rotate 2024-02-19T18:41:58.642 INFO:teuthology.orchestra.run.smithi174.stdout:Rotating logs: 2024-02-19T18:41:58.645 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-19T18:41:58.645 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-02-19T18:41:58.662 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-02-19T18:41:58.662 DEBUG:teuthology.orchestra.run.smithi174:> /usr/sbin/getenforce 2024-02-19T18:41:58.694 INFO:teuthology.orchestra.run.smithi174.stdout:Permissive 2024-02-19T18:41:58.695 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi114.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi174.front.sepia.ceph.com': 'permissive'} 2024-02-19T18:41:58.695 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T18:41:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T18:41:58.733 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T18:41:58.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T18:41:58.786 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-19T18:41:58.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-02-19T18:41:58.822 DEBUG:teuthology.orchestra.run.smithi174:> sudo /usr/sbin/setenforce permissive 2024-02-19T18:41:59.081 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-19T18:41:59.093 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-19T18:41:59.145 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-19T18:41:59.147 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi114.front.sepia.ceph.com,smithi174.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-19T18:47:47.387 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi114.front.sepia.ceph.com'), Remote(name='ubuntu@smithi174.front.sepia.ceph.com')] 2024-02-19T18:47:47.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-02-19T18:47:47.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:47:47.491 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-02-19T18:47:47.571 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-02-19T18:47:47.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T18:47:47.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:47:47.673 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-02-19T18:47:47.708 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T18:47:47.708 INFO:teuthology.run_tasks:Running task clock... 2024-02-19T18:47:47.724 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-19T18:47:47.724 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T18:47:47.724 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T18:47:47.727 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T18:47:47.727 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T18:47:47.755 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T18:47:47.771 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T18:47:47.779 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T18:47:47.796 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T18:47:47.800 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-02-19T18:47:47.814 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-02-19T18:47:47.825 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: ntpd: command not found 2024-02-19T18:47:47.830 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T18:47:47.839 INFO:teuthology.orchestra.run.smithi174.stdout:506 Cannot talk to daemon 2024-02-19T18:47:47.847 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T18:47:47.856 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T18:47:47.873 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T18:47:47.899 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-02-19T18:47:47.903 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T18:47:47.903 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-02-19T18:47:47.903 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:47:47.903 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:47:47.903 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:47:47.903 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:47:47.924 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-02-19T18:47:47.927 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T18:47:47.927 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-02-19T18:47:47.928 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:47:47.928 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:47:47.928 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:47:47.928 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:47:47.929 INFO:teuthology.run_tasks:Running task install... 2024-02-19T18:47:47.942 DEBUG:teuthology.task.install:project ceph 2024-02-19T18:47:47.942 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275'}} 2024-02-19T18:47:47.942 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v18.2.1', 'flavor': 'default', 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275'} 2024-02-19T18:47:47.942 INFO:teuthology.task.install:Using flavor: default 2024-02-19T18:47:47.950 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-19T18:47:47.950 INFO:teuthology.task.install:extra packages: [] 2024-02-19T18:47:47.951 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-02-19T18:47:47.951 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T18:47:47.951 INFO:teuthology.packaging:ref: None 2024-02-19T18:47:47.951 INFO:teuthology.packaging:tag: v18.2.1 2024-02-19T18:47:47.951 INFO:teuthology.packaging:branch: None 2024-02-19T18:47:47.951 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:47:48.269 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph-ci v18.2.1^{} -> None 2024-02-19T18:47:48.269 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-02-19T18:47:49.946 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph v18.2.1^{} -> 7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-02-19T18:47:49.946 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-02-19T18:47:49.949 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-02-19T18:47:49.950 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T18:47:49.950 INFO:teuthology.packaging:ref: None 2024-02-19T18:47:49.950 INFO:teuthology.packaging:tag: v18.2.1 2024-02-19T18:47:49.950 INFO:teuthology.packaging:branch: None 2024-02-19T18:47:49.950 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:47:49.950 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-02-19T18:47:49.950 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-02-19T18:47:50.095 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/ 2024-02-19T18:47:50.095 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-02-19T18:47:50.101 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/ 2024-02-19T18:47:50.101 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-02-19T18:47:50.142 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T18:47:50.143 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-02-19T18:47:50.143 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T18:47:50.145 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T18:47:50.146 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T18:47:50.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T18:47:50.178 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T18:47:50.178 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T18:47:50.179 INFO:teuthology.packaging:ref: None 2024-02-19T18:47:50.179 INFO:teuthology.packaging:tag: v18.2.1 2024-02-19T18:47:50.179 INFO:teuthology.packaging:branch: None 2024-02-19T18:47:50.179 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:47:50.179 DEBUG:teuthology.orchestra.run.smithi174:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T18:47:50.180 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T18:47:50.180 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T18:47:50.180 INFO:teuthology.packaging:ref: None 2024-02-19T18:47:50.180 INFO:teuthology.packaging:tag: v18.2.1 2024-02-19T18:47:50.180 INFO:teuthology.packaging:branch: None 2024-02-19T18:47:50.180 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:47:50.180 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T18:47:50.252 DEBUG:teuthology.orchestra.run.smithi174:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T18:47:50.253 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T18:47:50.335 DEBUG:teuthology.orchestra.run.smithi174:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T18:47:50.340 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T18:47:50.404 INFO:teuthology.orchestra.run.smithi174.stdout:check_obsoletes = 1 2024-02-19T18:47:50.405 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum clean all 2024-02-19T18:47:50.409 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-02-19T18:47:50.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-02-19T18:47:50.649 INFO:teuthology.orchestra.run.smithi174.stdout:74 files removed 2024-02-19T18:47:50.651 INFO:teuthology.orchestra.run.smithi114.stdout:74 files removed 2024-02-19T18:47:50.676 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-radosgw 2024-02-19T18:47:50.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw 2024-02-19T18:47:51.161 INFO:teuthology.orchestra.run.smithi174.stdout:ceph packages for x86_64 467 kB/s | 76 kB 00:00 2024-02-19T18:47:51.165 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 444 kB/s | 76 kB 00:00 2024-02-19T18:47:51.337 INFO:teuthology.orchestra.run.smithi174.stdout:ceph noarch packages 73 kB/s | 9.4 kB 00:00 2024-02-19T18:47:51.352 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 70 kB/s | 9.4 kB 00:00 2024-02-19T18:47:51.513 INFO:teuthology.orchestra.run.smithi174.stdout:ceph source packages 16 kB/s | 2.2 kB 00:00 2024-02-19T18:47:51.525 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 16 kB/s | 2.2 kB 00:00 2024-02-19T18:47:52.132 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-02-19T18:47:52.477 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - AppStream 7.3 MB/s | 6.7 MB 00:00 2024-02-19T18:47:53.541 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-02-19T18:47:54.003 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2024-02-19T18:47:54.423 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-02-19T18:47:56.182 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-02-19T18:47:56.702 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 3.4 MB/s | 8.0 MB 00:02 2024-02-19T18:48:01.083 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.1 MB 00:01 2024-02-19T18:48:02.527 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 4.0 MB/s | 19 MB 00:04 2024-02-19T18:48:02.834 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 15 kB 00:00 2024-02-19T18:48:05.099 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 9.5 MB/s | 20 MB 00:02 2024-02-19T18:48:07.356 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 6.1 MB 00:01 2024-02-19T18:48:09.129 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 15 kB 00:00 2024-02-19T18:48:09.952 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-02-19T18:48:12.531 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:48:12.538 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:48:12.538 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-02-19T18:48:12.538 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:48:12.539 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-02-19T18:48:12.539 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw x86_64 2:18.2.1-0.el9 ceph 7.7 M 2024-02-19T18:48:12.539 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrading: 2024-02-19T18:48:12.539 INFO:teuthology.orchestra.run.smithi174.stdout: librados2 x86_64 2:18.2.1-0.el9 ceph 3.3 M 2024-02-19T18:48:12.539 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1 x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-02-19T18:48:12.539 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-02-19T18:48:12.539 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-19T18:48:12.539 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base x86_64 2:18.2.1-0.el9 ceph 5.2 M 2024-02-19T18:48:12.539 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common x86_64 2:18.2.1-0.el9 ceph 18 M 2024-02-19T18:48:12.539 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux x86_64 2:18.2.1-0.el9 ceph 24 k 2024-02-19T18:48:12.539 INFO:teuthology.orchestra.run.smithi174.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-02-19T18:48:12.539 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-19T18:48:12.539 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-19T18:48:12.540 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-19T18:48:12.540 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2 x86_64 2:18.2.1-0.el9 ceph 658 k 2024-02-19T18:48:12.540 INFO:teuthology.orchestra.run.smithi174.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-19T18:48:12.540 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-19T18:48:12.540 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-19T18:48:12.540 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1 x86_64 2:18.2.1-0.el9 ceph 474 k 2024-02-19T18:48:12.540 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-19T18:48:12.540 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2 x86_64 2:18.2.1-0.el9 ceph 4.5 M 2024-02-19T18:48:12.540 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-19T18:48:12.540 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-19T18:48:12.540 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-19T18:48:12.540 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el9 ceph 45 k 2024-02-19T18:48:12.540 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common x86_64 2:18.2.1-0.el9 ceph 124 k 2024-02-19T18:48:12.541 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs x86_64 2:18.2.1-0.el9 ceph 161 k 2024-02-19T18:48:12.541 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-19T18:48:12.541 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados x86_64 2:18.2.1-0.el9 ceph 321 k 2024-02-19T18:48:12.541 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd x86_64 2:18.2.1-0.el9 ceph 297 k 2024-02-19T18:48:12.541 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw x86_64 2:18.2.1-0.el9 ceph 99 k 2024-02-19T18:48:12.541 INFO:teuthology.orchestra.run.smithi174.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-19T18:48:12.541 INFO:teuthology.orchestra.run.smithi174.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-19T18:48:12.541 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:12.541 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-02-19T18:48:12.541 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:48:12.541 INFO:teuthology.orchestra.run.smithi174.stdout:Install 28 Packages 2024-02-19T18:48:12.541 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrade 2 Packages 2024-02-19T18:48:12.541 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:12.542 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 53 M 2024-02-19T18:48:12.542 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-02-19T18:48:12.944 INFO:teuthology.orchestra.run.smithi174.stdout:(1/30): ceph-base-18.2.1-0.el9.x86_64.rpm 29 MB/s | 5.2 MB 00:00 2024-02-19T18:48:12.978 INFO:teuthology.orchestra.run.smithi174.stdout:(2/30): ceph-selinux-18.2.1-0.el9.x86_64.rpm 722 kB/s | 24 kB 00:00 2024-02-19T18:48:13.012 INFO:teuthology.orchestra.run.smithi174.stdout:(3/30): libcephfs2-18.2.1-0.el9.x86_64.rpm 19 MB/s | 658 kB 00:00 2024-02-19T18:48:13.045 INFO:teuthology.orchestra.run.smithi174.stdout:(4/30): libradosstriper1-18.2.1-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-02-19T18:48:13.162 INFO:teuthology.orchestra.run.smithi174.stdout:(5/30): ceph-radosgw-18.2.1-0.el9.x86_64.rpm 19 MB/s | 7.7 MB 00:00 2024-02-19T18:48:13.187 INFO:teuthology.orchestra.run.smithi174.stdout:(6/30): python3-ceph-argparse-18.2.1-0.el9.x86_ 1.8 MB/s | 45 kB 00:00 2024-02-19T18:48:13.213 INFO:teuthology.orchestra.run.smithi174.stdout:(7/30): python3-ceph-common-18.2.1-0.el9.x86_64 4.9 MB/s | 124 kB 00:00 2024-02-19T18:48:13.238 INFO:teuthology.orchestra.run.smithi174.stdout:(8/30): python3-cephfs-18.2.1-0.el9.x86_64.rpm 6.3 MB/s | 161 kB 00:00 2024-02-19T18:48:13.263 INFO:teuthology.orchestra.run.smithi174.stdout:(9/30): python3-rados-18.2.1-0.el9.x86_64.rpm 13 MB/s | 321 kB 00:00 2024-02-19T18:48:13.289 INFO:teuthology.orchestra.run.smithi174.stdout:(10/30): python3-rbd-18.2.1-0.el9.x86_64.rpm 12 MB/s | 297 kB 00:00 2024-02-19T18:48:13.314 INFO:teuthology.orchestra.run.smithi174.stdout:(11/30): python3-rgw-18.2.1-0.el9.x86_64.rpm 3.9 MB/s | 99 kB 00:00 2024-02-19T18:48:13.381 INFO:teuthology.orchestra.run.smithi174.stdout:(12/30): librgw2-18.2.1-0.el9.x86_64.rpm 13 MB/s | 4.5 MB 00:00 2024-02-19T18:48:13.598 INFO:teuthology.orchestra.run.smithi174.stdout:(13/30): ceph-common-18.2.1-0.el9.x86_64.rpm 22 MB/s | 18 MB 00:00 2024-02-19T18:48:13.765 INFO:teuthology.orchestra.run.smithi174.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 235 kB/s | 106 kB 00:00 2024-02-19T18:48:13.782 INFO:teuthology.orchestra.run.smithi174.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 280 kB/s | 44 kB 00:00 2024-02-19T18:48:13.807 INFO:teuthology.orchestra.run.smithi174.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 396 kB/s | 159 kB 00:00 2024-02-19T18:48:13.991 INFO:teuthology.orchestra.run.smithi174.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.9 MB/s | 661 kB 00:00 2024-02-19T18:48:14.074 INFO:teuthology.orchestra.run.smithi174.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 966 kB/s | 282 kB 00:00 2024-02-19T18:48:14.116 INFO:teuthology.orchestra.run.smithi174.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 133 kB/s | 41 kB 00:00 2024-02-19T18:48:14.441 INFO:teuthology.orchestra.run.smithi174.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 246 kB/s | 111 kB 00:00 2024-02-19T18:48:14.519 INFO:teuthology.orchestra.run.smithi174.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 346 kB/s | 26 kB 00:00 2024-02-19T18:48:14.542 INFO:teuthology.orchestra.run.smithi174.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 698 kB/s | 309 kB 00:00 2024-02-19T18:48:14.601 INFO:teuthology.orchestra.run.smithi174.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 581 kB/s | 49 kB 00:00 2024-02-19T18:48:14.643 INFO:teuthology.orchestra.run.smithi174.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 673 kB/s | 67 kB 00:00 2024-02-19T18:48:14.818 INFO:teuthology.orchestra.run.smithi174.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-02-19T18:48:15.735 INFO:teuthology.orchestra.run.smithi174.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 739 kB/s | 838 kB 00:01 2024-02-19T18:48:15.801 INFO:teuthology.orchestra.run.smithi174.stdout:(27/30): librados2-18.2.1-0.el9.x86_64.rpm 49 MB/s | 3.3 MB 00:00 2024-02-19T18:48:15.843 INFO:teuthology.orchestra.run.smithi174.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 1.5 MB/s | 1.6 MB 00:01 2024-02-19T18:48:15.902 INFO:teuthology.orchestra.run.smithi174.stdout:(29/30): librbd1-18.2.1-0.el9.x86_64.rpm 30 MB/s | 3.0 MB 00:00 2024-02-19T18:48:16.477 INFO:teuthology.orchestra.run.smithi174.stdout:(30/30): libarrow-9.0.0-10.el9.x86_64.rpm 1.9 MB/s | 4.4 MB 00:02 2024-02-19T18:48:16.485 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:48:16.485 INFO:teuthology.orchestra.run.smithi174.stdout:Total 13 MB/s | 53 MB 00:03 2024-02-19T18:48:16.485 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-02-19T18:48:16.521 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-02-19T18:48:16.521 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-02-19T18:48:16.978 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-02-19T18:48:16.978 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-02-19T18:48:18.349 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-02-19T18:48:18.428 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-02-19T18:48:18.465 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-02-19T18:48:18.516 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-02-19T18:48:18.886 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-02-19T18:48:18.922 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librados2-2:18.2.1-0.el9.x86_64 5/32 2024-02-19T18:48:18.997 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:18.2.1-0.el9.x86_64 5/32 2024-02-19T18:48:19.078 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rados-2:18.2.1-0.el9.x86_64 6/32 2024-02-19T18:48:19.124 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-02-19T18:48:19.170 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-02-19T18:48:19.199 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-02-19T18:48:19.242 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 9/32 2024-02-19T18:48:19.308 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-02-19T18:48:19.334 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-02-19T18:48:19.400 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-02-19T18:48:19.507 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-02-19T18:48:19.539 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-02-19T18:48:19.611 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-02-19T18:48:19.640 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-02-19T18:48:19.667 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-02-19T18:48:19.702 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-02-19T18:48:19.736 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-02-19T18:48:20.035 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-02-19T18:48:20.065 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-02-19T18:48:20.139 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-02-19T18:48:20.195 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rbd-2:18.2.1-0.el9.x86_64 21/32 2024-02-19T18:48:20.238 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-02-19T18:48:20.639 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-02-19T18:48:20.661 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-02-19T18:48:20.731 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-02-19T18:48:20.797 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rgw-2:18.2.1-0.el9.x86_64 25/32 2024-02-19T18:48:20.837 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-common-2:18.2.1-0.el9.x86_64 26/32 2024-02-19T18:48:22.903 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-02-19T18:48:22.950 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-02-19T18:48:23.501 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-02-19T18:48:23.544 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-02-19T18:48:23.629 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-02-19T18:48:23.629 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-19T18:48:23.629 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-19T18:48:23.630 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:23.671 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-02-19T18:48:33.373 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-02-19T18:48:33.374 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /sys 2024-02-19T18:48:33.374 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /proc 2024-02-19T18:48:33.374 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /mnt 2024-02-19T18:48:33.374 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /var/tmp 2024-02-19T18:48:33.374 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /home 2024-02-19T18:48:33.374 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /root 2024-02-19T18:48:33.374 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /tmp 2024-02-19T18:48:33.374 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:34.116 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-02-19T18:48:34.141 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-02-19T18:48:34.141 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:48:34.141 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-19T18:48:34.141 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-19T18:48:34.141 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-19T18:48:34.142 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:34.143 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-02-19T18:48:34.210 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-02-19T18:48:34.210 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-02-19T18:48:35.341 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-02-19T18:48:35.341 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-base-2:18.2.1-0.el9.x86_64 1/32 2024-02-19T18:48:35.341 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-common-2:18.2.1-0.el9.x86_64 2/32 2024-02-19T18:48:35.341 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el9.x86_64 3/32 2024-02-19T18:48:35.341 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-selinux-2:18.2.1-0.el9.x86_64 4/32 2024-02-19T18:48:35.341 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs2-2:18.2.1-0.el9.x86_64 5/32 2024-02-19T18:48:35.341 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libradosstriper1-2:18.2.1-0.el9.x86_64 6/32 2024-02-19T18:48:35.341 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librgw2-2:18.2.1-0.el9.x86_64 7/32 2024-02-19T18:48:35.341 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 8/32 2024-02-19T18:48:35.341 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el9.x86_64 9/32 2024-02-19T18:48:35.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-02-19T18:48:35.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rados-2:18.2.1-0.el9.x86_64 11/32 2024-02-19T18:48:35.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rbd-2:18.2.1-0.el9.x86_64 12/32 2024-02-19T18:48:35.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rgw-2:18.2.1-0.el9.x86_64 13/32 2024-02-19T18:48:35.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-02-19T18:48:35.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-02-19T18:48:35.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-02-19T18:48:35.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-02-19T18:48:35.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-02-19T18:48:35.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-02-19T18:48:35.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-02-19T18:48:35.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-02-19T18:48:35.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-02-19T18:48:35.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-02-19T18:48:35.342 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-02-19T18:48:35.343 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-02-19T18:48:35.343 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-02-19T18:48:35.343 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-02-19T18:48:35.343 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-02-19T18:48:35.343 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:18.2.1-0.el9.x86_64 29/32 2024-02-19T18:48:35.343 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-02-19T18:48:35.343 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:18.2.1-0.el9.x86_64 31/32 2024-02-19T18:48:36.336 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-02-19T18:48:36.336 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:36.336 INFO:teuthology.orchestra.run.smithi174.stdout:Upgraded: 2024-02-19T18:48:36.336 INFO:teuthology.orchestra.run.smithi174.stdout: librados2-2:18.2.1-0.el9.x86_64 librbd1-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:36.336 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-02-19T18:48:36.336 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-19T18:48:36.337 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:36.337 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:36.337 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:36.337 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:36.337 INFO:teuthology.orchestra.run.smithi174.stdout: fmt-8.1.1-5.el9.x86_64 2024-02-19T18:48:36.337 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-19T18:48:36.337 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-19T18:48:36.337 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-19T18:48:36.337 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:36.337 INFO:teuthology.orchestra.run.smithi174.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-19T18:48:36.337 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-19T18:48:36.337 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-19T18:48:36.337 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:36.337 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-19T18:48:36.338 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:36.338 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-19T18:48:36.338 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-19T18:48:36.338 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-19T18:48:36.338 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:36.338 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:36.338 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:36.338 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-19T18:48:36.338 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:36.338 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:36.338 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:36.338 INFO:teuthology.orchestra.run.smithi174.stdout: re2-1:20211101-3.el9.x86_64 2024-02-19T18:48:36.338 INFO:teuthology.orchestra.run.smithi174.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-19T18:48:36.338 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:36.338 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:48:36.506 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 766 kB/s | 20 MB 00:27 2024-02-19T18:48:36.578 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-test 2024-02-19T18:48:37.155 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:28 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:48:37.327 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:48:37.328 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:48:37.328 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-02-19T18:48:37.328 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:48:37.328 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-02-19T18:48:37.328 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test x86_64 2:18.2.1-0.el9 ceph 40 M 2024-02-19T18:48:37.328 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-02-19T18:48:37.329 INFO:teuthology.orchestra.run.smithi174.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-19T18:48:37.329 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite x86_64 2:18.2.1-0.el9 ceph 165 k 2024-02-19T18:48:37.329 INFO:teuthology.orchestra.run.smithi174.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-19T18:48:37.329 INFO:teuthology.orchestra.run.smithi174.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-19T18:48:37.329 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-19T18:48:37.329 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:37.329 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-02-19T18:48:37.329 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:48:37.329 INFO:teuthology.orchestra.run.smithi174.stdout:Install 6 Packages 2024-02-19T18:48:37.329 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:37.330 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 41 M 2024-02-19T18:48:37.330 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 167 M 2024-02-19T18:48:37.330 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-02-19T18:48:37.464 INFO:teuthology.orchestra.run.smithi174.stdout:(1/6): libcephsqlite-18.2.1-0.el9.x86_64.rpm 1.2 MB/s | 165 kB 00:00 2024-02-19T18:48:37.547 INFO:teuthology.orchestra.run.smithi174.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-02-19T18:48:37.564 INFO:teuthology.orchestra.run.smithi174.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 626 kB/s | 63 kB 00:00 2024-02-19T18:48:37.589 INFO:teuthology.orchestra.run.smithi174.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-02-19T18:48:37.614 INFO:teuthology.orchestra.run.smithi174.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-02-19T18:48:38.060 INFO:teuthology.orchestra.run.smithi174.stdout:(6/6): ceph-test-18.2.1-0.el9.x86_64.rpm 55 MB/s | 40 MB 00:00 2024-02-19T18:48:38.060 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:48:38.060 INFO:teuthology.orchestra.run.smithi174.stdout:Total 56 MB/s | 41 MB 00:00 2024-02-19T18:48:38.060 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-02-19T18:48:38.079 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-02-19T18:48:38.079 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-02-19T18:48:38.432 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-02-19T18:48:38.432 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-02-19T18:48:38.983 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-02-19T18:48:39.063 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-19T18:48:39.097 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-19T18:48:39.154 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-19T18:48:39.198 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-19T18:48:39.226 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-02-19T18:48:41.394 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-02-19T18:48:42.947 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-02-19T18:48:43.016 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-02-19T18:48:43.960 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:48:43.974 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:48:43.974 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-19T18:48:43.975 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:48:43.975 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-19T18:48:43.975 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:18.2.1-0.el9 ceph 7.7 M 2024-02-19T18:48:43.975 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-02-19T18:48:43.975 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:18.2.1-0.el9 ceph 3.3 M 2024-02-19T18:48:43.975 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-02-19T18:48:43.975 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-19T18:48:43.975 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-19T18:48:43.975 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:18.2.1-0.el9 ceph 5.2 M 2024-02-19T18:48:43.975 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:18.2.1-0.el9 ceph 18 M 2024-02-19T18:48:43.975 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:18.2.1-0.el9 ceph 24 k 2024-02-19T18:48:43.975 INFO:teuthology.orchestra.run.smithi114.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-02-19T18:48:43.975 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-19T18:48:43.975 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-19T18:48:43.976 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-19T18:48:43.976 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:18.2.1-0.el9 ceph 658 k 2024-02-19T18:48:43.976 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-19T18:48:43.976 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-19T18:48:43.976 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-19T18:48:43.976 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:18.2.1-0.el9 ceph 474 k 2024-02-19T18:48:43.976 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-19T18:48:43.976 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:18.2.1-0.el9 ceph 4.5 M 2024-02-19T18:48:43.976 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-19T18:48:43.976 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-19T18:48:43.976 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-19T18:48:43.976 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el9 ceph 45 k 2024-02-19T18:48:43.976 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:18.2.1-0.el9 ceph 124 k 2024-02-19T18:48:43.976 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:18.2.1-0.el9 ceph 161 k 2024-02-19T18:48:43.977 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-19T18:48:43.977 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:18.2.1-0.el9 ceph 321 k 2024-02-19T18:48:43.977 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:18.2.1-0.el9 ceph 297 k 2024-02-19T18:48:43.977 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:18.2.1-0.el9 ceph 99 k 2024-02-19T18:48:43.977 INFO:teuthology.orchestra.run.smithi114.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-19T18:48:43.977 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-19T18:48:43.977 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:48:43.977 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-19T18:48:43.977 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:48:43.977 INFO:teuthology.orchestra.run.smithi114.stdout:Install 28 Packages 2024-02-19T18:48:43.978 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 2 Packages 2024-02-19T18:48:43.978 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:48:43.978 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 53 M 2024-02-19T18:48:43.978 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-19T18:48:44.429 INFO:teuthology.orchestra.run.smithi114.stdout:(1/30): ceph-base-18.2.1-0.el9.x86_64.rpm 26 MB/s | 5.2 MB 00:00 2024-02-19T18:48:44.446 INFO:teuthology.orchestra.run.smithi114.stdout:(2/30): ceph-selinux-18.2.1-0.el9.x86_64.rpm 1.4 MB/s | 24 kB 00:00 2024-02-19T18:48:44.479 INFO:teuthology.orchestra.run.smithi114.stdout:(3/30): libcephfs2-18.2.1-0.el9.x86_64.rpm 19 MB/s | 658 kB 00:00 2024-02-19T18:48:44.513 INFO:teuthology.orchestra.run.smithi114.stdout:(4/30): libradosstriper1-18.2.1-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-02-19T18:48:44.621 INFO:teuthology.orchestra.run.smithi114.stdout:(5/30): ceph-radosgw-18.2.1-0.el9.x86_64.rpm 20 MB/s | 7.7 MB 00:00 2024-02-19T18:48:44.638 INFO:teuthology.orchestra.run.smithi114.stdout:(6/30): python3-ceph-argparse-18.2.1-0.el9.x86_ 2.6 MB/s | 45 kB 00:00 2024-02-19T18:48:44.663 INFO:teuthology.orchestra.run.smithi114.stdout:(7/30): python3-ceph-common-18.2.1-0.el9.x86_64 4.9 MB/s | 124 kB 00:00 2024-02-19T18:48:44.680 INFO:teuthology.orchestra.run.smithi114.stdout:(8/30): python3-cephfs-18.2.1-0.el9.x86_64.rpm 9.5 MB/s | 161 kB 00:00 2024-02-19T18:48:44.706 INFO:teuthology.orchestra.run.smithi114.stdout:(9/30): python3-rados-18.2.1-0.el9.x86_64.rpm 13 MB/s | 321 kB 00:00 2024-02-19T18:48:44.731 INFO:teuthology.orchestra.run.smithi114.stdout:(10/30): python3-rbd-18.2.1-0.el9.x86_64.rpm 12 MB/s | 297 kB 00:00 2024-02-19T18:48:44.748 INFO:teuthology.orchestra.run.smithi114.stdout:(11/30): python3-rgw-18.2.1-0.el9.x86_64.rpm 5.9 MB/s | 99 kB 00:00 2024-02-19T18:48:44.815 INFO:teuthology.orchestra.run.smithi114.stdout:(12/30): librgw2-18.2.1-0.el9.x86_64.rpm 15 MB/s | 4.5 MB 00:00 2024-02-19T18:48:45.032 INFO:teuthology.orchestra.run.smithi114.stdout:(13/30): ceph-common-18.2.1-0.el9.x86_64.rpm 23 MB/s | 18 MB 00:00 2024-02-19T18:48:45.079 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-02-19T18:48:45.079 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-test-2:18.2.1-0.el9.x86_64 1/6 2024-02-19T18:48:45.079 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephsqlite-2:18.2.1-0.el9.x86_64 2/6 2024-02-19T18:48:45.079 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-19T18:48:45.079 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-19T18:48:45.079 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-19T18:48:45.132 INFO:teuthology.orchestra.run.smithi114.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 275 kB/s | 106 kB 00:00 2024-02-19T18:48:45.157 INFO:teuthology.orchestra.run.smithi114.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 441 kB/s | 44 kB 00:00 2024-02-19T18:48:45.191 INFO:teuthology.orchestra.run.smithi114.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 452 kB/s | 159 kB 00:00 2024-02-19T18:48:45.300 INFO:teuthology.orchestra.run.smithi114.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.9 MB/s | 661 kB 00:00 2024-02-19T18:48:45.325 INFO:teuthology.orchestra.run.smithi114.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 282 kB 00:00 2024-02-19T18:48:45.350 INFO:teuthology.orchestra.run.smithi114.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 259 kB/s | 41 kB 00:00 2024-02-19T18:48:45.458 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-19T18:48:45.459 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:45.459 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-02-19T18:48:45.459 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test-2:18.2.1-0.el9.x86_64 jq-1.6-16.el9.x86_64 2024-02-19T18:48:45.459 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite-2:18.2.1-0.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2024-02-19T18:48:45.459 INFO:teuthology.orchestra.run.smithi174.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-02-19T18:48:45.459 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:45.459 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:48:45.760 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph 2024-02-19T18:48:45.834 INFO:teuthology.orchestra.run.smithi114.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 207 kB/s | 111 kB 00:00 2024-02-19T18:48:45.943 INFO:teuthology.orchestra.run.smithi114.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-02-19T18:48:45.968 INFO:teuthology.orchestra.run.smithi114.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 481 kB/s | 309 kB 00:00 2024-02-19T18:48:46.068 INFO:teuthology.orchestra.run.smithi114.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 388 kB/s | 49 kB 00:00 2024-02-19T18:48:46.093 INFO:teuthology.orchestra.run.smithi114.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 538 kB/s | 67 kB 00:00 2024-02-19T18:48:46.235 INFO:teuthology.orchestra.run.smithi114.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-02-19T18:48:46.319 INFO:teuthology.orchestra.run.smithi114.stdout:(26/30): libarrow-9.0.0-10.el9.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2024-02-19T18:48:46.340 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:37 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:48:46.386 INFO:teuthology.orchestra.run.smithi114.stdout:(27/30): librados2-18.2.1-0.el9.x86_64.rpm 49 MB/s | 3.3 MB 00:00 2024-02-19T18:48:46.453 INFO:teuthology.orchestra.run.smithi114.stdout:(28/30): librbd1-18.2.1-0.el9.x86_64.rpm 45 MB/s | 3.0 MB 00:00 2024-02-19T18:48:46.478 INFO:teuthology.orchestra.run.smithi114.stdout:(29/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.0 MB/s | 838 kB 00:00 2024-02-19T18:48:46.513 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:48:46.515 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:48:46.515 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-02-19T18:48:46.515 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:48:46.515 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-02-19T18:48:46.516 INFO:teuthology.orchestra.run.smithi174.stdout: ceph x86_64 2:18.2.1-0.el9 ceph 6.4 k 2024-02-19T18:48:46.516 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-02-19T18:48:46.516 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds x86_64 2:18.2.1-0.el9 ceph 2.1 M 2024-02-19T18:48:46.516 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr x86_64 2:18.2.1-0.el9 ceph 1.4 M 2024-02-19T18:48:46.516 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el9 ceph-noarch 242 k 2024-02-19T18:48:46.516 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon x86_64 2:18.2.1-0.el9 ceph 4.4 M 2024-02-19T18:48:46.516 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd x86_64 2:18.2.1-0.el9 ceph 18 M 2024-02-19T18:48:46.516 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-19T18:48:46.516 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-19T18:48:46.516 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-19T18:48:46.516 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-19T18:48:46.516 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-19T18:48:46.516 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-19T18:48:46.517 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-19T18:48:46.517 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-19T18:48:46.517 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-19T18:48:46.517 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-19T18:48:46.517 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-19T18:48:46.517 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-19T18:48:46.517 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-19T18:48:46.517 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-19T18:48:46.517 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-19T18:48:46.517 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-19T18:48:46.517 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-19T18:48:46.517 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-19T18:48:46.517 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-19T18:48:46.517 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-19T18:48:46.517 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-19T18:48:46.517 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-19T18:48:46.518 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-19T18:48:46.518 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-19T18:48:46.518 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-19T18:48:46.518 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-19T18:48:46.518 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-19T18:48:46.518 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:46.518 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-02-19T18:48:46.518 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:48:46.518 INFO:teuthology.orchestra.run.smithi174.stdout:Install 33 Packages 2024-02-19T18:48:46.518 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:46.519 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 30 M 2024-02-19T18:48:46.519 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 110 M 2024-02-19T18:48:46.519 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-02-19T18:48:46.586 INFO:teuthology.orchestra.run.smithi114.stdout:(30/30): thrift-0.15.0-2.el9.x86_64.rpm 4.5 MB/s | 1.6 MB 00:00 2024-02-19T18:48:46.592 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:48:46.592 INFO:teuthology.orchestra.run.smithi114.stdout:Total 20 MB/s | 53 MB 00:02 2024-02-19T18:48:46.593 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-19T18:48:46.623 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-19T18:48:46.623 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-19T18:48:46.765 INFO:teuthology.orchestra.run.smithi174.stdout:(1/33): ceph-18.2.1-0.el9.x86_64.rpm 51 kB/s | 6.4 kB 00:00 2024-02-19T18:48:46.823 INFO:teuthology.orchestra.run.smithi174.stdout:(2/33): ceph-mds-18.2.1-0.el9.x86_64.rpm 11 MB/s | 2.1 MB 00:00 2024-02-19T18:48:46.865 INFO:teuthology.orchestra.run.smithi174.stdout:(3/33): ceph-mgr-18.2.1-0.el9.x86_64.rpm 6.4 MB/s | 1.4 MB 00:00 2024-02-19T18:48:46.891 INFO:teuthology.orchestra.run.smithi174.stdout:(4/33): ceph-mgr-modules-core-18.2.1-0.el9.noar 9.5 MB/s | 242 kB 00:00 2024-02-19T18:48:47.016 INFO:teuthology.orchestra.run.smithi174.stdout:(5/33): ceph-mon-18.2.1-0.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-02-19T18:48:47.058 INFO:teuthology.orchestra.run.smithi174.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 780 kB/s | 32 kB 00:00 2024-02-19T18:48:47.080 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-19T18:48:47.081 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-19T18:48:47.091 INFO:teuthology.orchestra.run.smithi174.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 719 kB/s | 144 kB 00:00 2024-02-19T18:48:47.117 INFO:teuthology.orchestra.run.smithi174.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 815 kB/s | 47 kB 00:00 2024-02-19T18:48:47.142 INFO:teuthology.orchestra.run.smithi174.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 770 kB/s | 39 kB 00:00 2024-02-19T18:48:47.159 INFO:teuthology.orchestra.run.smithi174.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-02-19T18:48:47.201 INFO:teuthology.orchestra.run.smithi174.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 20 MB/s | 1.1 MB 00:00 2024-02-19T18:48:47.226 INFO:teuthology.orchestra.run.smithi174.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-02-19T18:48:47.251 INFO:teuthology.orchestra.run.smithi174.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-02-19T18:48:47.277 INFO:teuthology.orchestra.run.smithi174.stdout:(14/33): python3-requests-2.25.1-8.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-02-19T18:48:47.302 INFO:teuthology.orchestra.run.smithi174.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-02-19T18:48:47.494 INFO:teuthology.orchestra.run.smithi174.stdout:(16/33): ceph-osd-18.2.1-0.el9.x86_64.rpm 26 MB/s | 18 MB 00:00 2024-02-19T18:48:47.736 INFO:teuthology.orchestra.run.smithi174.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 95 kB/s | 43 kB 00:00 2024-02-19T18:48:47.811 INFO:teuthology.orchestra.run.smithi174.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 337 kB/s | 172 kB 00:00 2024-02-19T18:48:47.836 INFO:teuthology.orchestra.run.smithi174.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-02-19T18:48:47.903 INFO:teuthology.orchestra.run.smithi174.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-02-19T18:48:47.929 INFO:teuthology.orchestra.run.smithi174.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-02-19T18:48:48.012 INFO:teuthology.orchestra.run.smithi174.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-02-19T18:48:48.029 INFO:teuthology.orchestra.run.smithi174.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-02-19T18:48:48.079 INFO:teuthology.orchestra.run.smithi174.stdout:(24/33): python3-cherrypy-18.6.1-2.el9.noarch.r 613 kB/s | 358 kB 00:00 2024-02-19T18:48:48.096 INFO:teuthology.orchestra.run.smithi174.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 552 kB/s | 46 kB 00:00 2024-02-19T18:48:48.147 INFO:teuthology.orchestra.run.smithi174.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 673 kB/s | 79 kB 00:00 2024-02-19T18:48:48.197 INFO:teuthology.orchestra.run.smithi174.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-02-19T18:48:48.214 INFO:teuthology.orchestra.run.smithi174.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 140 kB/s | 16 kB 00:00 2024-02-19T18:48:48.256 INFO:teuthology.orchestra.run.smithi174.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 825 kB/s | 90 kB 00:00 2024-02-19T18:48:48.300 INFO:teuthology.orchestra.run.smithi174.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 356 kB/s | 36 kB 00:00 2024-02-19T18:48:48.398 INFO:teuthology.orchestra.run.smithi174.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-02-19T18:48:48.423 INFO:teuthology.orchestra.run.smithi174.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 160 kB/s | 20 kB 00:00 2024-02-19T18:48:48.496 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-19T18:48:48.532 INFO:teuthology.orchestra.run.smithi174.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.5 MB/s | 427 kB 00:00 2024-02-19T18:48:48.535 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:48:48.535 INFO:teuthology.orchestra.run.smithi174.stdout:Total 15 MB/s | 30 MB 00:02 2024-02-19T18:48:48.536 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-02-19T18:48:48.570 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-02-19T18:48:48.574 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-02-19T18:48:48.574 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-02-19T18:48:48.599 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-02-19T18:48:48.650 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-02-19T18:48:48.932 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-02-19T18:48:48.933 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-02-19T18:48:49.012 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-02-19T18:48:49.048 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:18.2.1-0.el9.x86_64 5/32 2024-02-19T18:48:49.141 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:18.2.1-0.el9.x86_64 5/32 2024-02-19T18:48:49.220 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:18.2.1-0.el9.x86_64 6/32 2024-02-19T18:48:49.252 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-02-19T18:48:49.315 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-02-19T18:48:49.343 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-02-19T18:48:49.388 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 9/32 2024-02-19T18:48:49.453 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-02-19T18:48:49.480 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-02-19T18:48:49.545 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-02-19T18:48:49.652 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-02-19T18:48:49.686 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-02-19T18:48:49.736 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-02-19T18:48:49.785 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-02-19T18:48:49.812 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-02-19T18:48:49.847 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-02-19T18:48:49.880 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-02-19T18:48:50.179 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-02-19T18:48:50.182 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-02-19T18:48:50.210 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-02-19T18:48:50.251 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-19T18:48:50.283 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-19T18:48:50.300 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-02-19T18:48:50.316 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-19T18:48:50.332 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:18.2.1-0.el9.x86_64 21/32 2024-02-19T18:48:50.376 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-02-19T18:48:50.414 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-19T18:48:50.493 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-19T18:48:50.535 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-19T18:48:50.614 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-19T18:48:50.668 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-19T18:48:50.707 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-19T18:48:50.755 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-19T18:48:50.776 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-02-19T18:48:50.796 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-02-19T18:48:50.813 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-19T18:48:50.851 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-02-19T18:48:50.918 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:18.2.1-0.el9.x86_64 25/32 2024-02-19T18:48:50.949 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:18.2.1-0.el9.x86_64 26/32 2024-02-19T18:48:51.003 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-19T18:48:51.053 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-19T18:48:51.097 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-19T18:48:51.134 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-19T18:48:51.165 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-19T18:48:51.241 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-19T18:48:51.283 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-19T18:48:51.325 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-19T18:48:51.364 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-19T18:48:51.400 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-19T18:48:51.443 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-19T18:48:51.531 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-19T18:48:51.565 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-19T18:48:51.646 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-19T18:48:51.783 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-19T18:48:51.908 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-19T18:48:52.070 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 28/33 2024-02-19T18:48:52.103 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-02-19T18:48:52.128 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-02-19T18:48:52.128 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:48:52.128 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-19T18:48:52.128 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-19T18:48:52.128 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-19T18:48:52.128 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:52.918 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-02-19T18:48:52.954 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-02-19T18:48:53.496 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-02-19T18:48:53.532 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-02-19T18:48:53.604 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-02-19T18:48:53.605 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-19T18:48:53.605 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-19T18:48:53.605 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:48:53.633 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-02-19T18:48:53.638 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-02-19T18:48:53.664 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-02-19T18:48:53.664 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:48:53.665 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-19T18:48:53.665 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-19T18:48:53.665 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-19T18:48:53.665 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:54.113 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-02-19T18:48:54.137 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-02-19T18:48:54.137 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:48:54.137 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-19T18:48:54.137 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-19T18:48:54.137 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-19T18:48:54.137 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:54.356 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-02-19T18:48:54.382 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-02-19T18:48:54.382 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:48:54.382 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-19T18:48:54.382 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-19T18:48:54.382 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-19T18:48:54.383 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:54.453 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-2:18.2.1-0.el9.x86_64 33/33 2024-02-19T18:48:56.368 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-2:18.2.1-0.el9.x86_64 33/33 2024-02-19T18:48:56.369 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-2:18.2.1-0.el9.x86_64 1/33 2024-02-19T18:48:56.369 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mds-2:18.2.1-0.el9.x86_64 2/33 2024-02-19T18:48:56.369 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-2:18.2.1-0.el9.x86_64 3/33 2024-02-19T18:48:56.369 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mon-2:18.2.1-0.el9.x86_64 4/33 2024-02-19T18:48:56.369 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-osd-2:18.2.1-0.el9.x86_64 5/33 2024-02-19T18:48:56.369 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 6/33 2024-02-19T18:48:56.369 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-19T18:48:56.369 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-19T18:48:56.369 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-19T18:48:56.369 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-19T18:48:56.369 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-19T18:48:56.369 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-19T18:48:56.369 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-19T18:48:56.370 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-19T18:48:56.370 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-19T18:48:56.370 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-19T18:48:56.370 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-19T18:48:56.370 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-19T18:48:56.370 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-19T18:48:56.370 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-19T18:48:56.370 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-19T18:48:56.370 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-19T18:48:56.370 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-19T18:48:56.370 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-19T18:48:56.370 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-19T18:48:56.370 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-19T18:48:56.370 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-19T18:48:56.370 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-19T18:48:56.371 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-19T18:48:56.371 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-19T18:48:56.371 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-19T18:48:56.371 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-19T18:48:57.145 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-19T18:48:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:57.146 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-02-19T18:48:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 2024-02-19T18:48:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd-2:18.2.1-0.el9.x86_64 2024-02-19T18:48:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-19T18:48:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-19T18:48:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-19T18:48:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-19T18:48:57.146 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-19T18:48:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-19T18:48:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-19T18:48:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-19T18:48:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-19T18:48:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-19T18:48:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-19T18:48:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-19T18:48:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-19T18:48:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-19T18:48:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-19T18:48:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply-3.11-14.el9.noarch 2024-02-19T18:48:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-19T18:48:57.147 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-19T18:48:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-19T18:48:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-19T18:48:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-19T18:48:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-19T18:48:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-19T18:48:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-19T18:48:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-19T18:48:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-19T18:48:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-19T18:48:57.148 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:57.148 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:48:57.382 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-base 2024-02-19T18:48:57.967 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:48 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:48:58.081 INFO:teuthology.orchestra.run.smithi174.stdout:Package ceph-base-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:48:58.131 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:48:58.132 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-02-19T18:48:58.132 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:48:58.194 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install cephadm 2024-02-19T18:48:58.773 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:49 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:48:58.940 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:48:58.941 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:48:58.943 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-02-19T18:48:58.943 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:48:58.943 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-02-19T18:48:58.943 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm noarch 2:18.2.1-0.el9 ceph-noarch 221 k 2024-02-19T18:48:58.943 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:58.943 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-02-19T18:48:58.943 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:48:58.943 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-02-19T18:48:58.943 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:48:58.943 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 221 k 2024-02-19T18:48:58.943 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 213 k 2024-02-19T18:48:58.944 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-02-19T18:48:59.002 INFO:teuthology.orchestra.run.smithi174.stdout:cephadm-18.2.1-0.el9.noarch.rpm 3.6 MB/s | 221 kB 00:00 2024-02-19T18:48:59.003 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:48:59.003 INFO:teuthology.orchestra.run.smithi174.stdout:Total 3.5 MB/s | 221 kB 00:00 2024-02-19T18:48:59.003 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-02-19T18:48:59.008 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-02-19T18:48:59.008 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-02-19T18:48:59.011 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-02-19T18:48:59.011 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-02-19T18:48:59.083 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-02-19T18:48:59.496 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-02-19T18:48:59.590 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-02-19T18:48:59.980 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-02-19T18:49:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-02-19T18:49:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:00.277 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-02-19T18:49:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm-2:18.2.1-0.el9.noarch 2024-02-19T18:49:00.277 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:00.277 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:00.485 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T18:49:01.075 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:52 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:01.242 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:01.242 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:01.242 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repo Size 2024-02-19T18:49:01.242 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:01.243 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-02-19T18:49:01.243 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el9 ceph 142 k 2024-02-19T18:49:01.243 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:01.243 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-02-19T18:49:01.243 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:01.243 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-02-19T18:49:01.243 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:01.244 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 142 k 2024-02-19T18:49:01.244 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 395 k 2024-02-19T18:49:01.244 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-02-19T18:49:01.307 INFO:teuthology.orchestra.run.smithi174.stdout:ceph-immutable-object-cache-18.2.1-0.el9.x86_64 2.2 MB/s | 142 kB 00:00 2024-02-19T18:49:01.308 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:01.308 INFO:teuthology.orchestra.run.smithi174.stdout:Total 2.1 MB/s | 142 kB 00:00 2024-02-19T18:49:01.308 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-02-19T18:49:01.315 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-02-19T18:49:01.316 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-02-19T18:49:01.367 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-02-19T18:49:01.368 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-02-19T18:49:01.507 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-02-19T18:49:01.566 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:01.591 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:01.591 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:49:01.591 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-19T18:49:01.591 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:02.607 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:02.607 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:02.607 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-02-19T18:49:02.607 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:02.607 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:02.607 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:02.845 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr 2024-02-19T18:49:02.957 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-02-19T18:49:02.957 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-02-19T18:49:02.957 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-02-19T18:49:02.957 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-02-19T18:49:02.958 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-02-19T18:49:02.958 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-02-19T18:49:02.958 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /root 2024-02-19T18:49:02.958 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-02-19T18:49:02.958 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:03.440 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:54 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:03.556 INFO:teuthology.orchestra.run.smithi174.stdout:Package ceph-mgr-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:49:03.608 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:03.609 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-02-19T18:49:03.609 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:03.675 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T18:49:03.722 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-02-19T18:49:03.753 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-02-19T18:49:03.753 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:49:03.753 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-19T18:49:03.753 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-19T18:49:03.753 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-19T18:49:03.753 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:03.755 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-02-19T18:49:03.841 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-02-19T18:49:03.842 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-02-19T18:49:04.252 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:55 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:04.420 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:04.421 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:04.421 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-02-19T18:49:04.421 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:04.421 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-02-19T18:49:04.421 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el9 ceph-noarch 1.8 M 2024-02-19T18:49:04.421 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-02-19T18:49:04.421 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el9 ceph-noarch 23 k 2024-02-19T18:49:04.421 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el9 ceph-noarch 15 k 2024-02-19T18:49:04.422 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-19T18:49:04.422 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-19T18:49:04.422 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-19T18:49:04.422 INFO:teuthology.orchestra.run.smithi174.stdout:Installing weak dependencies: 2024-02-19T18:49:04.422 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-19T18:49:04.422 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:04.422 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-02-19T18:49:04.422 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:04.422 INFO:teuthology.orchestra.run.smithi174.stdout:Install 7 Packages 2024-02-19T18:49:04.422 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:04.423 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 2.1 M 2024-02-19T18:49:04.423 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 8.5 M 2024-02-19T18:49:04.423 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-02-19T18:49:04.586 INFO:teuthology.orchestra.run.smithi174.stdout:(1/7): ceph-grafana-dashboards-18.2.1-0.el9.noa 201 kB/s | 23 kB 00:00 2024-02-19T18:49:04.611 INFO:teuthology.orchestra.run.smithi174.stdout:(2/7): ceph-prometheus-alerts-18.2.1-0.el9.noar 104 kB/s | 15 kB 00:00 2024-02-19T18:49:04.661 INFO:teuthology.orchestra.run.smithi174.stdout:(3/7): ceph-mgr-dashboard-18.2.1-0.el9.noarch.r 9.2 MB/s | 1.8 MB 00:00 2024-02-19T18:49:04.845 INFO:teuthology.orchestra.run.smithi174.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 35 kB/s | 9.0 kB 00:00 2024-02-19T18:49:04.948 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-02-19T18:49:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:18.2.1-0.el9.x86_64 1/32 2024-02-19T18:49:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:18.2.1-0.el9.x86_64 2/32 2024-02-19T18:49:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el9.x86_64 3/32 2024-02-19T18:49:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:18.2.1-0.el9.x86_64 4/32 2024-02-19T18:49:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:18.2.1-0.el9.x86_64 5/32 2024-02-19T18:49:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:18.2.1-0.el9.x86_64 6/32 2024-02-19T18:49:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:18.2.1-0.el9.x86_64 7/32 2024-02-19T18:49:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 8/32 2024-02-19T18:49:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el9.x86_64 9/32 2024-02-19T18:49:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-02-19T18:49:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:18.2.1-0.el9.x86_64 11/32 2024-02-19T18:49:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:18.2.1-0.el9.x86_64 12/32 2024-02-19T18:49:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:18.2.1-0.el9.x86_64 13/32 2024-02-19T18:49:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-02-19T18:49:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-02-19T18:49:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-02-19T18:49:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-02-19T18:49:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-02-19T18:49:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-02-19T18:49:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-02-19T18:49:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-02-19T18:49:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-02-19T18:49:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-02-19T18:49:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-02-19T18:49:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-02-19T18:49:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-02-19T18:49:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-02-19T18:49:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-02-19T18:49:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:18.2.1-0.el9.x86_64 29/32 2024-02-19T18:49:04.951 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-02-19T18:49:04.951 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:18.2.1-0.el9.x86_64 31/32 2024-02-19T18:49:05.011 INFO:teuthology.orchestra.run.smithi174.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 88 kB/s | 31 kB 00:00 2024-02-19T18:49:05.037 INFO:teuthology.orchestra.run.smithi174.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 96 kB/s | 41 kB 00:00 2024-02-19T18:49:05.287 INFO:teuthology.orchestra.run.smithi174.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 426 kB/s | 188 kB 00:00 2024-02-19T18:49:05.289 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:05.289 INFO:teuthology.orchestra.run.smithi174.stdout:Total 2.4 MB/s | 2.1 MB 00:00 2024-02-19T18:49:05.289 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-02-19T18:49:05.297 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-02-19T18:49:05.297 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-02-19T18:49:05.359 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-02-19T18:49:05.360 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-02-19T18:49:05.654 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-02-19T18:49:05.730 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-19T18:49:05.763 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-19T18:49:05.818 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-19T18:49:05.859 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-19T18:49:05.891 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 5/7 2024-02-19T18:49:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-02-19T18:49:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:06.093 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-02-19T18:49:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:18.2.1-0.el9.x86_64 librbd1-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:06.093 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-19T18:49:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-19T18:49:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: fmt-8.1.1-5.el9.x86_64 2024-02-19T18:49:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-19T18:49:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-19T18:49:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-19T18:49:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-19T18:49:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-19T18:49:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-19T18:49:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-19T18:49:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-19T18:49:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-19T18:49:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-19T18:49:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-19T18:49:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: re2-1:20211101-3.el9.x86_64 2024-02-19T18:49:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-19T18:49:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:06.096 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:49:06.112 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 6/7 2024-02-19T18:49:06.141 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-02-19T18:49:06.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-test 2024-02-19T18:49:06.720 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-02-19T18:49:06.720 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 1/7 2024-02-19T18:49:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2/7 2024-02-19T18:49:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 3/7 2024-02-19T18:49:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-19T18:49:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-19T18:49:06.721 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-19T18:49:06.979 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:25 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:49:07.057 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-19T18:49:07.057 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:07.058 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-02-19T18:49:07.058 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 2024-02-19T18:49:07.058 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2024-02-19T18:49:07.058 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 2024-02-19T18:49:07.058 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-19T18:49:07.058 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-19T18:49:07.058 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-19T18:49:07.058 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-19T18:49:07.058 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:07.058 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:07.145 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:49:07.146 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:07.146 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-19T18:49:07.146 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:07.146 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-19T18:49:07.146 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:18.2.1-0.el9 ceph 40 M 2024-02-19T18:49:07.146 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-19T18:49:07.146 INFO:teuthology.orchestra.run.smithi114.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-19T18:49:07.146 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:18.2.1-0.el9 ceph 165 k 2024-02-19T18:49:07.146 INFO:teuthology.orchestra.run.smithi114.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-19T18:49:07.146 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-19T18:49:07.146 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-19T18:49:07.146 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:07.147 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-19T18:49:07.147 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:07.147 INFO:teuthology.orchestra.run.smithi114.stdout:Install 6 Packages 2024-02-19T18:49:07.147 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:07.147 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 41 M 2024-02-19T18:49:07.147 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 167 M 2024-02-19T18:49:07.147 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-19T18:49:07.258 INFO:teuthology.orchestra.run.smithi114.stdout:(1/6): libcephsqlite-18.2.1-0.el9.x86_64.rpm 1.5 MB/s | 165 kB 00:00 2024-02-19T18:49:07.272 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T18:49:07.817 INFO:teuthology.orchestra.run.smithi114.stdout:(2/6): ceph-test-18.2.1-0.el9.x86_64.rpm 59 MB/s | 40 MB 00:00 2024-02-19T18:49:07.842 INFO:teuthology.orchestra.run.smithi114.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 432 kB/s | 300 kB 00:00 2024-02-19T18:49:07.857 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:58 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:07.875 INFO:teuthology.orchestra.run.smithi114.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 101 kB/s | 63 kB 00:00 2024-02-19T18:49:07.909 INFO:teuthology.orchestra.run.smithi114.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 186 kB 00:00 2024-02-19T18:49:07.976 INFO:teuthology.orchestra.run.smithi114.stdout:(6/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.1 MB/s | 217 kB 00:00 2024-02-19T18:49:07.980 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:07.980 INFO:teuthology.orchestra.run.smithi114.stdout:Total 49 MB/s | 41 MB 00:00 2024-02-19T18:49:07.980 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-19T18:49:08.000 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-19T18:49:08.000 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-19T18:49:08.024 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:08.025 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:08.025 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-02-19T18:49:08.025 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:08.026 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-02-19T18:49:08.026 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el9 ceph-noarch 7.4 M 2024-02-19T18:49:08.026 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-02-19T18:49:08.026 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-19T18:49:08.026 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-19T18:49:08.026 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-19T18:49:08.026 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-19T18:49:08.026 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-19T18:49:08.026 INFO:teuthology.orchestra.run.smithi174.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-19T18:49:08.026 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-19T18:49:08.026 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-19T18:49:08.026 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-19T18:49:08.027 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-19T18:49:08.027 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-19T18:49:08.027 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:08.027 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-02-19T18:49:08.027 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:08.027 INFO:teuthology.orchestra.run.smithi174.stdout:Install 12 Packages 2024-02-19T18:49:08.027 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:08.027 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 37 M 2024-02-19T18:49:08.028 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 209 M 2024-02-19T18:49:08.028 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-02-19T18:49:08.260 INFO:teuthology.orchestra.run.smithi174.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.1-0. 39 MB/s | 7.4 MB 00:00 2024-02-19T18:49:08.366 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-19T18:49:08.367 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-19T18:49:08.394 INFO:teuthology.orchestra.run.smithi174.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 98 kB/s | 32 kB 00:00 2024-02-19T18:49:08.411 INFO:teuthology.orchestra.run.smithi174.stdout:(3/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 116 kB/s | 17 kB 00:00 2024-02-19T18:49:08.528 INFO:teuthology.orchestra.run.smithi174.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 256 kB/s | 34 kB 00:00 2024-02-19T18:49:08.803 INFO:teuthology.orchestra.run.smithi174.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.1 MB/s | 3.0 MB 00:00 2024-02-19T18:49:08.923 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-19T18:49:08.991 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-19T18:49:09.025 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-19T18:49:09.084 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-19T18:49:09.103 INFO:teuthology.orchestra.run.smithi174.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 6.8 MB/s | 4.7 MB 00:00 2024-02-19T18:49:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-19T18:49:09.154 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-02-19T18:49:09.161 INFO:teuthology.orchestra.run.smithi174.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.5 MB/s | 4.8 MB 00:00 2024-02-19T18:49:09.212 INFO:teuthology.orchestra.run.smithi174.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 534 kB/s | 217 kB 00:00 2024-02-19T18:49:09.521 INFO:teuthology.orchestra.run.smithi174.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 802 kB 00:00 2024-02-19T18:49:09.746 INFO:teuthology.orchestra.run.smithi174.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 364 kB/s | 194 kB 00:00 2024-02-19T18:49:09.904 INFO:teuthology.orchestra.run.smithi174.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-02-19T18:49:09.963 INFO:teuthology.orchestra.run.smithi174.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 557 kB/s | 245 kB 00:00 2024-02-19T18:49:09.966 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:09.966 INFO:teuthology.orchestra.run.smithi174.stdout:Total 19 MB/s | 37 MB 00:01 2024-02-19T18:49:09.982 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-02-19T18:49:10.006 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-02-19T18:49:10.007 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-02-19T18:49:10.434 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-02-19T18:49:10.434 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-02-19T18:49:11.143 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-02-19T18:49:11.211 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-19T18:49:11.242 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-19T18:49:11.332 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-19T18:49:11.386 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-19T18:49:11.555 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-19T18:49:11.577 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-19T18:49:11.656 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-19T18:49:12.324 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-19T18:49:12.431 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-19T18:49:12.877 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el9.x86_64 5/6 2024-02-19T18:49:12.962 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-02-19T18:49:13.695 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-19T18:49:14.722 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-19T18:49:14.759 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-02-19T18:49:15.096 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el9.x86_64 6/6 2024-02-19T18:49:15.096 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:18.2.1-0.el9.x86_64 1/6 2024-02-19T18:49:15.096 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:18.2.1-0.el9.x86_64 2/6 2024-02-19T18:49:15.096 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-19T18:49:15.096 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-19T18:49:15.096 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-19T18:49:15.471 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-19T18:49:15.471 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:15.471 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-19T18:49:15.471 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:18.2.1-0.el9.x86_64 jq-1.6-16.el9.x86_64 2024-02-19T18:49:15.471 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:18.2.1-0.el9.x86_64 oniguruma-6.9.6-1.el9.5.x86_64 2024-02-19T18:49:15.471 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-02-19T18:49:15.471 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:15.471 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:49:15.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph 2024-02-19T18:49:16.352 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:35 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:49:16.527 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:49:16.530 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:16.530 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-19T18:49:16.530 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:16.530 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-19T18:49:16.530 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:18.2.1-0.el9 ceph 6.4 k 2024-02-19T18:49:16.530 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-19T18:49:16.530 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:18.2.1-0.el9 ceph 2.1 M 2024-02-19T18:49:16.530 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:18.2.1-0.el9 ceph 1.4 M 2024-02-19T18:49:16.530 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el9 ceph-noarch 242 k 2024-02-19T18:49:16.530 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:18.2.1-0.el9 ceph 4.4 M 2024-02-19T18:49:16.530 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:18.2.1-0.el9 ceph 18 M 2024-02-19T18:49:16.530 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-19T18:49:16.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-19T18:49:16.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-19T18:49:16.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-19T18:49:16.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-19T18:49:16.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-19T18:49:16.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-19T18:49:16.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-19T18:49:16.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-19T18:49:16.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-19T18:49:16.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-19T18:49:16.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-19T18:49:16.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-19T18:49:16.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-19T18:49:16.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-19T18:49:16.531 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-19T18:49:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-19T18:49:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-19T18:49:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-19T18:49:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-19T18:49:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-19T18:49:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-19T18:49:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-19T18:49:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-19T18:49:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-19T18:49:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-19T18:49:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-19T18:49:16.532 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:16.532 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-19T18:49:16.533 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:16.533 INFO:teuthology.orchestra.run.smithi114.stdout:Install 33 Packages 2024-02-19T18:49:16.533 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:16.533 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 30 M 2024-02-19T18:49:16.533 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 110 M 2024-02-19T18:49:16.533 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-19T18:49:16.962 INFO:teuthology.orchestra.run.smithi114.stdout:(1/33): ceph-18.2.1-0.el9.x86_64.rpm 59 kB/s | 6.4 kB 00:00 2024-02-19T18:49:17.020 INFO:teuthology.orchestra.run.smithi114.stdout:(2/33): ceph-mds-18.2.1-0.el9.x86_64.rpm 13 MB/s | 2.1 MB 00:00 2024-02-19T18:49:17.062 INFO:teuthology.orchestra.run.smithi114.stdout:(3/33): ceph-mgr-18.2.1-0.el9.x86_64.rpm 6.9 MB/s | 1.4 MB 00:00 2024-02-19T18:49:17.088 INFO:teuthology.orchestra.run.smithi114.stdout:(4/33): ceph-mgr-modules-core-18.2.1-0.el9.noar 9.5 MB/s | 242 kB 00:00 2024-02-19T18:49:17.196 INFO:teuthology.orchestra.run.smithi114.stdout:(5/33): ceph-mon-18.2.1-0.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2024-02-19T18:49:17.348 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-02-19T18:49:17.348 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 1/12 2024-02-19T18:49:17.348 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-19T18:49:17.348 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-19T18:49:17.348 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-19T18:49:17.348 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-19T18:49:17.348 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-19T18:49:17.348 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-19T18:49:17.348 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-19T18:49:17.348 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-19T18:49:17.348 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-19T18:49:17.349 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-19T18:49:17.413 INFO:teuthology.orchestra.run.smithi114.stdout:(6/33): ceph-osd-18.2.1-0.el9.x86_64.rpm 45 MB/s | 18 MB 00:00 2024-02-19T18:49:17.538 INFO:teuthology.orchestra.run.smithi114.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 320 kB/s | 144 kB 00:00 2024-02-19T18:49:17.647 INFO:teuthology.orchestra.run.smithi114.stdout:(8/33): python3-markupsafe-1.1.1-12.el9.x86_64. 70 kB/s | 32 kB 00:00 2024-02-19T18:49:17.664 INFO:teuthology.orchestra.run.smithi114.stdout:(9/33): python3-pytz-2021.1-5.el9.noarch.rpm 189 kB/s | 47 kB 00:00 2024-02-19T18:49:17.714 INFO:teuthology.orchestra.run.smithi114.stdout:(10/33): python3-toml-0.10.2-6.el9.noarch.rpm 219 kB/s | 39 kB 00:00 2024-02-19T18:49:17.773 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-19T18:49:17.774 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:17.774 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-02-19T18:49:17.774 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarch 2024-02-19T18:49:17.774 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-19T18:49:17.774 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-19T18:49:17.774 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-19T18:49:17.774 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-19T18:49:17.774 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-19T18:49:17.774 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-19T18:49:17.774 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-19T18:49:17.774 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-19T18:49:17.774 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-19T18:49:17.774 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-19T18:49:17.774 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-19T18:49:17.775 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:17.775 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:18.056 INFO:teuthology.orchestra.run.smithi114.stdout:(11/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 589 kB/s | 241 kB 00:00 2024-02-19T18:49:18.178 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-rook 2024-02-19T18:49:18.767 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:09 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:18.933 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:18.934 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:18.934 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-02-19T18:49:18.935 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:18.935 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-02-19T18:49:18.935 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el9 ceph-noarch 50 k 2024-02-19T18:49:18.935 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-02-19T18:49:18.935 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-19T18:49:18.935 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-19T18:49:18.935 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-19T18:49:18.935 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-19T18:49:18.935 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-19T18:49:18.935 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-19T18:49:18.935 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-19T18:49:18.935 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-19T18:49:18.935 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-19T18:49:18.936 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-19T18:49:18.936 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-19T18:49:18.936 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-19T18:49:18.936 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:18.936 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-02-19T18:49:18.936 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:18.936 INFO:teuthology.orchestra.run.smithi174.stdout:Install 13 Packages 2024-02-19T18:49:18.936 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:18.937 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 2.0 M 2024-02-19T18:49:18.937 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 26 M 2024-02-19T18:49:18.937 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-02-19T18:49:19.274 INFO:teuthology.orchestra.run.smithi174.stdout:(1/13): ceph-mgr-rook-18.2.1-0.el9.noarch.rpm 870 kB/s | 50 kB 00:00 2024-02-19T18:49:19.356 INFO:teuthology.orchestra.run.smithi114.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 63 kB/s | 103 kB 00:01 2024-02-19T18:49:19.424 INFO:teuthology.orchestra.run.smithi174.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 117 kB/s | 24 kB 00:00 2024-02-19T18:49:19.448 INFO:teuthology.orchestra.run.smithi114.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 89 kB/s | 124 kB 00:01 2024-02-19T18:49:19.458 INFO:teuthology.orchestra.run.smithi174.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 75 kB/s | 18 kB 00:00 2024-02-19T18:49:19.634 INFO:teuthology.orchestra.run.smithi174.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 634 kB/s | 132 kB 00:00 2024-02-19T18:49:19.650 INFO:teuthology.orchestra.run.smithi114.stdout:(14/33): python3-cryptography-36.0.1-4.el9.x86_ 590 kB/s | 1.1 MB 00:01 2024-02-19T18:49:19.674 INFO:teuthology.orchestra.run.smithi114.stdout:(15/33): python3-requests-2.25.1-8.el9.noarch.r 356 kB/s | 113 kB 00:00 2024-02-19T18:49:19.675 INFO:teuthology.orchestra.run.smithi174.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 491 kB/s | 174 kB 00:00 2024-02-19T18:49:19.700 INFO:teuthology.orchestra.run.smithi174.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 870 kB/s | 211 kB 00:00 2024-02-19T18:49:19.726 INFO:teuthology.orchestra.run.smithi174.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 465 kB/s | 43 kB 00:00 2024-02-19T18:49:19.807 INFO:teuthology.orchestra.run.smithi114.stdout:(16/33): python3-urllib3-1.26.5-5.el9.noarch.rp 522 kB/s | 187 kB 00:00 2024-02-19T18:49:20.043 INFO:teuthology.orchestra.run.smithi174.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 88 kB/s | 32 kB 00:00 2024-02-19T18:49:20.049 INFO:teuthology.orchestra.run.smithi114.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 109 kB/s | 43 kB 00:00 2024-02-19T18:49:20.068 INFO:teuthology.orchestra.run.smithi174.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 41 kB/s | 15 kB 00:00 2024-02-19T18:49:20.150 INFO:teuthology.orchestra.run.smithi114.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 361 kB/s | 172 kB 00:00 2024-02-19T18:49:20.175 INFO:teuthology.orchestra.run.smithi114.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 85 kB/s | 11 kB 00:00 2024-02-19T18:49:20.250 INFO:teuthology.orchestra.run.smithi114.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-02-19T18:49:20.275 INFO:teuthology.orchestra.run.smithi114.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-02-19T18:49:20.326 INFO:teuthology.orchestra.run.smithi174.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 229 kB/s | 59 kB 00:00 2024-02-19T18:49:20.351 INFO:teuthology.orchestra.run.smithi114.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-02-19T18:49:20.376 INFO:teuthology.orchestra.run.smithi114.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-02-19T18:49:20.401 INFO:teuthology.orchestra.run.smithi114.stdout:(24/33): python3-cherrypy-18.6.1-2.el9.noarch.r 604 kB/s | 358 kB 00:00 2024-02-19T18:49:20.401 INFO:teuthology.orchestra.run.smithi174.stdout:(11/13): python3-google-auth-2.27.0-1.el9.noarc 300 kB/s | 203 kB 00:00 2024-02-19T18:49:20.460 INFO:teuthology.orchestra.run.smithi114.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 425 kB/s | 46 kB 00:00 2024-02-19T18:49:20.569 INFO:teuthology.orchestra.run.smithi114.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-02-19T18:49:20.593 INFO:teuthology.orchestra.run.smithi174.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 336 kB/s | 90 kB 00:00 2024-02-19T18:49:20.727 INFO:teuthology.orchestra.run.smithi114.stdout:(27/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 569 kB/s | 90 kB 00:00 2024-02-19T18:49:20.752 INFO:teuthology.orchestra.run.smithi114.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 776 kB/s | 272 kB 00:00 2024-02-19T18:49:20.986 INFO:teuthology.orchestra.run.smithi114.stdout:(29/33): python3-webob-1.8.7-6.el9.noarch.rpm 985 kB/s | 230 kB 00:00 2024-02-19T18:49:21.044 INFO:teuthology.orchestra.run.smithi114.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 113 kB/s | 36 kB 00:00 2024-02-19T18:49:21.070 INFO:teuthology.orchestra.run.smithi114.stdout:(31/33): python3-more-itertools-8.12.0-2.el9.no 114 kB/s | 79 kB 00:00 2024-02-19T18:49:21.085 INFO:teuthology.orchestra.run.smithi174.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.0 MB/s | 1.0 MB 00:01 2024-02-19T18:49:21.088 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:21.089 INFO:teuthology.orchestra.run.smithi174.stdout:Total 974 kB/s | 2.0 MB 00:02 2024-02-19T18:49:21.089 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-02-19T18:49:21.103 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-02-19T18:49:21.103 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-02-19T18:49:21.136 INFO:teuthology.orchestra.run.smithi114.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-02-19T18:49:21.209 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-02-19T18:49:21.211 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-02-19T18:49:21.462 INFO:teuthology.orchestra.run.smithi114.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 898 kB/s | 427 kB 00:00 2024-02-19T18:49:21.469 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:21.469 INFO:teuthology.orchestra.run.smithi114.stdout:Total 6.0 MB/s | 30 MB 00:04 2024-02-19T18:49:21.469 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-19T18:49:21.509 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-19T18:49:21.509 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-19T18:49:21.727 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-02-19T18:49:21.879 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-19T18:49:21.880 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-19T18:49:21.889 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-19T18:49:21.944 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-19T18:49:21.999 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-19T18:49:22.029 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-19T18:49:22.070 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-19T18:49:22.163 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-19T18:49:22.263 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-19T18:49:22.312 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-19T18:49:23.031 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-19T18:49:23.068 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-19T18:49:23.102 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-19T18:49:23.106 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-19T18:49:23.149 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-19T18:49:23.164 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-19T18:49:23.172 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-02-19T18:49:23.188 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-19T18:49:23.221 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-19T18:49:23.320 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-19T18:49:23.411 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-19T18:49:23.457 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-19T18:49:23.528 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-19T18:49:23.582 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-19T18:49:23.621 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-19T18:49:23.677 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-19T18:49:23.743 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-19T18:49:23.924 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-19T18:49:23.975 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-19T18:49:24.006 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-19T18:49:24.040 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-19T18:49:24.057 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-02-19T18:49:24.058 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el9.noarch 1/13 2024-02-19T18:49:24.058 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-19T18:49:24.058 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-19T18:49:24.058 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-19T18:49:24.058 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-19T18:49:24.058 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-19T18:49:24.058 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-19T18:49:24.058 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-19T18:49:24.058 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-19T18:49:24.058 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-19T18:49:24.058 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-19T18:49:24.058 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-19T18:49:24.070 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-19T18:49:24.146 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-19T18:49:24.180 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-19T18:49:24.214 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-19T18:49:24.255 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-19T18:49:24.289 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-19T18:49:24.324 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-19T18:49:24.412 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-19T18:49:24.428 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-19T18:49:24.428 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:24.428 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-02-19T18:49:24.428 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook-2:18.2.1-0.el9.noarch 2024-02-19T18:49:24.428 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-19T18:49:24.428 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-19T18:49:24.428 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-19T18:49:24.428 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-19T18:49:24.428 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-19T18:49:24.429 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-19T18:49:24.429 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-19T18:49:24.429 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-19T18:49:24.429 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-19T18:49:24.429 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-19T18:49:24.429 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-19T18:49:24.429 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-19T18:49:24.429 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:24.429 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:24.446 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-19T18:49:24.517 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-19T18:49:24.656 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-19T18:49:24.682 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T18:49:24.772 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-19T18:49:24.935 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 28/33 2024-02-19T18:49:24.959 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-02-19T18:49:24.985 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-02-19T18:49:24.985 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:49:24.985 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-19T18:49:24.985 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-19T18:49:24.985 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-19T18:49:24.985 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:25.277 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:16 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:25.454 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:25.455 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-02-19T18:49:25.455 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:25.455 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-02-19T18:49:25.455 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el9 ceph-noarch 132 k 2024-02-19T18:49:25.455 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-02-19T18:49:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-02-19T18:49:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-19T18:49:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-19T18:49:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-19T18:49:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-19T18:49:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:25.456 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-02-19T18:49:25.456 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:25.456 INFO:teuthology.orchestra.run.smithi174.stdout:Install 6 Packages 2024-02-19T18:49:25.456 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:25.457 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 6.8 M 2024-02-19T18:49:25.457 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 33 M 2024-02-19T18:49:25.457 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-02-19T18:49:25.811 INFO:teuthology.orchestra.run.smithi174.stdout:(1/6): ceph-mgr-cephadm-18.2.1-0.el9.noarch.rpm 1.0 MB/s | 132 kB 00:00 2024-02-19T18:49:26.086 INFO:teuthology.orchestra.run.smithi174.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 560 kB/s | 227 kB 00:00 2024-02-19T18:49:26.462 INFO:teuthology.orchestra.run.smithi174.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 7.4 MB/s | 5.8 MB 00:00 2024-02-19T18:49:26.553 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-02-19T18:49:26.581 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-02-19T18:49:26.581 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:49:26.581 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-19T18:49:26.581 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-19T18:49:26.581 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-19T18:49:26.581 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:27.028 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-02-19T18:49:27.052 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-02-19T18:49:27.052 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:49:27.052 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-19T18:49:27.052 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-19T18:49:27.052 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-19T18:49:27.052 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:27.254 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-02-19T18:49:27.277 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-02-19T18:49:27.277 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:49:27.277 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-19T18:49:27.277 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-19T18:49:27.278 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-19T18:49:27.278 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:27.343 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:18.2.1-0.el9.x86_64 33/33 2024-02-19T18:49:27.370 INFO:teuthology.orchestra.run.smithi174.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 46 kB/s | 58 kB 00:01 2024-02-19T18:49:27.821 INFO:teuthology.orchestra.run.smithi174.stdout:(5/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 273 kB/s | 548 kB 00:02 2024-02-19T18:49:27.837 INFO:teuthology.orchestra.run.smithi174.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 37 kB/s | 51 kB 00:01 2024-02-19T18:49:27.842 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:27.842 INFO:teuthology.orchestra.run.smithi174.stdout:Total 2.8 MB/s | 6.8 MB 00:02 2024-02-19T18:49:27.842 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-02-19T18:49:27.852 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-02-19T18:49:27.852 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-02-19T18:49:27.945 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-02-19T18:49:27.945 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-02-19T18:49:28.247 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-02-19T18:49:28.420 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-19T18:49:28.464 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-02-19T18:49:28.985 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-19T18:49:29.066 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-19T18:49:29.126 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-19T18:49:29.150 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-02-19T18:49:29.169 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-2:18.2.1-0.el9.x86_64 33/33 2024-02-19T18:49:29.169 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:18.2.1-0.el9.x86_64 1/33 2024-02-19T18:49:29.169 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:18.2.1-0.el9.x86_64 2/33 2024-02-19T18:49:29.169 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:18.2.1-0.el9.x86_64 3/33 2024-02-19T18:49:29.169 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:18.2.1-0.el9.x86_64 4/33 2024-02-19T18:49:29.169 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:18.2.1-0.el9.x86_64 5/33 2024-02-19T18:49:29.169 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 6/33 2024-02-19T18:49:29.169 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-19T18:49:29.169 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-19T18:49:29.169 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-19T18:49:29.169 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-19T18:49:29.169 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-19T18:49:29.170 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-19T18:49:29.170 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-19T18:49:29.170 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-19T18:49:29.170 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-19T18:49:29.170 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-19T18:49:29.170 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-19T18:49:29.170 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-19T18:49:29.170 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-19T18:49:29.170 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-19T18:49:29.170 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-19T18:49:29.170 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-19T18:49:29.170 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-19T18:49:29.170 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-19T18:49:29.170 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-19T18:49:29.170 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-19T18:49:29.170 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-19T18:49:29.171 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-19T18:49:29.171 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-19T18:49:29.171 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-19T18:49:29.171 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-19T18:49:29.171 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-19T18:49:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-02-19T18:49:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 1/6 2024-02-19T18:49:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-19T18:49:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-19T18:49:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-02-19T18:49:30.021 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-19T18:49:30.150 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-19T18:49:30.150 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:30.150 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-19T18:49:30.150 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:30.150 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:30.150 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:30.150 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 2024-02-19T18:49:30.150 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:30.150 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:30.150 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-19T18:49:30.150 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-19T18:49:30.150 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-19T18:49:30.151 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-19T18:49:30.151 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-19T18:49:30.151 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-19T18:49:30.151 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-19T18:49:30.151 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-19T18:49:30.151 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-19T18:49:30.151 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-19T18:49:30.151 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-19T18:49:30.151 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-19T18:49:30.151 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-19T18:49:30.151 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-19T18:49:30.151 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-19T18:49:30.151 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply-3.11-14.el9.noarch 2024-02-19T18:49:30.151 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-19T18:49:30.151 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-19T18:49:30.151 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-19T18:49:30.152 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-19T18:49:30.152 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-19T18:49:30.152 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-19T18:49:30.152 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-19T18:49:30.152 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-19T18:49:30.152 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-19T18:49:30.152 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-19T18:49:30.152 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-19T18:49:30.152 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:30.152 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:49:30.353 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-19T18:49:30.353 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:30.353 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-02-19T18:49:30.353 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 2024-02-19T18:49:30.353 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-02-19T18:49:30.353 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-19T18:49:30.353 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-19T18:49:30.353 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-19T18:49:30.353 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-19T18:49:30.353 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:30.353 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:30.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-base 2024-02-19T18:49:30.619 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-fuse 2024-02-19T18:49:31.096 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:50 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:49:31.204 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:22 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:31.209 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-base-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:49:31.260 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:49:31.261 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-19T18:49:31.261 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:49:31.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install cephadm 2024-02-19T18:49:31.367 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:31.368 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:31.368 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-02-19T18:49:31.368 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:31.368 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-02-19T18:49:31.368 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse x86_64 2:18.2.1-0.el9 ceph 839 k 2024-02-19T18:49:31.368 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-02-19T18:49:31.369 INFO:teuthology.orchestra.run.smithi174.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-19T18:49:31.369 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:31.369 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-02-19T18:49:31.369 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:31.369 INFO:teuthology.orchestra.run.smithi174.stdout:Install 2 Packages 2024-02-19T18:49:31.369 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:31.369 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 918 k 2024-02-19T18:49:31.369 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 2.7 M 2024-02-19T18:49:31.369 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-02-19T18:49:31.449 INFO:teuthology.orchestra.run.smithi174.stdout:(1/2): ceph-fuse-18.2.1-0.el9.x86_64.rpm 10 MB/s | 839 kB 00:00 2024-02-19T18:49:31.707 INFO:teuthology.orchestra.run.smithi174.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 233 kB/s | 79 kB 00:00 2024-02-19T18:49:31.708 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:31.709 INFO:teuthology.orchestra.run.smithi174.stdout:Total 2.6 MB/s | 918 kB 00:00 2024-02-19T18:49:31.709 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-02-19T18:49:31.729 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-02-19T18:49:31.729 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-02-19T18:49:31.804 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-02-19T18:49:31.805 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-02-19T18:49:31.905 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:50 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:49:31.994 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-02-19T18:49:32.068 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:49:32.069 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:32.069 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-02-19T18:49:32.069 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:32.069 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-19T18:49:32.069 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:18.2.1-0.el9 ceph-noarch 221 k 2024-02-19T18:49:32.069 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:32.069 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-19T18:49:32.069 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:32.069 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-02-19T18:49:32.069 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:32.070 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 221 k 2024-02-19T18:49:32.070 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 213 k 2024-02-19T18:49:32.070 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-19T18:49:32.129 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-19T18:49:32.147 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm-18.2.1-0.el9.noarch.rpm 2.8 MB/s | 221 kB 00:00 2024-02-19T18:49:32.148 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:32.148 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.8 MB/s | 221 kB 00:00 2024-02-19T18:49:32.148 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-19T18:49:32.153 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-19T18:49:32.153 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-19T18:49:32.156 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-19T18:49:32.156 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-19T18:49:32.206 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-02-19T18:49:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-19T18:49:32.691 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-02-19T18:49:32.777 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-02-19T18:49:32.935 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-02-19T18:49:32.935 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-fuse-2:18.2.1-0.el9.x86_64 1/2 2024-02-19T18:49:33.178 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-02-19T18:49:33.259 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-19T18:49:33.259 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:33.259 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-02-19T18:49:33.259 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse-2:18.2.1-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-19T18:49:33.259 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:33.259 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:33.475 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-02-19T18:49:33.475 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:33.476 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-19T18:49:33.476 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:18.2.1-0.el9.noarch 2024-02-19T18:49:33.476 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:33.476 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:49:33.566 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install librados-devel 2024-02-19T18:49:33.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T18:49:34.164 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:25 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:34.256 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:53 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:49:34.331 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:34.332 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:34.332 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-02-19T18:49:34.332 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:34.332 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-02-19T18:49:34.332 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel x86_64 2:18.2.1-0.el9 ceph 127 k 2024-02-19T18:49:34.332 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:34.332 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-02-19T18:49:34.332 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:34.332 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-02-19T18:49:34.332 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:34.332 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 127 k 2024-02-19T18:49:34.332 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 456 k 2024-02-19T18:49:34.332 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-02-19T18:49:34.403 INFO:teuthology.orchestra.run.smithi174.stdout:librados-devel-18.2.1-0.el9.x86_64.rpm 1.8 MB/s | 127 kB 00:00 2024-02-19T18:49:34.403 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:34.404 INFO:teuthology.orchestra.run.smithi174.stdout:Total 1.7 MB/s | 127 kB 00:00 2024-02-19T18:49:34.404 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-02-19T18:49:34.409 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-02-19T18:49:34.409 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-02-19T18:49:34.423 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:49:34.424 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-02-19T18:49:34.424 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:34.424 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-19T18:49:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el9 ceph 142 k 2024-02-19T18:49:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:34.424 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-19T18:49:34.424 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:34.424 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-02-19T18:49:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:34.424 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 142 k 2024-02-19T18:49:34.424 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 395 k 2024-02-19T18:49:34.425 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-19T18:49:34.464 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-02-19T18:49:34.467 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-02-19T18:49:34.523 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-immutable-object-cache-18.2.1-0.el9.x86_64 1.4 MB/s | 142 kB 00:00 2024-02-19T18:49:34.523 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:34.524 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.4 MB/s | 142 kB 00:00 2024-02-19T18:49:34.524 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-19T18:49:34.531 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-19T18:49:34.532 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-19T18:49:34.582 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-19T18:49:34.583 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-19T18:49:34.606 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-02-19T18:49:34.692 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:34.720 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-19T18:49:34.779 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:34.803 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:34.803 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:49:34.804 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-19T18:49:34.804 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:35.173 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:35.509 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:35.509 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:35.509 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-02-19T18:49:35.509 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:35.510 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:35.510 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:35.780 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install libcephfs2 2024-02-19T18:49:35.783 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:35.783 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:35.783 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-19T18:49:35.783 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:35.783 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:35.783 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:49:36.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr 2024-02-19T18:49:36.374 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:27 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:36.520 INFO:teuthology.orchestra.run.smithi174.stdout:Package libcephfs2-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:49:36.572 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:36.573 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-02-19T18:49:36.573 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:36.616 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:55 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:49:36.639 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install libcephfs-devel 2024-02-19T18:49:36.731 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-mgr-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:49:36.782 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:49:36.783 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-19T18:49:36.783 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:49:36.857 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T18:49:37.228 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:28 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:37.394 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:37.395 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:37.395 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-02-19T18:49:37.395 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:37.395 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-02-19T18:49:37.395 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel x86_64 2:18.2.1-0.el9 ceph 31 k 2024-02-19T18:49:37.395 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:37.395 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-02-19T18:49:37.395 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:37.395 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-02-19T18:49:37.395 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:37.395 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 31 k 2024-02-19T18:49:37.395 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 139 k 2024-02-19T18:49:37.395 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-02-19T18:49:37.432 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:56 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:49:37.464 INFO:teuthology.orchestra.run.smithi174.stdout:libcephfs-devel-18.2.1-0.el9.x86_64.rpm 451 kB/s | 31 kB 00:00 2024-02-19T18:49:37.464 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:37.465 INFO:teuthology.orchestra.run.smithi174.stdout:Total 442 kB/s | 31 kB 00:00 2024-02-19T18:49:37.465 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-02-19T18:49:37.468 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-02-19T18:49:37.468 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-02-19T18:49:37.492 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-02-19T18:49:37.493 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-02-19T18:49:37.573 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-02-19T18:49:37.597 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:49:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:37.598 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-19T18:49:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-19T18:49:37.598 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el9 ceph-noarch 1.8 M 2024-02-19T18:49:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-19T18:49:37.598 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el9 ceph-noarch 23 k 2024-02-19T18:49:37.598 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el9 ceph-noarch 15 k 2024-02-19T18:49:37.598 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-19T18:49:37.598 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-19T18:49:37.598 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-19T18:49:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-02-19T18:49:37.598 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-19T18:49:37.598 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-19T18:49:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:Install 7 Packages 2024-02-19T18:49:37.599 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 2.1 M 2024-02-19T18:49:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 8.5 M 2024-02-19T18:49:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-19T18:49:37.670 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:38.105 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:38.174 INFO:teuthology.orchestra.run.smithi114.stdout:(1/7): ceph-grafana-dashboards-18.2.1-0.el9.noa 207 kB/s | 23 kB 00:00 2024-02-19T18:49:38.199 INFO:teuthology.orchestra.run.smithi114.stdout:(2/7): ceph-prometheus-alerts-18.2.1-0.el9.noar 106 kB/s | 15 kB 00:00 2024-02-19T18:49:38.267 INFO:teuthology.orchestra.run.smithi114.stdout:(3/7): ceph-mgr-dashboard-18.2.1-0.el9.noarch.r 8.6 MB/s | 1.8 MB 00:00 2024-02-19T18:49:38.383 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:38.383 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:38.383 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-02-19T18:49:38.383 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:38.383 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:38.383 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:38.392 INFO:teuthology.orchestra.run.smithi114.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 42 kB/s | 9.0 kB 00:00 2024-02-19T18:49:38.475 INFO:teuthology.orchestra.run.smithi114.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 148 kB/s | 41 kB 00:00 2024-02-19T18:49:38.500 INFO:teuthology.orchestra.run.smithi114.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 164 kB/s | 31 kB 00:00 2024-02-19T18:49:38.559 INFO:teuthology.orchestra.run.smithi114.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.1 MB/s | 188 kB 00:00 2024-02-19T18:49:38.562 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install librados2 2024-02-19T18:49:38.563 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:38.563 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.1 MB/s | 2.1 MB 00:00 2024-02-19T18:49:38.564 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-19T18:49:38.570 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-19T18:49:38.570 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-19T18:49:38.633 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-19T18:49:38.634 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-19T18:49:38.932 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-19T18:49:38.994 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-19T18:49:39.019 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-19T18:49:39.074 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-19T18:49:39.106 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-19T18:49:39.134 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 5/7 2024-02-19T18:49:39.152 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:30 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:39.269 INFO:teuthology.orchestra.run.smithi174.stdout:Package librados2-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:49:39.320 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:39.321 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-02-19T18:49:39.321 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:39.370 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 6/7 2024-02-19T18:49:39.389 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install librbd1 2024-02-19T18:49:39.396 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-02-19T18:49:39.924 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-02-19T18:49:39.924 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 1/7 2024-02-19T18:49:39.924 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2/7 2024-02-19T18:49:39.924 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 3/7 2024-02-19T18:49:39.924 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-19T18:49:39.924 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-19T18:49:39.924 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-19T18:49:39.972 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:30 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:40.086 INFO:teuthology.orchestra.run.smithi174.stdout:Package librbd1-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:49:40.139 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:40.140 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-02-19T18:49:40.140 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:40.205 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-rados 2024-02-19T18:49:40.234 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-19T18:49:40.234 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:40.234 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-19T18:49:40.234 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 2024-02-19T18:49:40.234 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2024-02-19T18:49:40.235 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 2024-02-19T18:49:40.235 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-19T18:49:40.235 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-19T18:49:40.235 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-19T18:49:40.235 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-19T18:49:40.235 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:40.235 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:49:40.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T18:49:40.778 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:31 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:40.891 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-rados-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:49:40.943 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:40.944 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-02-19T18:49:40.944 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:41.019 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-rgw 2024-02-19T18:49:41.022 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:00 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:49:41.187 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:49:41.188 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:41.189 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-19T18:49:41.189 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:41.189 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-19T18:49:41.189 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el9 ceph-noarch 7.4 M 2024-02-19T18:49:41.189 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-19T18:49:41.189 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-19T18:49:41.189 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-19T18:49:41.189 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-19T18:49:41.189 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-19T18:49:41.189 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-19T18:49:41.189 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-19T18:49:41.189 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-19T18:49:41.189 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-19T18:49:41.189 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-19T18:49:41.189 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-19T18:49:41.190 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-19T18:49:41.190 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:41.190 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-19T18:49:41.190 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:41.190 INFO:teuthology.orchestra.run.smithi114.stdout:Install 12 Packages 2024-02-19T18:49:41.190 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:41.190 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 37 M 2024-02-19T18:49:41.190 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 209 M 2024-02-19T18:49:41.191 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-19T18:49:41.541 INFO:teuthology.orchestra.run.smithi114.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.1-0. 37 MB/s | 7.4 MB 00:00 2024-02-19T18:49:41.566 INFO:teuthology.orchestra.run.smithi114.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 139 kB/s | 32 kB 00:00 2024-02-19T18:49:41.591 INFO:teuthology.orchestra.run.smithi114.stdout:(3/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 343 kB/s | 17 kB 00:00 2024-02-19T18:49:41.602 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:32 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:41.625 INFO:teuthology.orchestra.run.smithi114.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 581 kB/s | 34 kB 00:00 2024-02-19T18:49:41.716 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-rgw-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:49:41.767 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:41.768 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-02-19T18:49:41.768 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:41.837 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-cephfs 2024-02-19T18:49:42.025 INFO:teuthology.orchestra.run.smithi114.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2024-02-19T18:49:42.092 INFO:teuthology.orchestra.run.smithi114.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 4.8 MB 00:00 2024-02-19T18:49:42.117 INFO:teuthology.orchestra.run.smithi114.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-02-19T18:49:42.423 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:33 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:42.536 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-cephfs-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:49:42.568 INFO:teuthology.orchestra.run.smithi114.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 802 kB 00:00 2024-02-19T18:49:42.587 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:42.588 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-02-19T18:49:42.588 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:42.655 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-rbd 2024-02-19T18:49:42.668 INFO:teuthology.orchestra.run.smithi114.stdout:(9/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.2 MB/s | 3.0 MB 00:01 2024-02-19T18:49:42.868 INFO:teuthology.orchestra.run.smithi114.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-02-19T18:49:42.893 INFO:teuthology.orchestra.run.smithi114.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 601 kB/s | 194 kB 00:00 2024-02-19T18:49:42.977 INFO:teuthology.orchestra.run.smithi114.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 801 kB/s | 245 kB 00:00 2024-02-19T18:49:42.979 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:42.980 INFO:teuthology.orchestra.run.smithi114.stdout:Total 21 MB/s | 37 MB 00:01 2024-02-19T18:49:43.001 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-19T18:49:43.025 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-19T18:49:43.025 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-19T18:49:43.236 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:34 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:43.350 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-rbd-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:49:43.402 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:43.403 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-02-19T18:49:43.403 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:43.450 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-19T18:49:43.450 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-19T18:49:43.478 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install rbd-fuse 2024-02-19T18:49:44.061 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:35 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:44.148 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-19T18:49:44.216 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-19T18:49:44.227 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:44.228 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:44.228 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-02-19T18:49:44.228 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:44.228 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-02-19T18:49:44.228 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse x86_64 2:18.2.1-0.el9 ceph 86 k 2024-02-19T18:49:44.228 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:44.228 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-02-19T18:49:44.228 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:44.228 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-02-19T18:49:44.228 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:44.229 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 86 k 2024-02-19T18:49:44.229 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 226 k 2024-02-19T18:49:44.229 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-02-19T18:49:44.248 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-19T18:49:44.300 INFO:teuthology.orchestra.run.smithi174.stdout:rbd-fuse-18.2.1-0.el9.x86_64.rpm 1.2 MB/s | 86 kB 00:00 2024-02-19T18:49:44.300 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:44.301 INFO:teuthology.orchestra.run.smithi174.stdout:Total 1.2 MB/s | 86 kB 00:00 2024-02-19T18:49:44.301 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-02-19T18:49:44.307 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-02-19T18:49:44.308 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-02-19T18:49:44.341 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-19T18:49:44.362 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-02-19T18:49:44.362 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-02-19T18:49:44.376 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-19T18:49:44.509 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-02-19T18:49:44.554 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-19T18:49:44.583 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-19T18:49:44.606 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:44.668 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-19T18:49:45.044 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:45.357 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:45.357 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:45.357 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-02-19T18:49:45.357 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:45.357 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:45.358 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:45.367 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-19T18:49:45.487 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-19T18:49:45.602 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install rbd-mirror 2024-02-19T18:49:46.191 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:37 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:46.361 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:46.361 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:46.362 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-02-19T18:49:46.362 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:46.362 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-02-19T18:49:46.362 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-02-19T18:49:46.362 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:46.362 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-02-19T18:49:46.362 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:46.362 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-02-19T18:49:46.362 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:46.363 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 3.0 M 2024-02-19T18:49:46.363 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 12 M 2024-02-19T18:49:46.363 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-02-19T18:49:46.452 INFO:teuthology.orchestra.run.smithi174.stdout:rbd-mirror-18.2.1-0.el9.x86_64.rpm 33 MB/s | 3.0 MB 00:00 2024-02-19T18:49:46.453 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:46.453 INFO:teuthology.orchestra.run.smithi174.stdout:Total 33 MB/s | 3.0 MB 00:00 2024-02-19T18:49:46.453 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-02-19T18:49:46.462 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-02-19T18:49:46.462 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-02-19T18:49:46.534 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-02-19T18:49:46.534 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-02-19T18:49:46.725 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-19T18:49:46.938 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-02-19T18:49:47.003 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:47.027 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:47.027 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:49:47.027 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-19T18:49:47.027 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-19T18:49:47.027 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-19T18:49:47.027 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:47.753 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-19T18:49:47.782 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-02-19T18:49:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:48.241 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-02-19T18:49:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:48.241 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:48.241 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:48.481 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install rbd-nbd 2024-02-19T18:49:49.079 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:40 ago on Mon 19 Feb 2024 06:48:09 PM UTC. 2024-02-19T18:49:49.247 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-02-19T18:49:49.247 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:49.247 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-02-19T18:49:49.247 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:49.247 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-02-19T18:49:49.247 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd x86_64 2:18.2.1-0.el9 ceph 171 k 2024-02-19T18:49:49.248 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:49.248 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-02-19T18:49:49.248 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-02-19T18:49:49.248 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-02-19T18:49:49.248 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:49.248 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 171 k 2024-02-19T18:49:49.248 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 494 k 2024-02-19T18:49:49.249 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-02-19T18:49:49.314 INFO:teuthology.orchestra.run.smithi174.stdout:rbd-nbd-18.2.1-0.el9.x86_64.rpm 2.6 MB/s | 171 kB 00:00 2024-02-19T18:49:49.314 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:49.314 INFO:teuthology.orchestra.run.smithi174.stdout:Total 2.5 MB/s | 171 kB 00:00 2024-02-19T18:49:49.314 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-02-19T18:49:49.322 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-02-19T18:49:49.322 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-02-19T18:49:49.378 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-02-19T18:49:49.378 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-02-19T18:49:49.526 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-02-19T18:49:49.627 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:50.090 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:50.362 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-02-19T18:49:50.362 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 1/12 2024-02-19T18:49:50.362 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-19T18:49:50.362 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-19T18:49:50.362 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-19T18:49:50.362 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-19T18:49:50.362 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-19T18:49:50.362 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-19T18:49:50.362 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-19T18:49:50.363 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-19T18:49:50.363 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-19T18:49:50.363 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-19T18:49:50.389 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:49:50.389 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:50.389 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-02-19T18:49:50.389 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd-2:18.2.1-0.el9.x86_64 2024-02-19T18:49:50.389 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:49:50.389 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-02-19T18:49:50.642 DEBUG:teuthology.parallel:result is None 2024-02-19T18:49:50.772 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-19T18:49:50.773 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:50.773 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-19T18:49:50.773 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarch 2024-02-19T18:49:50.773 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-19T18:49:50.773 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-19T18:49:50.773 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-19T18:49:50.773 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-19T18:49:50.773 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-19T18:49:50.773 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-19T18:49:50.773 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-19T18:49:50.773 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-19T18:49:50.773 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-19T18:49:50.773 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-19T18:49:50.774 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-19T18:49:50.774 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:50.774 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:49:51.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-rook 2024-02-19T18:49:51.743 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:10 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:49:51.910 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:49:51.911 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:51.911 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-19T18:49:51.911 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:51.911 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-19T18:49:51.911 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el9 ceph-noarch 50 k 2024-02-19T18:49:51.911 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-19T18:49:51.911 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-19T18:49:51.911 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-19T18:49:51.912 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-19T18:49:51.912 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-19T18:49:51.912 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-19T18:49:51.912 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-19T18:49:51.912 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-19T18:49:51.912 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-19T18:49:51.912 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-19T18:49:51.912 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-19T18:49:51.912 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-19T18:49:51.912 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-19T18:49:51.912 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:51.912 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-19T18:49:51.913 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:51.913 INFO:teuthology.orchestra.run.smithi114.stdout:Install 13 Packages 2024-02-19T18:49:51.913 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:51.913 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 2.0 M 2024-02-19T18:49:51.913 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 26 M 2024-02-19T18:49:51.914 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-19T18:49:52.424 INFO:teuthology.orchestra.run.smithi114.stdout:(1/13): ceph-mgr-rook-18.2.1-0.el9.noarch.rpm 770 kB/s | 50 kB 00:00 2024-02-19T18:49:52.558 INFO:teuthology.orchestra.run.smithi114.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 123 kB/s | 24 kB 00:00 2024-02-19T18:49:52.600 INFO:teuthology.orchestra.run.smithi114.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 75 kB/s | 18 kB 00:00 2024-02-19T18:49:52.700 INFO:teuthology.orchestra.run.smithi114.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 932 kB/s | 132 kB 00:00 2024-02-19T18:49:52.733 INFO:teuthology.orchestra.run.smithi114.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 664 kB/s | 174 kB 00:00 2024-02-19T18:49:52.759 INFO:teuthology.orchestra.run.smithi114.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 211 kB 00:00 2024-02-19T18:49:52.784 INFO:teuthology.orchestra.run.smithi114.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 508 kB/s | 43 kB 00:00 2024-02-19T18:49:53.060 INFO:teuthology.orchestra.run.smithi114.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2024-02-19T18:49:53.085 INFO:teuthology.orchestra.run.smithi114.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 92 kB/s | 32 kB 00:00 2024-02-19T18:49:53.327 INFO:teuthology.orchestra.run.smithi114.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 244 kB/s | 59 kB 00:00 2024-02-19T18:49:53.377 INFO:teuthology.orchestra.run.smithi114.stdout:(11/13): python3-google-auth-2.27.0-1.el9.noarc 342 kB/s | 203 kB 00:00 2024-02-19T18:49:53.544 INFO:teuthology.orchestra.run.smithi114.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 414 kB/s | 90 kB 00:00 2024-02-19T18:49:53.886 INFO:teuthology.orchestra.run.smithi114.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.2 MB/s | 1.0 MB 00:00 2024-02-19T18:49:53.889 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:53.889 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.0 MB/s | 2.0 MB 00:01 2024-02-19T18:49:53.889 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-19T18:49:53.903 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-19T18:49:53.903 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-19T18:49:54.003 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-19T18:49:54.004 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-19T18:49:54.532 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-19T18:49:54.691 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-19T18:49:54.745 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-19T18:49:54.809 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-19T18:49:54.846 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-19T18:49:54.887 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-19T18:49:54.982 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-19T18:49:55.089 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-19T18:49:55.138 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-19T18:49:55.858 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-19T18:49:55.903 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-19T18:49:55.936 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-19T18:49:55.984 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-19T18:49:56.006 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-02-19T18:49:56.799 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-02-19T18:49:56.799 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el9.noarch 1/13 2024-02-19T18:49:56.799 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-19T18:49:56.799 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-19T18:49:56.799 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-19T18:49:56.799 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-19T18:49:56.800 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-19T18:49:56.800 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-19T18:49:56.800 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-19T18:49:56.800 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-19T18:49:56.800 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-19T18:49:56.800 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-19T18:49:56.800 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-19T18:49:57.187 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-19T18:49:57.188 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:57.188 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-19T18:49:57.188 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:18.2.1-0.el9.noarch 2024-02-19T18:49:57.188 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-19T18:49:57.188 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-19T18:49:57.188 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-19T18:49:57.188 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-19T18:49:57.188 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-19T18:49:57.188 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-19T18:49:57.188 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-19T18:49:57.188 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-19T18:49:57.188 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-19T18:49:57.188 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-19T18:49:57.189 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-19T18:49:57.189 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-19T18:49:57.189 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:57.189 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:49:57.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T18:49:58.012 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:16 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:49:58.179 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:49:58.180 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:58.180 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-02-19T18:49:58.180 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:58.180 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-19T18:49:58.181 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el9 ceph-noarch 132 k 2024-02-19T18:49:58.181 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-19T18:49:58.181 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-02-19T18:49:58.181 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-19T18:49:58.181 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-19T18:49:58.181 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-19T18:49:58.181 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-19T18:49:58.181 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:58.181 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-19T18:49:58.181 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:49:58.181 INFO:teuthology.orchestra.run.smithi114.stdout:Install 6 Packages 2024-02-19T18:49:58.181 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:49:58.182 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 6.8 M 2024-02-19T18:49:58.182 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 33 M 2024-02-19T18:49:58.182 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-19T18:49:58.688 INFO:teuthology.orchestra.run.smithi114.stdout:(1/6): ceph-mgr-cephadm-18.2.1-0.el9.noarch.rpm 1.1 MB/s | 132 kB 00:00 2024-02-19T18:49:58.846 INFO:teuthology.orchestra.run.smithi114.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 822 kB/s | 227 kB 00:00 2024-02-19T18:49:59.155 INFO:teuthology.orchestra.run.smithi114.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 9.9 MB/s | 5.8 MB 00:00 2024-02-19T18:49:59.622 INFO:teuthology.orchestra.run.smithi114.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 109 kB/s | 51 kB 00:00 2024-02-19T18:50:00.364 INFO:teuthology.orchestra.run.smithi114.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 38 kB/s | 58 kB 00:01 2024-02-19T18:50:01.672 INFO:teuthology.orchestra.run.smithi114.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 184 kB/s | 548 kB 00:02 2024-02-19T18:50:01.674 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:50:01.674 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.9 MB/s | 6.8 MB 00:03 2024-02-19T18:50:01.675 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-19T18:50:01.684 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-19T18:50:01.685 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-19T18:50:01.778 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-19T18:50:01.778 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-19T18:50:02.070 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-19T18:50:02.234 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-19T18:50:02.289 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-02-19T18:50:02.809 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-19T18:50:02.891 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-19T18:50:02.953 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-19T18:50:02.977 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-02-19T18:50:03.821 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-02-19T18:50:03.821 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 1/6 2024-02-19T18:50:03.821 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-19T18:50:03.821 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-19T18:50:03.821 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-02-19T18:50:03.821 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-19T18:50:04.122 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-19T18:50:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:04.123 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-19T18:50:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 2024-02-19T18:50:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-02-19T18:50:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-19T18:50:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-19T18:50:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-19T18:50:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-19T18:50:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:04.123 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:50:04.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-fuse 2024-02-19T18:50:04.976 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:23 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:50:05.142 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:50:05.143 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:05.143 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-02-19T18:50:05.143 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:05.143 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-19T18:50:05.143 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:18.2.1-0.el9 ceph 839 k 2024-02-19T18:50:05.143 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-02-19T18:50:05.143 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-19T18:50:05.143 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:05.143 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-19T18:50:05.143 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:05.143 INFO:teuthology.orchestra.run.smithi114.stdout:Install 2 Packages 2024-02-19T18:50:05.143 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:05.143 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 918 k 2024-02-19T18:50:05.143 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 2.7 M 2024-02-19T18:50:05.144 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-19T18:50:05.224 INFO:teuthology.orchestra.run.smithi114.stdout:(1/2): ceph-fuse-18.2.1-0.el9.x86_64.rpm 10 MB/s | 839 kB 00:00 2024-02-19T18:50:05.424 INFO:teuthology.orchestra.run.smithi114.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 280 kB/s | 79 kB 00:00 2024-02-19T18:50:05.425 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:50:05.426 INFO:teuthology.orchestra.run.smithi114.stdout:Total 3.2 MB/s | 918 kB 00:00 2024-02-19T18:50:05.426 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-19T18:50:05.446 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-19T18:50:05.446 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-19T18:50:05.521 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-19T18:50:05.521 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-19T18:50:05.703 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-19T18:50:05.831 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-19T18:50:05.898 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-02-19T18:50:06.572 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-02-19T18:50:06.572 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:18.2.1-0.el9.x86_64 1/2 2024-02-19T18:50:06.915 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-19T18:50:06.915 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:06.915 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-19T18:50:06.915 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:18.2.1-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-19T18:50:06.916 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:06.916 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:50:07.181 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados-devel 2024-02-19T18:50:07.792 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:26 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:50:07.956 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:50:07.957 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:07.957 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-02-19T18:50:07.957 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:07.957 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-19T18:50:07.957 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:18.2.1-0.el9 ceph 127 k 2024-02-19T18:50:07.957 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:07.957 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-19T18:50:07.957 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:07.957 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-02-19T18:50:07.958 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:07.958 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 127 k 2024-02-19T18:50:07.958 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 456 k 2024-02-19T18:50:07.958 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-19T18:50:08.029 INFO:teuthology.orchestra.run.smithi114.stdout:librados-devel-18.2.1-0.el9.x86_64.rpm 1.8 MB/s | 127 kB 00:00 2024-02-19T18:50:08.030 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:50:08.030 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.7 MB/s | 127 kB 00:00 2024-02-19T18:50:08.030 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-19T18:50:08.035 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-19T18:50:08.035 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-19T18:50:08.090 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-19T18:50:08.090 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-19T18:50:08.230 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-19T18:50:08.327 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:08.776 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:09.082 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:09.083 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:09.083 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-19T18:50:09.083 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:09.083 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:09.083 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:50:09.355 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs2 2024-02-19T18:50:09.941 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:28 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:50:10.059 INFO:teuthology.orchestra.run.smithi114.stdout:Package libcephfs2-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:50:10.112 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:50:10.113 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-19T18:50:10.113 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:50:10.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs-devel 2024-02-19T18:50:10.759 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:29 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:50:10.925 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:50:10.926 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:10.926 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-02-19T18:50:10.926 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:10.926 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-19T18:50:10.926 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:18.2.1-0.el9 ceph 31 k 2024-02-19T18:50:10.927 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:10.927 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-19T18:50:10.927 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:10.927 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-02-19T18:50:10.927 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:10.927 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 31 k 2024-02-19T18:50:10.927 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 139 k 2024-02-19T18:50:10.927 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-19T18:50:10.998 INFO:teuthology.orchestra.run.smithi114.stdout:libcephfs-devel-18.2.1-0.el9.x86_64.rpm 433 kB/s | 31 kB 00:00 2024-02-19T18:50:10.999 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:50:10.999 INFO:teuthology.orchestra.run.smithi114.stdout:Total 425 kB/s | 31 kB 00:00 2024-02-19T18:50:10.999 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-19T18:50:11.003 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-19T18:50:11.003 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-19T18:50:11.036 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-19T18:50:11.037 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-19T18:50:11.130 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-19T18:50:11.246 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:11.661 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:11.949 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:11.949 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:11.949 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-19T18:50:11.949 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:11.949 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:11.949 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:50:12.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados2 2024-02-19T18:50:12.693 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:31 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:50:12.806 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:50:12.859 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:50:12.860 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-19T18:50:12.860 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:50:12.918 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librbd1 2024-02-19T18:50:13.493 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:32 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:50:13.605 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:50:13.656 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:50:13.657 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-19T18:50:13.657 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:50:13.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rados 2024-02-19T18:50:14.290 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:33 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:50:14.403 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rados-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:50:14.457 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:50:14.458 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-19T18:50:14.458 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:50:14.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rgw 2024-02-19T18:50:15.092 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:34 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:50:15.204 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rgw-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:50:15.255 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:50:15.256 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-19T18:50:15.256 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:50:15.316 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-cephfs 2024-02-19T18:50:15.900 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:34 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:50:16.014 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-cephfs-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:50:16.066 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:50:16.067 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-19T18:50:16.067 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:50:16.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rbd 2024-02-19T18:50:16.703 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:35 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:50:16.815 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rbd-2:18.2.1-0.el9.x86_64 is already installed. 2024-02-19T18:50:16.866 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:50:16.867 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-02-19T18:50:16.867 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:50:16.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-fuse 2024-02-19T18:50:17.520 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:36 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:50:17.685 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:50:17.686 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:17.686 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-02-19T18:50:17.686 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:17.686 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-19T18:50:17.686 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:18.2.1-0.el9 ceph 86 k 2024-02-19T18:50:17.686 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:17.686 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-19T18:50:17.686 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:17.686 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-02-19T18:50:17.686 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:17.687 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 86 k 2024-02-19T18:50:17.687 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 226 k 2024-02-19T18:50:17.687 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-19T18:50:17.760 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-fuse-18.2.1-0.el9.x86_64.rpm 1.1 MB/s | 86 kB 00:00 2024-02-19T18:50:17.760 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:50:17.761 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.1 MB/s | 86 kB 00:00 2024-02-19T18:50:17.761 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-19T18:50:17.767 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-19T18:50:17.767 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-19T18:50:17.822 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-19T18:50:17.822 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-19T18:50:17.958 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-19T18:50:18.049 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:18.519 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:18.836 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:18.837 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:18.837 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-19T18:50:18.837 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:18.837 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:18.837 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:50:19.061 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-mirror 2024-02-19T18:50:19.648 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:38 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:50:19.817 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:50:19.818 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:19.818 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-02-19T18:50:19.818 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:19.818 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-19T18:50:19.818 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-02-19T18:50:19.818 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:19.818 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-19T18:50:19.818 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:19.818 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-02-19T18:50:19.818 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:19.818 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 3.0 M 2024-02-19T18:50:19.818 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 12 M 2024-02-19T18:50:19.818 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-19T18:50:19.921 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-mirror-18.2.1-0.el9.x86_64.rpm 29 MB/s | 3.0 MB 00:00 2024-02-19T18:50:19.921 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:50:19.922 INFO:teuthology.orchestra.run.smithi114.stdout:Total 29 MB/s | 3.0 MB 00:00 2024-02-19T18:50:19.922 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-19T18:50:19.931 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-19T18:50:19.931 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-19T18:50:20.002 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-19T18:50:20.003 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-19T18:50:20.412 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-19T18:50:20.480 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:20.504 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:20.504 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:50:20.504 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-19T18:50:20.504 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-19T18:50:20.504 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-19T18:50:20.504 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:21.621 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:21.621 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:21.621 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-19T18:50:21.621 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:21.621 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:21.621 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:50:21.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-nbd 2024-02-19T18:50:22.461 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:41 ago on Mon 19 Feb 2024 06:48:41 PM UTC. 2024-02-19T18:50:22.627 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-02-19T18:50:22.628 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:22.628 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-02-19T18:50:22.628 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:22.628 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-02-19T18:50:22.628 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:18.2.1-0.el9 ceph 171 k 2024-02-19T18:50:22.629 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:22.629 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-02-19T18:50:22.629 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-02-19T18:50:22.629 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-02-19T18:50:22.629 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:22.629 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 171 k 2024-02-19T18:50:22.629 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 494 k 2024-02-19T18:50:22.629 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-02-19T18:50:22.683 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-nbd-18.2.1-0.el9.x86_64.rpm 3.1 MB/s | 171 kB 00:00 2024-02-19T18:50:22.683 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:50:22.684 INFO:teuthology.orchestra.run.smithi114.stdout:Total 3.0 MB/s | 171 kB 00:00 2024-02-19T18:50:22.684 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-02-19T18:50:22.691 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-02-19T18:50:22.692 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-02-19T18:50:22.746 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-02-19T18:50:22.746 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-02-19T18:50:22.899 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-02-19T18:50:23.006 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:23.501 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:23.829 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-02-19T18:50:23.829 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:23.829 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-02-19T18:50:23.829 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:18.2.1-0.el9.x86_64 2024-02-19T18:50:23.829 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:50:23.829 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-02-19T18:50:24.037 DEBUG:teuthology.parallel:result is None 2024-02-19T18:50:24.037 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T18:50:24.037 INFO:teuthology.packaging:ref: None 2024-02-19T18:50:24.038 INFO:teuthology.packaging:tag: v18.2.1 2024-02-19T18:50:24.038 INFO:teuthology.packaging:branch: None 2024-02-19T18:50:24.038 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:50:24.038 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-02-19T18:50:24.038 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-02-19T18:50:24.192 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T18:50:24.213 INFO:teuthology.orchestra.run.smithi114.stdout:18.2.1-0.el9 2024-02-19T18:50:24.213 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el9 2024-02-19T18:50:24.213 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-02-19T18:50:24.215 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T18:50:24.215 INFO:teuthology.packaging:ref: None 2024-02-19T18:50:24.215 INFO:teuthology.packaging:tag: v18.2.1 2024-02-19T18:50:24.215 INFO:teuthology.packaging:branch: None 2024-02-19T18:50:24.215 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:50:24.215 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-02-19T18:50:24.216 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-02-19T18:50:24.429 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T18:50:24.454 INFO:teuthology.orchestra.run.smithi174.stdout:18.2.1-0.el9 2024-02-19T18:50:24.454 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el9 2024-02-19T18:50:24.454 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-02-19T18:50:24.456 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-19T18:50:24.456 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T18:50:24.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T18:50:24.484 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-02-19T18:50:24.484 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T18:50:24.526 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-19T18:50:24.526 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T18:50:24.526 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T18:50:24.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T18:50:24.619 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-02-19T18:50:24.619 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T18:50:24.650 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T18:50:24.720 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-19T18:50:24.720 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T18:50:24.721 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T18:50:24.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T18:50:24.814 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-02-19T18:50:24.814 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T18:50:24.843 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T18:50:24.908 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-19T18:50:24.909 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T18:50:24.909 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T18:50:24.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T18:50:25.001 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-02-19T18:50:25.002 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T18:50:25.029 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T18:50:25.096 INFO:teuthology.run_tasks:Running task print... 2024-02-19T18:50:25.106 INFO:teuthology.task.print:**** done install task... 2024-02-19T18:50:25.106 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-19T18:50:25.200 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.io/ceph/ceph:v18.2.1', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275'} 2024-02-19T18:50:25.201 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v18.2.1 2024-02-19T18:50:25.201 INFO:tasks.cephadm:Cluster fsid is be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T18:50:25.201 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-19T18:50:25.201 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-19T18:50:25.201 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi114': '172.21.15.114', 'mon.smithi174': '172.21.15.174'} 2024-02-19T18:50:25.202 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-19T18:50:25.202 DEBUG:teuthology.orchestra.run.smithi114:> sudo hostname $(hostname -s) 2024-02-19T18:50:25.229 DEBUG:teuthology.orchestra.run.smithi174:> sudo hostname $(hostname -s) 2024-02-19T18:50:25.256 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-02-19T18:50:25.256 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:50:25.495 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/76790/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1450-gf78a58c0', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1450.gf78a58c0'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240215.160715/f78a58c0ffd401d1493058a1022c35f011d65275/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-02-15 19:25:41.455235', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240215.160715', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240215.160715/f78a58c0ffd401d1493058a1022c35f011d65275/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/76792/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1450-gf78a58c0', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1450.gf78a58c0'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240215.160715/f78a58c0ffd401d1493058a1022c35f011d65275/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-02-15 19:17:06.436952', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240215.160715', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240215.160715/f78a58c0ffd401d1493058a1022c35f011d65275/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-02-19T18:50:25.895 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref reef, sha1 bb2b5a32fc53d75aa463fb74c73b7d0b0ada6320 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-02-19T18:50:25.896 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/reef/bb2b5a32fc53d75aa463fb74c73b7d0b0ada6320/centos/9/x86_64/flavors/default/cephadm 2024-02-19T18:50:25.896 DEBUG:teuthology.orchestra.run.smithi114:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/reef/bb2b5a32fc53d75aa463fb74c73b7d0b0ada6320/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T18:50:26.284 INFO:teuthology.orchestra.run.smithi114.stdout:-rw-r--r--. 1 ubuntu ubuntu 214738 Feb 19 18:50 /home/ubuntu/cephtest/cephadm 2024-02-19T18:50:26.284 DEBUG:teuthology.orchestra.run.smithi174:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/reef/bb2b5a32fc53d75aa463fb74c73b7d0b0ada6320/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T18:50:26.661 INFO:teuthology.orchestra.run.smithi174.stdout:-rw-r--r--. 1 ubuntu ubuntu 214738 Feb 19 18:50 /home/ubuntu/cephtest/cephadm 2024-02-19T18:50:26.661 DEBUG:teuthology.orchestra.run.smithi114:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T18:50:26.680 DEBUG:teuthology.orchestra.run.smithi174:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T18:50:26.707 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v18.2.1 on all hosts... 2024-02-19T18:50:26.708 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-02-19T18:50:26.722 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-02-19T18:50:27.559 INFO:teuthology.orchestra.run.smithi174.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-02-19T18:50:27.627 INFO:teuthology.orchestra.run.smithi114.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-02-19T18:50:50.076 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-02-19T18:50:50.077 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-02-19T18:50:50.077 INFO:teuthology.orchestra.run.smithi174.stdout: "image_id": "94eadd990e0a5619d90a7220f97c70d1fe3558d9b0fb9e0c1935661dd49a6d5f", 2024-02-19T18:50:50.077 INFO:teuthology.orchestra.run.smithi174.stdout: "repo_digests": [ 2024-02-19T18:50:50.077 INFO:teuthology.orchestra.run.smithi174.stdout: "quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0", 2024-02-19T18:50:50.077 INFO:teuthology.orchestra.run.smithi174.stdout: "quay.io/ceph/ceph@sha256:e540726c37a4335a713579a0c9bf4413cb31ad156d22ea1ec00ed9bbc4317f66" 2024-02-19T18:50:50.077 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-02-19T18:50:50.077 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-02-19T18:51:23.222 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T18:51:23.223 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-02-19T18:51:23.223 INFO:teuthology.orchestra.run.smithi114.stdout: "image_id": "94eadd990e0a5619d90a7220f97c70d1fe3558d9b0fb9e0c1935661dd49a6d5f", 2024-02-19T18:51:23.223 INFO:teuthology.orchestra.run.smithi114.stdout: "repo_digests": [ 2024-02-19T18:51:23.223 INFO:teuthology.orchestra.run.smithi114.stdout: "quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0", 2024-02-19T18:51:23.223 INFO:teuthology.orchestra.run.smithi114.stdout: "quay.io/ceph/ceph@sha256:e540726c37a4335a713579a0c9bf4413cb31ad156d22ea1ec00ed9bbc4317f66" 2024-02-19T18:51:23.223 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-02-19T18:51:23.223 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T18:51:23.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph 2024-02-19T18:51:23.289 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph 2024-02-19T18:51:23.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /etc/ceph 2024-02-19T18:51:23.350 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /etc/ceph 2024-02-19T18:51:23.395 INFO:tasks.cephadm:Writing seed config... 2024-02-19T18:51:23.396 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-02-19T18:51:23.396 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-02-19T18:51:23.396 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-19T18:51:23.396 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-19T18:51:23.396 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-19T18:51:23.396 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-19T18:51:23.396 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-19T18:51:23.396 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-19T18:51:23.397 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-19T18:51:23.397 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-19T18:51:23.397 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-19T18:51:23.397 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-19T18:51:23.397 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-19T18:51:23.397 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-19T18:51:23.397 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-19T18:51:23.397 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-19T18:51:23.397 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-19T18:51:23.397 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-19T18:51:23.397 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-19T18:51:23.397 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-19T18:51:23.397 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-19T18:51:23.397 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-19T18:51:23.397 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-19T18:51:23.398 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-19T18:51:23.398 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-19T18:51:23.398 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-19T18:51:23.398 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-19T18:51:23.398 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-19T18:51:23.398 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-19T18:51:23.398 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-19T18:51:23.398 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-19T18:51:23.398 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-19T18:51:23.398 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-19T18:51:23.398 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-19T18:51:23.398 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-19T18:51:23.398 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-19T18:51:23.398 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-19T18:51:23.399 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-19T18:51:23.399 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-19T18:51:23.399 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-19T18:51:23.399 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T18:51:23.399 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-19T18:51:23.414 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = be02a9a6-cf57-11ee-95bb-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-19T18:51:23.414 DEBUG:teuthology.orchestra.run.smithi114:mon.smithi114> sudo journalctl -f -n 0 -u ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@mon.smithi114.service 2024-02-19T18:51:23.457 INFO:tasks.cephadm:Bootstrapping... 2024-02-19T18:51:23.457 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 -v bootstrap --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.114 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-19T18:51:23.612 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:51:23.613 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm ['--image', 'quay.io/ceph/ceph:v18.2.1', '-v', 'bootstrap', '--fsid', 'be02a9a6-cf57-11ee-95bb-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.114', '--skip-admin-label'] 2024-02-19T18:51:23.638 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-19T18:51:23.638 INFO:teuthology.orchestra.run.smithi114.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-19T18:51:23.638 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying podman|docker is present... 2024-02-19T18:51:23.661 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-19T18:51:23.661 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying lvm2 is present... 2024-02-19T18:51:23.661 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying time synchronization is in place... 2024-02-19T18:51:23.669 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-19T18:51:23.669 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T18:51:23.674 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-19T18:51:23.675 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-02-19T18:51:23.681 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout enabled 2024-02-19T18:51:23.686 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout active 2024-02-19T18:51:23.686 INFO:teuthology.orchestra.run.smithi114.stdout:Unit chronyd.service is enabled and running 2024-02-19T18:51:23.687 INFO:teuthology.orchestra.run.smithi114.stdout:Repeating the final host check... 2024-02-19T18:51:23.709 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-19T18:51:23.710 INFO:teuthology.orchestra.run.smithi114.stdout:podman (/bin/podman) version 4.9.3 is present 2024-02-19T18:51:23.710 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl is present 2024-02-19T18:51:23.710 INFO:teuthology.orchestra.run.smithi114.stdout:lvcreate is present 2024-02-19T18:51:23.716 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-19T18:51:23.716 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T18:51:23.721 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-19T18:51:23.721 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-02-19T18:51:23.727 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout enabled 2024-02-19T18:51:23.733 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout active 2024-02-19T18:51:23.733 INFO:teuthology.orchestra.run.smithi114.stdout:Unit chronyd.service is enabled and running 2024-02-19T18:51:23.733 INFO:teuthology.orchestra.run.smithi114.stdout:Host looks OK 2024-02-19T18:51:23.733 INFO:teuthology.orchestra.run.smithi114.stdout:Cluster fsid: be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T18:51:23.733 INFO:teuthology.orchestra.run.smithi114.stdout:Acquiring lock 140076366895328 on /run/cephadm/be02a9a6-cf57-11ee-95bb-87774f69a715.lock 2024-02-19T18:51:23.734 INFO:teuthology.orchestra.run.smithi114.stdout:Lock 140076366895328 acquired on /run/cephadm/be02a9a6-cf57-11ee-95bb-87774f69a715.lock 2024-02-19T18:51:23.734 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying IP 172.21.15.114 port 3300 ... 2024-02-19T18:51:23.734 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying IP 172.21.15.114 port 6789 ... 2024-02-19T18:51:23.734 INFO:teuthology.orchestra.run.smithi114.stdout:Base mon IP(s) is [172.21.15.114:3300, 172.21.15.114:6789], mon addrv is [v2:172.21.15.114:3300,v1:172.21.15.114:6789] 2024-02-19T18:51:23.738 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.114 metric 100 2024-02-19T18:51:23.738 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.114 metric 100 2024-02-19T18:51:23.740 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-19T18:51:23.740 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-19T18:51:23.741 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1786sec hoplimit 64 pref medium 2024-02-19T18:51:23.742 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-19T18:51:23.743 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-19T18:51:23.743 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-19T18:51:23.743 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-19T18:51:23.743 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link 2024-02-19T18:51:23.743 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-19T18:51:23.743 INFO:teuthology.orchestra.run.smithi114.stdout:Mon IP `172.21.15.114` is in CIDR network `172.21.0.0/20` 2024-02-19T18:51:23.743 INFO:teuthology.orchestra.run.smithi114.stdout:Mon IP `172.21.15.114` is in CIDR network `172.21.0.0/20` 2024-02-19T18:51:23.744 INFO:teuthology.orchestra.run.smithi114.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-19T18:51:23.744 INFO:teuthology.orchestra.run.smithi114.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-19T18:51:23.745 INFO:teuthology.orchestra.run.smithi114.stdout:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-02-19T18:51:25.401 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 94eadd990e0a5619d90a7220f97c70d1fe3558d9b0fb9e0c1935661dd49a6d5f 2024-02-19T18:51:25.401 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Trying to pull quay.io/ceph/ceph:v18.2.1... 2024-02-19T18:51:25.402 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Getting image source signatures 2024-02-19T18:51:25.402 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying blob sha256:6345d49baed062606681bf114c10cebf2c69e90ec4d758270fae85ad0cc5c86b 2024-02-19T18:51:25.402 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying blob sha256:0f642e61a11afb42c0ccb5b05f5809250ad643ed143780d6ff49a5a4c492fc38 2024-02-19T18:51:25.402 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying config sha256:94eadd990e0a5619d90a7220f97c70d1fe3558d9b0fb9e0c1935661dd49a6d5f 2024-02-19T18:51:25.402 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-19T18:51:26.321 INFO:teuthology.orchestra.run.smithi114.stdout:ceph: stdout ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-02-19T18:51:26.321 INFO:teuthology.orchestra.run.smithi114.stdout:Ceph version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-02-19T18:51:26.321 INFO:teuthology.orchestra.run.smithi114.stdout:Extracting ceph user uid/gid from container image... 2024-02-19T18:51:27.244 INFO:teuthology.orchestra.run.smithi114.stdout:stat: stdout 167 167 2024-02-19T18:51:27.244 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial keys... 2024-02-19T18:51:28.099 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQAvo9NlkakaIBAAmgokNi4F2TgdW6vQAVfoNw== 2024-02-19T18:51:29.089 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQAwo9NlFTJ+GRAAq4OiafSwO/RcoRLLwEQE9Q== 2024-02-19T18:51:30.136 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQAxo9NlNtTQFRAAsFVGUd5pklht5F7n2RFFpw== 2024-02-19T18:51:30.137 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial monmap... 2024-02-19T18:51:31.009 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T18:51:31.009 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-02-19T18:51:31.009 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T18:51:31.009 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T18:51:31.009 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool for smithi114 [v2:172.21.15.114:3300,v1:172.21.15.114:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T18:51:31.010 INFO:teuthology.orchestra.run.smithi114.stdout:setting min_mon_release = pacific 2024-02-19T18:51:31.010 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: set fsid to be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T18:51:31.010 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T18:51:31.010 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:51:31.010 INFO:teuthology.orchestra.run.smithi114.stdout:Creating mon... 2024-02-19T18:51:31.823 INFO:teuthology.orchestra.run.smithi114.stdout:create mon.smithi114 on 2024-02-19T18:51:32.024 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-02-19T18:51:32.178 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-19T18:51:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-be02a9a6-cf57-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-be02a9a6-cf57-11ee-95bb-87774f69a715.target. 2024-02-19T18:51:32.366 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-be02a9a6-cf57-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-be02a9a6-cf57-11ee-95bb-87774f69a715.target. 2024-02-19T18:51:32.608 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@mon.smithi114 2024-02-19T18:51:32.608 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to reset failed state of unit ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@mon.smithi114.service: Unit ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@mon.smithi114.service not loaded. 2024-02-19T18:51:32.768 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-be02a9a6-cf57-11ee-95bb-87774f69a715.target.wants/ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@mon.smithi114.service → /etc/systemd/system/ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@.service. 2024-02-19T18:51:33.330 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-19T18:51:33.331 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-02-19T18:51:33.337 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-19T18:51:33.337 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-02-19T18:51:33.337 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-02-19T18:51:33.337 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to enable service . firewalld.service is not available 2024-02-19T18:51:33.338 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mon to start... 2024-02-19T18:51:33.338 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mon... 2024-02-19T18:51:34.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout cluster: 2024-02-19T18:51:34.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout id: be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T18:51:34.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-19T18:51:34.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-02-19T18:51:34.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout services: 2024-02-19T18:51:34.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi114 (age 0.606093s) 2024-02-19T18:51:34.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-19T18:51:34.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-19T18:51:34.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-02-19T18:51:34.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout data: 2024-02-19T18:51:34.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-19T18:51:34.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-19T18:51:34.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-19T18:51:34.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout pgs: 2024-02-19T18:51:34.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-02-19T18:51:34.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.811+0000 7f95afa1e700 1 Processor -- start 2024-02-19T18:51:34.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.811+0000 7f95afa1e700 1 -- start start 2024-02-19T18:51:34.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.811+0000 7f95afa1e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95a8106ef0 0x7f95a81092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:34.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.811+0000 7f95afa1e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a8074640 con 0x7f95a8106ef0 2024-02-19T18:51:34.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.812+0000 7f95ad7ba700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95a8106ef0 0x7f95a81092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:34.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.812+0000 7f95ad7ba700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95a8106ef0 0x7f95a81092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55376/0 (socket says 172.21.15.114:55376) 2024-02-19T18:51:34.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.812+0000 7f95ad7ba700 1 -- 172.21.15.114:0/3348235119 learned_addr learned my addr 172.21.15.114:0/3348235119 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:34.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.812+0000 7f95ad7ba700 1 -- 172.21.15.114:0/3348235119 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a8109820 con 0x7f95a8106ef0 2024-02-19T18:51:34.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.812+0000 7f95ad7ba700 1 --2- 172.21.15.114:0/3348235119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95a8106ef0 0x7f95a81092e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f95980098a0 tx=0x7f9598009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e1a68733754c1f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:34.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.812+0000 7f959ffff700 1 -- 172.21.15.114:0/3348235119 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9598004180 con 0x7f95a8106ef0 2024-02-19T18:51:34.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.813+0000 7f959ffff700 1 -- 172.21.15.114:0/3348235119 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95980042e0 con 0x7f95a8106ef0 2024-02-19T18:51:34.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.813+0000 7f959ffff700 1 -- 172.21.15.114:0/3348235119 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95980036a0 con 0x7f95a8106ef0 2024-02-19T18:51:34.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.813+0000 7f95afa1e700 1 -- 172.21.15.114:0/3348235119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95a8106ef0 msgr2=0x7f95a81092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:34.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.813+0000 7f95afa1e700 1 --2- 172.21.15.114:0/3348235119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95a8106ef0 0x7f95a81092e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f95980098a0 tx=0x7f9598009bb0 comp rx=0 tx=0).stop 2024-02-19T18:51:34.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.813+0000 7f95afa1e700 1 -- 172.21.15.114:0/3348235119 shutdown_connections 2024-02-19T18:51:34.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.813+0000 7f95afa1e700 1 --2- 172.21.15.114:0/3348235119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95a8106ef0 0x7f95a81092e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:34.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.813+0000 7f95afa1e700 1 -- 172.21.15.114:0/3348235119 >> 172.21.15.114:0/3348235119 conn(0x7f95a81008c0 msgr2=0x7f95a8102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:34.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.813+0000 7f95afa1e700 1 -- 172.21.15.114:0/3348235119 shutdown_connections 2024-02-19T18:51:34.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.813+0000 7f95afa1e700 1 -- 172.21.15.114:0/3348235119 wait complete. 2024-02-19T18:51:34.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.814+0000 7f95afa1e700 1 Processor -- start 2024-02-19T18:51:34.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.814+0000 7f95afa1e700 1 -- start start 2024-02-19T18:51:34.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.814+0000 7f95afa1e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95a8106ef0 0x7f95a8106850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:34.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.814+0000 7f95afa1e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a8106d90 con 0x7f95a8106ef0 2024-02-19T18:51:34.502 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.814+0000 7f95ad7ba700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95a8106ef0 0x7f95a8106850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:34.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.814+0000 7f95ad7ba700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95a8106ef0 0x7f95a8106850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55378/0 (socket says 172.21.15.114:55378) 2024-02-19T18:51:34.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.814+0000 7f95ad7ba700 1 -- 172.21.15.114:0/3457243487 learned_addr learned my addr 172.21.15.114:0/3457243487 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:34.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.814+0000 7f95ad7ba700 1 -- 172.21.15.114:0/3457243487 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9598009550 con 0x7f95a8106ef0 2024-02-19T18:51:34.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.814+0000 7f95ad7ba700 1 --2- 172.21.15.114:0/3457243487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95a8106ef0 0x7f95a8106850 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f95a8073a80 tx=0x7f9598004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:34.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.815+0000 7f959e7fc700 1 -- 172.21.15.114:0/3457243487 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9598003ce0 con 0x7f95a8106ef0 2024-02-19T18:51:34.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.815+0000 7f959e7fc700 1 -- 172.21.15.114:0/3457243487 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9598020070 con 0x7f95a8106ef0 2024-02-19T18:51:34.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.815+0000 7f959e7fc700 1 -- 172.21.15.114:0/3457243487 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9598011630 con 0x7f95a8106ef0 2024-02-19T18:51:34.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.815+0000 7f95afa1e700 1 -- 172.21.15.114:0/3457243487 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a8104ea0 con 0x7f95a8106ef0 2024-02-19T18:51:34.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.815+0000 7f95afa1e700 1 -- 172.21.15.114:0/3457243487 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a8105340 con 0x7f95a8106ef0 2024-02-19T18:51:34.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.815+0000 7f959e7fc700 1 -- 172.21.15.114:0/3457243487 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9598022030 con 0x7f95a8106ef0 2024-02-19T18:51:34.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.815+0000 7f959e7fc700 1 -- 172.21.15.114:0/3457243487 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9598025d40 con 0x7f95a8106ef0 2024-02-19T18:51:34.503 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.816+0000 7f95afa1e700 1 -- 172.21.15.114:0/3457243487 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a80623c0 con 0x7f95a8106ef0 2024-02-19T18:51:34.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.818+0000 7f959e7fc700 1 -- 172.21.15.114:0/3457243487 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f9598025480 con 0x7f95a8106ef0 2024-02-19T18:51:34.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.887+0000 7f95afa1e700 1 -- 172.21.15.114:0/3457243487 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f95a8193290 con 0x7f95a8106ef0 2024-02-19T18:51:34.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.887+0000 7f959e7fc700 1 -- 172.21.15.114:0/3457243487 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f9598011c20 con 0x7f95a8106ef0 2024-02-19T18:51:34.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.888+0000 7f95afa1e700 1 -- 172.21.15.114:0/3457243487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95a8106ef0 msgr2=0x7f95a8106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:34.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.888+0000 7f95afa1e700 1 --2- 172.21.15.114:0/3457243487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95a8106ef0 0x7f95a8106850 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f95a8073a80 tx=0x7f9598004750 comp rx=0 tx=0).stop 2024-02-19T18:51:34.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.888+0000 7f95afa1e700 1 -- 172.21.15.114:0/3457243487 shutdown_connections 2024-02-19T18:51:34.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.888+0000 7f95afa1e700 1 --2- 172.21.15.114:0/3457243487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95a8106ef0 0x7f95a8106850 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:34.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.888+0000 7f95afa1e700 1 -- 172.21.15.114:0/3457243487 >> 172.21.15.114:0/3457243487 conn(0x7f95a81008c0 msgr2=0x7f95a8102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:34.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.888+0000 7f95afa1e700 1 -- 172.21.15.114:0/3457243487 shutdown_connections 2024-02-19T18:51:34.504 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:33.888+0000 7f95afa1e700 1 -- 172.21.15.114:0/3457243487 wait complete. 2024-02-19T18:51:34.504 INFO:teuthology.orchestra.run.smithi114.stdout:mon is available 2024-02-19T18:51:34.504 INFO:teuthology.orchestra.run.smithi114.stdout:Assimilating anything we can from ceph.conf... 2024-02-19T18:51:35.521 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:35 smithi114 ceph-mon[29067]: from='client.? 172.21.15.114:0/1428782438' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-19T18:51:35.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-02-19T18:51:35.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [global] 2024-02-19T18:51:35.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout fsid = be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T18:51:35.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.114:3300,v1:172.21.15.114:6789] 2024-02-19T18:51:35.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-19T18:51:35.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-19T18:51:35.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-19T18:51:35.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-19T18:51:35.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-02-19T18:51:35.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-19T18:51:35.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-19T18:51:35.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-02-19T18:51:35.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [osd] 2024-02-19T18:51:35.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-19T18:51:35.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-19T18:51:35.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-19T18:51:35.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-19T18:51:35.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-19T18:51:35.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.879+0000 7ff98e68d700 1 Processor -- start 2024-02-19T18:51:35.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.879+0000 7ff98e68d700 1 -- start start 2024-02-19T18:51:35.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.879+0000 7ff98e68d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988106420 0x7ff988106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:35.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.879+0000 7ff98e68d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff988106e10 con 0x7ff988106420 2024-02-19T18:51:35.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.879+0000 7ff987fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988106420 0x7ff988106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:35.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.879+0000 7ff987fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988106420 0x7ff988106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55392/0 (socket says 172.21.15.114:55392) 2024-02-19T18:51:35.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.879+0000 7ff987fff700 1 -- 172.21.15.114:0/2851268785 learned_addr learned my addr 172.21.15.114:0/2851268785 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:35.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.880+0000 7ff987fff700 1 -- 172.21.15.114:0/2851268785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff988107620 con 0x7ff988106420 2024-02-19T18:51:35.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.880+0000 7ff987fff700 1 --2- 172.21.15.114:0/2851268785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988106420 0x7ff988106840 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff9780098a0 tx=0x7ff978009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b482ee3a2dc3511d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:35.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.880+0000 7ff986ffd700 1 -- 172.21.15.114:0/2851268785 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff978004180 con 0x7ff988106420 2024-02-19T18:51:35.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.880+0000 7ff986ffd700 1 -- 172.21.15.114:0/2851268785 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9780042e0 con 0x7ff988106420 2024-02-19T18:51:35.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.880+0000 7ff986ffd700 1 -- 172.21.15.114:0/2851268785 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9780036a0 con 0x7ff988106420 2024-02-19T18:51:35.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.880+0000 7ff98e68d700 1 -- 172.21.15.114:0/2851268785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988106420 msgr2=0x7ff988106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:35.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.880+0000 7ff98e68d700 1 --2- 172.21.15.114:0/2851268785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988106420 0x7ff988106840 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff9780098a0 tx=0x7ff978009bb0 comp rx=0 tx=0).stop 2024-02-19T18:51:35.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.880+0000 7ff98e68d700 1 -- 172.21.15.114:0/2851268785 shutdown_connections 2024-02-19T18:51:35.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.880+0000 7ff98e68d700 1 --2- 172.21.15.114:0/2851268785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988106420 0x7ff988106840 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:35.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.880+0000 7ff98e68d700 1 -- 172.21.15.114:0/2851268785 >> 172.21.15.114:0/2851268785 conn(0x7ff988101a00 msgr2=0x7ff988103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:35.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.880+0000 7ff98e68d700 1 -- 172.21.15.114:0/2851268785 shutdown_connections 2024-02-19T18:51:35.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.881+0000 7ff98e68d700 1 -- 172.21.15.114:0/2851268785 wait complete. 2024-02-19T18:51:35.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.881+0000 7ff98e68d700 1 Processor -- start 2024-02-19T18:51:35.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.881+0000 7ff98e68d700 1 -- start start 2024-02-19T18:51:35.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.881+0000 7ff98e68d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988106420 0x7ff988199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:35.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.881+0000 7ff98e68d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff988199dc0 con 0x7ff988106420 2024-02-19T18:51:35.533 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.881+0000 7ff987fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988106420 0x7ff988199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:35.533 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.881+0000 7ff987fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988106420 0x7ff988199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55408/0 (socket says 172.21.15.114:55408) 2024-02-19T18:51:35.533 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.881+0000 7ff987fff700 1 -- 172.21.15.114:0/1428782438 learned_addr learned my addr 172.21.15.114:0/1428782438 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:35.533 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.882+0000 7ff987fff700 1 -- 172.21.15.114:0/1428782438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff978009550 con 0x7ff988106420 2024-02-19T18:51:35.533 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.882+0000 7ff987fff700 1 --2- 172.21.15.114:0/1428782438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988106420 0x7ff988199880 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff978004000 tx=0x7ff978004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:35.533 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.882+0000 7ff9857fa700 1 -- 172.21.15.114:0/1428782438 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff978003ce0 con 0x7ff988106420 2024-02-19T18:51:35.533 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.882+0000 7ff9857fa700 1 -- 172.21.15.114:0/1428782438 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff978020070 con 0x7ff988106420 2024-02-19T18:51:35.533 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.882+0000 7ff9857fa700 1 -- 172.21.15.114:0/1428782438 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff978011630 con 0x7ff988106420 2024-02-19T18:51:35.533 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.882+0000 7ff98e68d700 1 -- 172.21.15.114:0/1428782438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff988199fc0 con 0x7ff988106420 2024-02-19T18:51:35.533 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.882+0000 7ff98e68d700 1 -- 172.21.15.114:0/1428782438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98819a460 con 0x7ff988106420 2024-02-19T18:51:35.533 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.882+0000 7ff9857fa700 1 -- 172.21.15.114:0/1428782438 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff978022030 con 0x7ff988106420 2024-02-19T18:51:35.533 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.882+0000 7ff9857fa700 1 -- 172.21.15.114:0/1428782438 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff978025d40 con 0x7ff988106420 2024-02-19T18:51:35.533 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.883+0000 7ff98e68d700 1 -- 172.21.15.114:0/1428782438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff988193160 con 0x7ff988106420 2024-02-19T18:51:35.534 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.885+0000 7ff9857fa700 1 -- 172.21.15.114:0/1428782438 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7ff978025480 con 0x7ff988106420 2024-02-19T18:51:35.534 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.953+0000 7ff98e68d700 1 -- 172.21.15.114:0/1428782438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7ff98802cf20 con 0x7ff988106420 2024-02-19T18:51:35.534 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.957+0000 7ff9857fa700 1 -- 172.21.15.114:0/1428782438 <== mon.0 v2:172.21.15.114:3300/0 7 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7ff978011c20 con 0x7ff988106420 2024-02-19T18:51:35.534 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.957+0000 7ff9857fa700 1 -- 172.21.15.114:0/1428782438 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7ff978035310 con 0x7ff988106420 2024-02-19T18:51:35.534 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.957+0000 7ff98e68d700 1 -- 172.21.15.114:0/1428782438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988106420 msgr2=0x7ff988199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:35.534 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.957+0000 7ff98e68d700 1 --2- 172.21.15.114:0/1428782438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988106420 0x7ff988199880 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff978004000 tx=0x7ff978004750 comp rx=0 tx=0).stop 2024-02-19T18:51:35.534 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.958+0000 7ff98e68d700 1 -- 172.21.15.114:0/1428782438 shutdown_connections 2024-02-19T18:51:35.534 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.958+0000 7ff98e68d700 1 --2- 172.21.15.114:0/1428782438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff988106420 0x7ff988199880 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:35.534 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.958+0000 7ff98e68d700 1 -- 172.21.15.114:0/1428782438 >> 172.21.15.114:0/1428782438 conn(0x7ff988101a00 msgr2=0x7ff98818dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:35.534 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.958+0000 7ff98e68d700 1 -- 172.21.15.114:0/1428782438 shutdown_connections 2024-02-19T18:51:35.534 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:34.958+0000 7ff98e68d700 1 -- 172.21.15.114:0/1428782438 wait complete. 2024-02-19T18:51:35.534 INFO:teuthology.orchestra.run.smithi114.stdout:Generating new minimal ceph.conf... 2024-02-19T18:51:36.485 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.864+0000 7fe4e8754700 1 Processor -- start 2024-02-19T18:51:36.485 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.864+0000 7fe4e8754700 1 -- start start 2024-02-19T18:51:36.486 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.864+0000 7fe4e8754700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4e0106420 0x7fe4e0106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:36.486 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.864+0000 7fe4e8754700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0106e10 con 0x7fe4e0106420 2024-02-19T18:51:36.486 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.864+0000 7fe4e64f0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4e0106420 0x7fe4e0106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:36.486 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.864+0000 7fe4e64f0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4e0106420 0x7fe4e0106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50378/0 (socket says 172.21.15.114:50378) 2024-02-19T18:51:36.486 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.864+0000 7fe4e64f0700 1 -- 172.21.15.114:0/2744137440 learned_addr learned my addr 172.21.15.114:0/2744137440 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:36.486 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.864+0000 7fe4e64f0700 1 -- 172.21.15.114:0/2744137440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e0107620 con 0x7fe4e0106420 2024-02-19T18:51:36.486 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.865+0000 7fe4e64f0700 1 --2- 172.21.15.114:0/2744137440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4e0106420 0x7fe4e0106840 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc0098a0 tx=0x7fe4dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d874cc845693896 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:36.486 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.865+0000 7fe4e54ee700 1 -- 172.21.15.114:0/2744137440 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4dc004180 con 0x7fe4e0106420 2024-02-19T18:51:36.486 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.865+0000 7fe4e54ee700 1 -- 172.21.15.114:0/2744137440 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe4dc0042e0 con 0x7fe4e0106420 2024-02-19T18:51:36.486 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.865+0000 7fe4e54ee700 1 -- 172.21.15.114:0/2744137440 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4dc011a50 con 0x7fe4e0106420 2024-02-19T18:51:36.486 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.865+0000 7fe4e8754700 1 -- 172.21.15.114:0/2744137440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4e0106420 msgr2=0x7fe4e0106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:36.486 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.865+0000 7fe4e8754700 1 --2- 172.21.15.114:0/2744137440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4e0106420 0x7fe4e0106840 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc0098a0 tx=0x7fe4dc009bb0 comp rx=0 tx=0).stop 2024-02-19T18:51:36.486 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.865+0000 7fe4e8754700 1 -- 172.21.15.114:0/2744137440 shutdown_connections 2024-02-19T18:51:36.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.865+0000 7fe4e8754700 1 --2- 172.21.15.114:0/2744137440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4e0106420 0x7fe4e0106840 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:36.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.865+0000 7fe4e8754700 1 -- 172.21.15.114:0/2744137440 >> 172.21.15.114:0/2744137440 conn(0x7fe4e0101a00 msgr2=0x7fe4e0103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:36.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.865+0000 7fe4e8754700 1 -- 172.21.15.114:0/2744137440 shutdown_connections 2024-02-19T18:51:36.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.865+0000 7fe4e8754700 1 -- 172.21.15.114:0/2744137440 wait complete. 2024-02-19T18:51:36.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.866+0000 7fe4e8754700 1 Processor -- start 2024-02-19T18:51:36.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.866+0000 7fe4e8754700 1 -- start start 2024-02-19T18:51:36.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.866+0000 7fe4e8754700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4e0106420 0x7fe4e019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:36.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.866+0000 7fe4e8754700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e019a5f0 con 0x7fe4e0106420 2024-02-19T18:51:36.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.866+0000 7fe4e64f0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4e0106420 0x7fe4e019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:36.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.866+0000 7fe4e64f0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4e0106420 0x7fe4e019a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50380/0 (socket says 172.21.15.114:50380) 2024-02-19T18:51:36.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.866+0000 7fe4e64f0700 1 -- 172.21.15.114:0/2024219034 learned_addr learned my addr 172.21.15.114:0/2024219034 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:36.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.866+0000 7fe4e64f0700 1 -- 172.21.15.114:0/2024219034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4dc009550 con 0x7fe4e0106420 2024-02-19T18:51:36.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.867+0000 7fe4e64f0700 1 --2- 172.21.15.114:0/2024219034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4e0106420 0x7fe4e019a0b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc006af0 tx=0x7fe4dc011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:36.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.867+0000 7fe4d77fe700 1 -- 172.21.15.114:0/2024219034 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4dc003b90 con 0x7fe4e0106420 2024-02-19T18:51:36.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.867+0000 7fe4e8754700 1 -- 172.21.15.114:0/2024219034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e019a7f0 con 0x7fe4e0106420 2024-02-19T18:51:36.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.867+0000 7fe4d77fe700 1 -- 172.21.15.114:0/2024219034 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe4dc01a8a0 con 0x7fe4e0106420 2024-02-19T18:51:36.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.867+0000 7fe4d77fe700 1 -- 172.21.15.114:0/2024219034 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4dc019880 con 0x7fe4e0106420 2024-02-19T18:51:36.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.867+0000 7fe4e8754700 1 -- 172.21.15.114:0/2024219034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e019ac90 con 0x7fe4e0106420 2024-02-19T18:51:36.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.867+0000 7fe4d77fe700 1 -- 172.21.15.114:0/2024219034 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe4dc02d440 con 0x7fe4e0106420 2024-02-19T18:51:36.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.867+0000 7fe4d77fe700 1 -- 172.21.15.114:0/2024219034 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe4dc0248a0 con 0x7fe4e0106420 2024-02-19T18:51:36.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.868+0000 7fe4e8754700 1 -- 172.21.15.114:0/2024219034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e00623c0 con 0x7fe4e0106420 2024-02-19T18:51:36.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.870+0000 7fe4d77fe700 1 -- 172.21.15.114:0/2024219034 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fe4dc01aa10 con 0x7fe4e0106420 2024-02-19T18:51:36.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.936+0000 7fe4e8754700 1 -- 172.21.15.114:0/2024219034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fe4e010a8d0 con 0x7fe4e0106420 2024-02-19T18:51:36.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.937+0000 7fe4d77fe700 1 -- 172.21.15.114:0/2024219034 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7fe4dc01abb0 con 0x7fe4e0106420 2024-02-19T18:51:36.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.937+0000 7fe4e8754700 1 -- 172.21.15.114:0/2024219034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4e0106420 msgr2=0x7fe4e019a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:36.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.937+0000 7fe4e8754700 1 --2- 172.21.15.114:0/2024219034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4e0106420 0x7fe4e019a0b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc006af0 tx=0x7fe4dc011a90 comp rx=0 tx=0).stop 2024-02-19T18:51:36.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.937+0000 7fe4e8754700 1 -- 172.21.15.114:0/2024219034 shutdown_connections 2024-02-19T18:51:36.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.937+0000 7fe4e8754700 1 --2- 172.21.15.114:0/2024219034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4e0106420 0x7fe4e019a0b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:36.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.937+0000 7fe4e8754700 1 -- 172.21.15.114:0/2024219034 >> 172.21.15.114:0/2024219034 conn(0x7fe4e0101a00 msgr2=0x7fe4e01027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:36.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.938+0000 7fe4e8754700 1 -- 172.21.15.114:0/2024219034 shutdown_connections 2024-02-19T18:51:36.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:35.938+0000 7fe4e8754700 1 -- 172.21.15.114:0/2024219034 wait complete. 2024-02-19T18:51:36.489 INFO:teuthology.orchestra.run.smithi114.stdout:Restarting the monitor... 2024-02-19T18:51:36.502 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:36 smithi114 ceph-mon[29067]: from='client.? 172.21.15.114:0/2024219034' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:51:36.779 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:36 smithi114 systemd[1]: Stopping Ceph mon.smithi114 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T18:51:36.779 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[29063]: 2024-02-19T18:51:36.732+0000 7f66d820b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi114 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T18:51:36.779 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[29063]: 2024-02-19T18:51:36.732+0000 7f66d820b700 -1 mon.smithi114@0(leader) e1 *** Got Signal Terminated *** 2024-02-19T18:51:37.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:36 smithi114 podman[29315]: 2024-02-19 18:51:36.764038582 +0000 UTC m=+0.136694016 container died 136390e37e42bb46f44a00c5fe29c2b35e5b4655a966fd2537ae63ea48578184 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T18:51:37.414 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 podman[29315]: 2024-02-19 18:51:37.04940898 +0000 UTC m=+0.422064412 container cleanup 136390e37e42bb46f44a00c5fe29c2b35e5b4655a966fd2537ae63ea48578184 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240214) 2024-02-19T18:51:37.414 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 bash[29315]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114 2024-02-19T18:51:37.414 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 podman[29329]: 2024-02-19 18:51:37.275197061 +0000 UTC m=+0.514069476 container remove 136390e37e42bb46f44a00c5fe29c2b35e5b4655a966fd2537ae63ea48578184 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.build-date=20240214, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-02-19T18:51:37.734 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@mon.smithi114.service: Deactivated successfully. 2024-02-19T18:51:37.734 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 systemd[1]: Stopped Ceph mon.smithi114 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T18:51:37.734 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 systemd[1]: Starting Ceph mon.smithi114 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T18:51:37.734 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 podman[29422]: 2024-02-19 18:51:37.618022143 +0000 UTC m=+0.108538321 container create 41468f811fc071a67d1e0ce6a9934c9a386bce3049acd8ce14d8d948c6908255 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T18:51:37.734 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 podman[29422]: 2024-02-19 18:51:37.527257274 +0000 UTC m=+0.017773455 image pull 94eadd990e0a5619d90a7220f97c70d1fe3558d9b0fb9e0c1935661dd49a6d5f quay.io/ceph/ceph:v18.2.1 2024-02-19T18:51:37.898 INFO:teuthology.orchestra.run.smithi114.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-19T18:51:38.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 podman[29422]: 2024-02-19 18:51:37.795369217 +0000 UTC m=+0.285885395 container init 41468f811fc071a67d1e0ce6a9934c9a386bce3049acd8ce14d8d948c6908255 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114, org.label-schema.build-date=20240214, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) 2024-02-19T18:51:38.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 podman[29422]: 2024-02-19 18:51:37.798208565 +0000 UTC m=+0.288724744 container start 41468f811fc071a67d1e0ce6a9934c9a386bce3049acd8ce14d8d948c6908255 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-19T18:51:38.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: set uid:gid to 167:167 (ceph:ceph) 2024-02-19T18:51:38.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable), process ceph-mon, pid 2 2024-02-19T18:51:38.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: pidfile_write: ignore empty --pid-file 2024-02-19T18:51:38.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: load: jerasure load: lrc 2024-02-19T18:51:38.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: RocksDB version: 7.9.2 2024-02-19T18:51:38.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Git sha 0 2024-02-19T18:51:38.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Compile date 2023-12-11 22:07:34 2024-02-19T18:51:38.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: DB SUMMARY 2024-02-19T18:51:38.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: DB Session ID: V8JVRHWMCLKJ6MZJS7NS 2024-02-19T18:51:38.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: CURRENT file: CURRENT 2024-02-19T18:51:38.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: IDENTITY file: IDENTITY 2024-02-19T18:51:38.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-02-19T18:51:38.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi114/store.db dir, Total Num: 1, files: 000008.sst 2024-02-19T18:51:38.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi114/store.db: 000009.log size: 97516 ; 2024-02-19T18:51:38.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.error_if_exists: 0 2024-02-19T18:51:38.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.create_if_missing: 0 2024-02-19T18:51:38.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.paranoid_checks: 1 2024-02-19T18:51:38.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-19T18:51:38.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-19T18:51:38.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-19T18:51:38.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.env: 0x5598386a8720 2024-02-19T18:51:38.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.fs: PosixFileSystem 2024-02-19T18:51:38.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.info_log: 0x55983b9b52e0 2024-02-19T18:51:38.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_file_opening_threads: 16 2024-02-19T18:51:38.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.statistics: (nil) 2024-02-19T18:51:38.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.use_fsync: 0 2024-02-19T18:51:38.025 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_log_file_size: 0 2024-02-19T18:51:38.025 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T18:51:38.025 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T18:51:38.025 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.keep_log_file_num: 1000 2024-02-19T18:51:38.025 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.recycle_log_file_num: 0 2024-02-19T18:51:38.025 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.allow_fallocate: 1 2024-02-19T18:51:38.026 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.allow_mmap_reads: 0 2024-02-19T18:51:38.026 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.allow_mmap_writes: 0 2024-02-19T18:51:38.026 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.use_direct_reads: 0 2024-02-19T18:51:38.026 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T18:51:38.026 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.create_missing_column_families: 0 2024-02-19T18:51:38.026 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.db_log_dir: 2024-02-19T18:51:38.026 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.wal_dir: 2024-02-19T18:51:38.026 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T18:51:38.026 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T18:51:38.026 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T18:51:38.026 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T18:51:38.027 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T18:51:38.027 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T18:51:38.027 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.advise_random_on_open: 1 2024-02-19T18:51:38.027 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.db_write_buffer_size: 0 2024-02-19T18:51:38.027 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.write_buffer_manager: 0x55983ac26140 2024-02-19T18:51:38.027 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T18:51:38.027 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T18:51:38.027 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T18:51:38.027 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.rate_limiter: (nil) 2024-02-19T18:51:38.027 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T18:51:38.027 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.wal_recovery_mode: 2 2024-02-19T18:51:38.027 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.enable_thread_tracking: 0 2024-02-19T18:51:38.028 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.enable_pipelined_write: 0 2024-02-19T18:51:38.028 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.unordered_write: 0 2024-02-19T18:51:38.028 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T18:51:38.028 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T18:51:38.028 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T18:51:38.028 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T18:51:38.028 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.row_cache: None 2024-02-19T18:51:38.028 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.wal_filter: None 2024-02-19T18:51:38.028 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T18:51:38.029 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.allow_ingest_behind: 0 2024-02-19T18:51:38.029 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.two_write_queues: 0 2024-02-19T18:51:38.029 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.manual_wal_flush: 0 2024-02-19T18:51:38.029 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.wal_compression: 0 2024-02-19T18:51:38.029 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.atomic_flush: 0 2024-02-19T18:51:38.029 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T18:51:38.029 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T18:51:38.029 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T18:51:38.029 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.log_readahead_size: 0 2024-02-19T18:51:38.029 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-19T18:51:38.029 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.best_efforts_recovery: 0 2024-02-19T18:51:38.029 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-19T18:51:38.029 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-19T18:51:38.029 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.allow_data_in_errors: 0 2024-02-19T18:51:38.029 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.db_host_id: __hostname__ 2024-02-19T18:51:38.029 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-19T18:51:38.030 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_background_jobs: 2 2024-02-19T18:51:38.030 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_background_compactions: -1 2024-02-19T18:51:38.030 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_subcompactions: 1 2024-02-19T18:51:38.030 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T18:51:38.030 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T18:51:38.030 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T18:51:38.030 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_total_wal_size: 0 2024-02-19T18:51:38.030 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T18:51:38.030 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T18:51:38.030 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T18:51:38.030 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T18:51:38.030 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_open_files: -1 2024-02-19T18:51:38.030 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.bytes_per_sync: 0 2024-02-19T18:51:38.030 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T18:51:38.030 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T18:51:38.030 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compaction_readahead_size: 0 2024-02-19T18:51:38.030 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_background_flushes: -1 2024-02-19T18:51:38.031 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Compression algorithms supported: 2024-02-19T18:51:38.031 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T18:51:38.031 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: kZSTD supported: 0 2024-02-19T18:51:38.031 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: kXpressCompression supported: 0 2024-02-19T18:51:38.031 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: kLZ4HCCompression supported: 1 2024-02-19T18:51:38.031 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: kZlibCompression supported: 1 2024-02-19T18:51:38.031 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: kSnappyCompression supported: 1 2024-02-19T18:51:38.031 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: kLZ4Compression supported: 1 2024-02-19T18:51:38.031 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: kBZip2Compression supported: 0 2024-02-19T18:51:38.031 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T18:51:38.031 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-19T18:51:38.031 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi114/store.db/MANIFEST-000010 2024-02-19T18:51:38.031 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-19T18:51:38.031 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T18:51:38.031 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.merge_operator: 2024-02-19T18:51:38.031 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compaction_filter: None 2024-02-19T18:51:38.032 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compaction_filter_factory: None 2024-02-19T18:51:38.032 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.sst_partitioner_factory: None 2024-02-19T18:51:38.032 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T18:51:38.032 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T18:51:38.032 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55983b9b53e0) 2024-02-19T18:51:38.032 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-02-19T18:51:38.032 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T18:51:38.032 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T18:51:38.032 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-02-19T18:51:38.032 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: index_type: 0 2024-02-19T18:51:38.032 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: data_block_index_type: 0 2024-02-19T18:51:38.032 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: index_shortening: 1 2024-02-19T18:51:38.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-19T18:51:38.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: checksum: 4 2024-02-19T18:51:38.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: no_block_cache: 0 2024-02-19T18:51:38.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_cache: 0x55983acc7350 2024-02-19T18:51:38.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-02-19T18:51:38.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_cache_options: 2024-02-19T18:51:38.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: capacity : 536870912 2024-02-19T18:51:38.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: num_shard_bits : 4 2024-02-19T18:51:38.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: strict_capacity_limit : 0 2024-02-19T18:51:38.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-02-19T18:51:38.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_cache_compressed: (nil) 2024-02-19T18:51:38.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: persistent_cache: (nil) 2024-02-19T18:51:38.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_size: 4096 2024-02-19T18:51:38.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_size_deviation: 10 2024-02-19T18:51:38.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_restart_interval: 16 2024-02-19T18:51:38.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: index_block_restart_interval: 1 2024-02-19T18:51:38.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: metadata_block_size: 4096 2024-02-19T18:51:38.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: partition_filters: 0 2024-02-19T18:51:38.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: use_delta_encoding: 1 2024-02-19T18:51:38.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: filter_policy: bloomfilter 2024-02-19T18:51:38.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: whole_key_filtering: 1 2024-02-19T18:51:38.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: verify_compression: 0 2024-02-19T18:51:38.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-02-19T18:51:38.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: format_version: 5 2024-02-19T18:51:38.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: enable_index_compression: 1 2024-02-19T18:51:38.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_align: 0 2024-02-19T18:51:38.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: max_auto_readahead_size: 262144 2024-02-19T18:51:38.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: prepopulate_block_cache: 0 2024-02-19T18:51:38.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: initial_auto_readahead_size: 8192 2024-02-19T18:51:38.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: num_file_reads_for_auto_readahead: 2 2024-02-19T18:51:38.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.write_buffer_size: 33554432 2024-02-19T18:51:38.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_write_buffer_number: 2 2024-02-19T18:51:38.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compression: NoCompression 2024-02-19T18:51:38.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.bottommost_compression: Disabled 2024-02-19T18:51:38.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.prefix_extractor: nullptr 2024-02-19T18:51:38.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T18:51:38.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.num_levels: 7 2024-02-19T18:51:38.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T18:51:38.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T18:51:38.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T18:51:38.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T18:51:38.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T18:51:38.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T18:51:38.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T18:51:38.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T18:51:38.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-19T18:51:38.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T18:51:38.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-19T18:51:38.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-19T18:51:38.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T18:51:38.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compression_opts.level: 32767 2024-02-19T18:51:38.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compression_opts.strategy: 0 2024-02-19T18:51:38.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T18:51:38.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T18:51:38.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-19T18:51:38.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-19T18:51:38.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compression_opts.enabled: false 2024-02-19T18:51:38.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-19T18:51:38.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T18:51:38.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T18:51:38.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T18:51:38.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.target_file_size_base: 67108864 2024-02-19T18:51:38.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T18:51:38.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T18:51:38.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T18:51:38.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T18:51:38.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T18:51:38.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T18:51:38.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T18:51:38.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T18:51:38.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T18:51:38.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T18:51:38.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T18:51:38.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T18:51:38.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T18:51:38.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-19T18:51:38.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.arena_block_size: 1048576 2024-02-19T18:51:38.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T18:51:38.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T18:51:38.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.disable_auto_compactions: 0 2024-02-19T18:51:38.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T18:51:38.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T18:51:38.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T18:51:38.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T18:51:38.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T18:51:38.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T18:51:38.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T18:51:38.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T18:51:38.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T18:51:38.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T18:51:38.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.table_properties_collectors: 2024-02-19T18:51:38.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.inplace_update_support: 0 2024-02-19T18:51:38.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T18:51:38.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T18:51:38.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T18:51:38.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T18:51:38.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.bloom_locality: 0 2024-02-19T18:51:38.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.max_successive_merges: 0 2024-02-19T18:51:38.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T18:51:38.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.paranoid_file_checks: 0 2024-02-19T18:51:38.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.force_consistency_checks: 1 2024-02-19T18:51:38.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.report_bg_io_stats: 0 2024-02-19T18:51:38.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.ttl: 2592000 2024-02-19T18:51:38.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T18:51:38.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-19T18:51:38.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-19T18:51:38.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.enable_blob_files: false 2024-02-19T18:51:38.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.min_blob_size: 0 2024-02-19T18:51:38.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.blob_file_size: 268435456 2024-02-19T18:51:38.041 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-19T18:51:38.041 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-19T18:51:38.041 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-19T18:51:38.041 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-19T18:51:38.041 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-19T18:51:38.041 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.blob_file_starting_level: 0 2024-02-19T18:51:38.041 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-19T18:51:38.041 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi114/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-02-19T18:51:38.041 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-02-19T18:51:38.041 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6047413c-9651-48e9-8fb4-2546bf462c87 2024-02-19T18:51:38.041 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708368697826761, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-02-19T18:51:38.042 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-02-19T18:51:38.042 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708368697827688, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92866, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 310, "table_properties": {"data_size": 90883, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13356, "raw_average_key_size": 49, "raw_value_size": 83598, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 269, "num_filter_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1708368697, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6047413c-9651-48e9-8fb4-2546bf462c87", "db_session_id": "V8JVRHWMCLKJ6MZJS7NS", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-02-19T18:51:38.042 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708368697827768, "job": 1, "event": "recovery_finished"} 2024-02-19T18:51:38.042 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-02-19T18:51:38.042 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi114/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-19T18:51:38.042 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55983ad64000 2024-02-19T18:51:38.042 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: DB pointer 0x55983ad50000 2024-02-19T18:51:38.042 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-19T18:51:38.043 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-19T18:51:38.043 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ** DB Stats ** 2024-02-19T18:51:38.043 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T18:51:38.043 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T18:51:38.043 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T18:51:38.043 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T18:51:38.044 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T18:51:38.044 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T18:51:38.044 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T18:51:38.044 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-02-19T18:51:38.044 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ** Compaction Stats [default] ** 2024-02-19T18:51:38.044 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-19T18:51:38.044 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-19T18:51:38.044 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: L0 2/0 92.56 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-19T18:51:38.044 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Sum 2/0 92.56 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-19T18:51:38.044 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 159.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-19T18:51:38.044 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-02-19T18:51:38.044 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ** Compaction Stats [default] ** 2024-02-19T18:51:38.045 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-19T18:51:38.045 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T18:51:38.045 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 159.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-19T18:51:38.045 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-02-19T18:51:38.045 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-19T18:51:38.045 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-02-19T18:51:38.045 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T18:51:38.045 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T18:51:38.045 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T18:51:38.045 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T18:51:38.045 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T18:51:38.045 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-19T18:51:38.045 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Cumulative compaction: 0.00 GB write, 30.24 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T18:51:38.045 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Interval compaction: 0.00 GB write, 30.24 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T18:51:38.046 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T18:51:38.046 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Block cache BinnedLRUCache@0x55983acc7350#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-02-19T18:51:38.046 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-02-19T18:51:38.046 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-02-19T18:51:38.046 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-19T18:51:38.046 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: starting mon.smithi114 rank 0 at public addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] at bind addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi114 fsid be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T18:51:38.046 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: mon.smithi114@-1(???) e1 preinit fsid be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T18:51:38.046 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: mon.smithi114@-1(???).mds e1 new map 2024-02-19T18:51:38.046 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: mon.smithi114@-1(???).mds e1 print_map 2024-02-19T18:51:38.046 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: e1 2024-02-19T18:51:38.046 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T18:51:38.046 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T18:51:38.046 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: legacy client fscid: -1 2024-02-19T18:51:38.047 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-02-19T18:51:38.047 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: No filesystems configured 2024-02-19T18:51:38.047 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: mon.smithi114@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-19T18:51:38.047 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: mon.smithi114@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T18:51:38.047 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: mon.smithi114@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T18:51:38.047 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: mon.smithi114@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T18:51:38.047 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: mon.smithi114@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-19T18:51:38.047 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 bash[29422]: 41468f811fc071a67d1e0ce6a9934c9a386bce3049acd8ce14d8d948c6908255 2024-02-19T18:51:38.047 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 systemd[1]: Started Ceph mon.smithi114 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T18:51:38.047 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: mon.smithi114 is new leader, mons smithi114 in quorum (ranks 0) 2024-02-19T18:51:38.047 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: monmap e1: 1 mons at {smithi114=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-19T18:51:38.047 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: fsmap 2024-02-19T18:51:38.047 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: osdmap e1: 0 total, 0 up, 0 in 2024-02-19T18:51:38.047 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:37 smithi114 ceph-mon[29441]: mgrmap e1: no daemons active 2024-02-19T18:51:42.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:41 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1190856371' entity='client.admin' 2024-02-19T18:51:46.347 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.728+0000 7f9be9012700 1 Processor -- start 2024-02-19T18:51:46.347 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.728+0000 7f9be9012700 1 -- start start 2024-02-19T18:51:46.348 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.728+0000 7f9be9012700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be4106f00 0x7f9be41092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:46.348 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.728+0000 7f9be9012700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be4079e60 con 0x7f9be4106f00 2024-02-19T18:51:46.348 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.728+0000 7f9be2d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be4106f00 0x7f9be41092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:46.348 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.728+0000 7f9be2d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be4106f00 0x7f9be41092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50386/0 (socket says 172.21.15.114:50386) 2024-02-19T18:51:46.348 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.728+0000 7f9be2d9d700 1 -- 172.21.15.114:0/2648171795 learned_addr learned my addr 172.21.15.114:0/2648171795 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:46.348 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.728+0000 7f9be2d9d700 1 -- 172.21.15.114:0/2648171795 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be4109830 con 0x7f9be4106f00 2024-02-19T18:51:46.348 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.729+0000 7f9be2d9d700 1 --2- 172.21.15.114:0/2648171795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be4106f00 0x7f9be41092f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9bd40098a0 tx=0x7f9bd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=480f74b1bd2fe052 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:46.348 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.729+0000 7f9be1d9b700 1 -- 172.21.15.114:0/2648171795 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bd4004180 con 0x7f9be4106f00 2024-02-19T18:51:46.348 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.729+0000 7f9be1d9b700 1 -- 172.21.15.114:0/2648171795 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9bd40042e0 con 0x7f9be4106f00 2024-02-19T18:51:46.348 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.729+0000 7f9be1d9b700 1 -- 172.21.15.114:0/2648171795 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bd4011a50 con 0x7f9be4106f00 2024-02-19T18:51:46.348 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.729+0000 7f9be9012700 1 -- 172.21.15.114:0/2648171795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be4106f00 msgr2=0x7f9be41092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:46.348 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.729+0000 7f9be9012700 1 --2- 172.21.15.114:0/2648171795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be4106f00 0x7f9be41092f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9bd40098a0 tx=0x7f9bd4009bb0 comp rx=0 tx=0).stop 2024-02-19T18:51:46.348 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.729+0000 7f9be9012700 1 -- 172.21.15.114:0/2648171795 shutdown_connections 2024-02-19T18:51:46.349 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.729+0000 7f9be9012700 1 --2- 172.21.15.114:0/2648171795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be4106f00 0x7f9be41092f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:46.349 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.729+0000 7f9be9012700 1 -- 172.21.15.114:0/2648171795 >> 172.21.15.114:0/2648171795 conn(0x7f9be41008b0 msgr2=0x7f9be4102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:46.349 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.729+0000 7f9be9012700 1 -- 172.21.15.114:0/2648171795 shutdown_connections 2024-02-19T18:51:46.349 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.729+0000 7f9be9012700 1 -- 172.21.15.114:0/2648171795 wait complete. 2024-02-19T18:51:46.349 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.730+0000 7f9be9012700 1 Processor -- start 2024-02-19T18:51:46.349 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.730+0000 7f9be9012700 1 -- start start 2024-02-19T18:51:46.349 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.730+0000 7f9be9012700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be4106f00 0x7f9be419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:46.349 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.731+0000 7f9be9012700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be419a670 con 0x7f9be4106f00 2024-02-19T18:51:46.349 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.731+0000 7f9be2d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be4106f00 0x7f9be419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:46.349 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.731+0000 7f9be2d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be4106f00 0x7f9be419a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50392/0 (socket says 172.21.15.114:50392) 2024-02-19T18:51:46.349 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.731+0000 7f9be2d9d700 1 -- 172.21.15.114:0/1190856371 learned_addr learned my addr 172.21.15.114:0/1190856371 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:46.349 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.731+0000 7f9be2d9d700 1 -- 172.21.15.114:0/1190856371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd4009550 con 0x7f9be4106f00 2024-02-19T18:51:46.349 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.731+0000 7f9be2d9d700 1 --2- 172.21.15.114:0/1190856371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be4106f00 0x7f9be419a130 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4003de0 tx=0x7f9bd4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:46.349 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.731+0000 7f9bcbfff700 1 -- 172.21.15.114:0/1190856371 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bd4003930 con 0x7f9be4106f00 2024-02-19T18:51:46.349 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.731+0000 7f9bcbfff700 1 -- 172.21.15.114:0/1190856371 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9bd4003a90 con 0x7f9be4106f00 2024-02-19T18:51:46.349 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.731+0000 7f9be9012700 1 -- 172.21.15.114:0/1190856371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be419a870 con 0x7f9be4106f00 2024-02-19T18:51:46.350 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.731+0000 7f9bcbfff700 1 -- 172.21.15.114:0/1190856371 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bd4019730 con 0x7f9be4106f00 2024-02-19T18:51:46.350 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.731+0000 7f9be9012700 1 -- 172.21.15.114:0/1190856371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be419ad10 con 0x7f9be4106f00 2024-02-19T18:51:46.350 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.731+0000 7f9bcbfff700 1 -- 172.21.15.114:0/1190856371 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9bd4019cd0 con 0x7f9be4106f00 2024-02-19T18:51:46.350 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.732+0000 7f9bcbfff700 1 -- 172.21.15.114:0/1190856371 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9bd402c3a0 con 0x7f9be4106f00 2024-02-19T18:51:46.350 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.732+0000 7f9be9012700 1 -- 172.21.15.114:0/1190856371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be40623c0 con 0x7f9be4106f00 2024-02-19T18:51:46.350 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.735+0000 7f9bcbfff700 1 -- 172.21.15.114:0/1190856371 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f9bd401abc0 con 0x7f9be4106f00 2024-02-19T18:51:46.350 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.799+0000 7f9be9012700 1 -- 172.21.15.114:0/1190856371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f9be419d860 con 0x7f9be4106f00 2024-02-19T18:51:46.350 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.802+0000 7f9bcbfff700 1 -- 172.21.15.114:0/1190856371 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9bd4031030 con 0x7f9be4106f00 2024-02-19T18:51:46.350 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.802+0000 7f9be9012700 1 -- 172.21.15.114:0/1190856371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be4106f00 msgr2=0x7f9be419a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:46.350 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.802+0000 7f9be9012700 1 --2- 172.21.15.114:0/1190856371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be4106f00 0x7f9be419a130 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4003de0 tx=0x7f9bd4011ab0 comp rx=0 tx=0).stop 2024-02-19T18:51:46.350 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.803+0000 7f9be9012700 1 -- 172.21.15.114:0/1190856371 shutdown_connections 2024-02-19T18:51:46.350 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.803+0000 7f9be9012700 1 --2- 172.21.15.114:0/1190856371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9be4106f00 0x7f9be419a130 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:46.350 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.803+0000 7f9be9012700 1 -- 172.21.15.114:0/1190856371 >> 172.21.15.114:0/1190856371 conn(0x7f9be41008b0 msgr2=0x7f9be4101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:46.350 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.803+0000 7f9be9012700 1 -- 172.21.15.114:0/1190856371 shutdown_connections 2024-02-19T18:51:46.350 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:40.803+0000 7f9be9012700 1 -- 172.21.15.114:0/1190856371 wait complete. 2024-02-19T18:51:46.351 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-19T18:51:46.363 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-19T18:51:46.363 INFO:teuthology.orchestra.run.smithi114.stdout:Creating mgr... 2024-02-19T18:51:46.363 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:9283 ... 2024-02-19T18:51:46.364 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:8765 ... 2024-02-19T18:51:46.364 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:8443 ... 2024-02-19T18:51:46.602 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@mgr.smithi114.pcvtrk 2024-02-19T18:51:46.603 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to reset failed state of unit ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@mgr.smithi114.pcvtrk.service: Unit ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@mgr.smithi114.pcvtrk.service not loaded. 2024-02-19T18:51:46.754 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-be02a9a6-cf57-11ee-95bb-87774f69a715.target.wants/ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@mgr.smithi114.pcvtrk.service → /etc/systemd/system/ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@.service. 2024-02-19T18:51:47.258 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-19T18:51:47.258 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-02-19T18:51:47.265 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-19T18:51:47.265 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-02-19T18:51:47.265 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-02-19T18:51:47.265 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to enable service . firewalld.service is not available 2024-02-19T18:51:47.273 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-19T18:51:47.273 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-02-19T18:51:47.280 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-19T18:51:47.281 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-02-19T18:51:47.281 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-02-19T18:51:47.281 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-02-19T18:51:47.281 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr to start... 2024-02-19T18:51:47.281 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr... 2024-02-19T18:51:48.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:47 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1539999942' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T18:51:48.281 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-02-19T18:51:48.281 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:51:48.281 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "be02a9a6-cf57-11ee-95bb-87774f69a715", 2024-02-19T18:51:48.282 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-02-19T18:51:48.282 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-19T18:51:48.282 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-19T18:51:48.282 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-19T18:51:48.282 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:48.282 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-19T18:51:48.282 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-19T18:51:48.282 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-02-19T18:51:48.282 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:51:48.282 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-19T18:51:48.282 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "smithi114" 2024-02-19T18:51:48.282 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:51:48.282 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-02-19T18:51:48.283 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-19T18:51:48.283 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:51:48.283 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-02-19T18:51:48.283 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-19T18:51:48.283 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:48.283 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-19T18:51:48.283 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:51:48.283 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-19T18:51:48.283 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-19T18:51:48.283 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-19T18:51:48.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-19T18:51:48.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-19T18:51:48.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-19T18:51:48.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:48.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-19T18:51:48.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-19T18:51:48.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-19T18:51:48.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-19T18:51:48.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-19T18:51:48.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-19T18:51:48.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-19T18:51:48.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-19T18:51:48.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-19T18:51:48.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:48.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-19T18:51:48.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:51:48.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-19T18:51:48.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-19T18:51:48.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:48.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-19T18:51:48.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-19T18:51:48.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-19T18:51:48.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-19T18:51:48.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-19T18:51:48.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-19T18:51:48.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-02-19T18:51:48.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:51:48.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:51:48.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:48.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-19T18:51:48.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:51:48.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-02-19T18:51:33.283529+0000", 2024-02-19T18:51:48.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:51:48.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:48.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-19T18:51:48.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:51:48.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.665+0000 7f4834845700 1 Processor -- start 2024-02-19T18:51:48.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.665+0000 7f4834845700 1 -- start start 2024-02-19T18:51:48.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.665+0000 7f4834845700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4830106f10 0x7f4830109300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:48.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.665+0000 7f4834845700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4830074620 con 0x7f4830106f10 2024-02-19T18:51:48.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.665+0000 7f482e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4830106f10 0x7f4830109300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:48.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.665+0000 7f482e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4830106f10 0x7f4830109300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40722/0 (socket says 172.21.15.114:40722) 2024-02-19T18:51:48.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.665+0000 7f482e59c700 1 -- 172.21.15.114:0/3420900855 learned_addr learned my addr 172.21.15.114:0/3420900855 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:48.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.666+0000 7f482e59c700 1 -- 172.21.15.114:0/3420900855 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4830109840 con 0x7f4830106f10 2024-02-19T18:51:48.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.666+0000 7f482e59c700 1 --2- 172.21.15.114:0/3420900855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4830106f10 0x7f4830109300 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f48240098a0 tx=0x7f4824009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fb6a8e4a33b69f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:48.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.666+0000 7f482d59a700 1 -- 172.21.15.114:0/3420900855 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4824004180 con 0x7f4830106f10 2024-02-19T18:51:48.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.666+0000 7f482d59a700 1 -- 172.21.15.114:0/3420900855 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f48240042e0 con 0x7f4830106f10 2024-02-19T18:51:48.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.666+0000 7f482d59a700 1 -- 172.21.15.114:0/3420900855 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4824011a50 con 0x7f4830106f10 2024-02-19T18:51:48.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.667+0000 7f4834845700 1 -- 172.21.15.114:0/3420900855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4830106f10 msgr2=0x7f4830109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:48.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.667+0000 7f4834845700 1 --2- 172.21.15.114:0/3420900855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4830106f10 0x7f4830109300 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f48240098a0 tx=0x7f4824009bb0 comp rx=0 tx=0).stop 2024-02-19T18:51:48.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.667+0000 7f4834845700 1 -- 172.21.15.114:0/3420900855 shutdown_connections 2024-02-19T18:51:48.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.667+0000 7f4834845700 1 --2- 172.21.15.114:0/3420900855 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4830106f10 0x7f4830109300 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:48.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.667+0000 7f4834845700 1 -- 172.21.15.114:0/3420900855 >> 172.21.15.114:0/3420900855 conn(0x7f48301008c0 msgr2=0x7f4830102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:48.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.667+0000 7f4834845700 1 -- 172.21.15.114:0/3420900855 shutdown_connections 2024-02-19T18:51:48.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.667+0000 7f4834845700 1 -- 172.21.15.114:0/3420900855 wait complete. 2024-02-19T18:51:48.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.667+0000 7f4834845700 1 Processor -- start 2024-02-19T18:51:48.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.668+0000 7f4834845700 1 -- start start 2024-02-19T18:51:48.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.668+0000 7f4834845700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4830106f10 0x7f4830106870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:48.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.668+0000 7f4834845700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4830106db0 con 0x7f4830106f10 2024-02-19T18:51:48.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.668+0000 7f482e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4830106f10 0x7f4830106870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:48.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.668+0000 7f482e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4830106f10 0x7f4830106870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40730/0 (socket says 172.21.15.114:40730) 2024-02-19T18:51:48.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.668+0000 7f482e59c700 1 -- 172.21.15.114:0/1539999942 learned_addr learned my addr 172.21.15.114:0/1539999942 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:48.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.668+0000 7f482e59c700 1 -- 172.21.15.114:0/1539999942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4824009550 con 0x7f4830106f10 2024-02-19T18:51:48.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.668+0000 7f482e59c700 1 --2- 172.21.15.114:0/1539999942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4830106f10 0x7f4830106870 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4824003f80 tx=0x7f4824011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:48.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.668+0000 7f481f7fe700 1 -- 172.21.15.114:0/1539999942 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4824003a80 con 0x7f4830106f10 2024-02-19T18:51:48.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.668+0000 7f4834845700 1 -- 172.21.15.114:0/1539999942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4830104ec0 con 0x7f4830106f10 2024-02-19T18:51:48.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.668+0000 7f481f7fe700 1 -- 172.21.15.114:0/1539999942 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4824003be0 con 0x7f4830106f10 2024-02-19T18:51:48.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.668+0000 7f481f7fe700 1 -- 172.21.15.114:0/1539999942 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4824019820 con 0x7f4830106f10 2024-02-19T18:51:48.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.669+0000 7f4834845700 1 -- 172.21.15.114:0/1539999942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4830105360 con 0x7f4830106f10 2024-02-19T18:51:48.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.669+0000 7f481f7fe700 1 -- 172.21.15.114:0/1539999942 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4824019ac0 con 0x7f4830106f10 2024-02-19T18:51:48.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.669+0000 7f481f7fe700 1 -- 172.21.15.114:0/1539999942 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f48240243f0 con 0x7f4830106f10 2024-02-19T18:51:48.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.670+0000 7f4834845700 1 -- 172.21.15.114:0/1539999942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f483010a560 con 0x7f4830106f10 2024-02-19T18:51:48.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.672+0000 7f481f7fe700 1 -- 172.21.15.114:0/1539999942 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f482401acc0 con 0x7f4830106f10 2024-02-19T18:51:48.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.743+0000 7f4834845700 1 -- 172.21.15.114:0/1539999942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f483010a750 con 0x7f4830106f10 2024-02-19T18:51:48.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.744+0000 7f481f7fe700 1 -- 172.21.15.114:0/1539999942 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f482401a730 con 0x7f4830106f10 2024-02-19T18:51:48.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.745+0000 7f4834845700 1 -- 172.21.15.114:0/1539999942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4830106f10 msgr2=0x7f4830106870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:48.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.745+0000 7f4834845700 1 --2- 172.21.15.114:0/1539999942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4830106f10 0x7f4830106870 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4824003f80 tx=0x7f4824011ab0 comp rx=0 tx=0).stop 2024-02-19T18:51:48.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.745+0000 7f4834845700 1 -- 172.21.15.114:0/1539999942 shutdown_connections 2024-02-19T18:51:48.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.745+0000 7f4834845700 1 --2- 172.21.15.114:0/1539999942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4830106f10 0x7f4830106870 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:48.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.745+0000 7f4834845700 1 -- 172.21.15.114:0/1539999942 >> 172.21.15.114:0/1539999942 conn(0x7f48301008c0 msgr2=0x7f4830101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:48.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.745+0000 7f4834845700 1 -- 172.21.15.114:0/1539999942 shutdown_connections 2024-02-19T18:51:48.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:47.745+0000 7f4834845700 1 -- 172.21.15.114:0/1539999942 wait complete. 2024-02-19T18:51:48.300 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (1/15)... 2024-02-19T18:51:51.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:50 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/4256409271' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T18:51:52.768 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-02-19T18:51:52.768 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:51:52.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "be02a9a6-cf57-11ee-95bb-87774f69a715", 2024-02-19T18:51:52.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-02-19T18:51:52.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-19T18:51:52.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-19T18:51:52.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-19T18:51:52.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:52.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-19T18:51:52.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-19T18:51:52.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-02-19T18:51:52.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:51:52.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-19T18:51:52.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "smithi114" 2024-02-19T18:51:52.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:51:52.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-02-19T18:51:52.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-19T18:51:52.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:51:52.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-02-19T18:51:52.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-19T18:51:52.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:52.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-19T18:51:52.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:51:52.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-19T18:51:52.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-19T18:51:52.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-19T18:51:52.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-19T18:51:52.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-19T18:51:52.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-19T18:51:52.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:52.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-19T18:51:52.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-19T18:51:52.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-19T18:51:52.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-19T18:51:52.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-19T18:51:52.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-19T18:51:52.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-19T18:51:52.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-19T18:51:52.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-19T18:51:52.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:52.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-19T18:51:52.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:51:52.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-19T18:51:52.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-19T18:51:52.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:52.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-19T18:51:52.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-19T18:51:52.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-19T18:51:52.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-19T18:51:52.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-19T18:51:52.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-19T18:51:52.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-02-19T18:51:52.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:51:52.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:51:52.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:52.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-19T18:51:52.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:51:52.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-02-19T18:51:33.283529+0000", 2024-02-19T18:51:52.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:51:52.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:52.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-19T18:51:52.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:51:52.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.653+0000 7fc52c799700 1 Processor -- start 2024-02-19T18:51:52.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.653+0000 7fc52c799700 1 -- start start 2024-02-19T18:51:52.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.653+0000 7fc52c799700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc524106f00 0x7fc5241092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:52.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.653+0000 7fc52c799700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc524079e80 con 0x7fc524106f00 2024-02-19T18:51:52.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.654+0000 7fc52a535700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc524106f00 0x7fc5241092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:52.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.654+0000 7fc52a535700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc524106f00 0x7fc5241092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40744/0 (socket says 172.21.15.114:40744) 2024-02-19T18:51:52.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.654+0000 7fc52a535700 1 -- 172.21.15.114:0/588760860 learned_addr learned my addr 172.21.15.114:0/588760860 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:52.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.654+0000 7fc52a535700 1 -- 172.21.15.114:0/588760860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc524109830 con 0x7fc524106f00 2024-02-19T18:51:52.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.654+0000 7fc52a535700 1 --2- 172.21.15.114:0/588760860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc524106f00 0x7fc5241092f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc5200098a0 tx=0x7fc520009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11329941dd0d665f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:52.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.654+0000 7fc529533700 1 -- 172.21.15.114:0/588760860 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc520004180 con 0x7fc524106f00 2024-02-19T18:51:52.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.654+0000 7fc529533700 1 -- 172.21.15.114:0/588760860 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc5200042e0 con 0x7fc524106f00 2024-02-19T18:51:52.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.654+0000 7fc529533700 1 -- 172.21.15.114:0/588760860 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc520011a50 con 0x7fc524106f00 2024-02-19T18:51:52.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.655+0000 7fc52c799700 1 -- 172.21.15.114:0/588760860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc524106f00 msgr2=0x7fc5241092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:52.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.655+0000 7fc52c799700 1 --2- 172.21.15.114:0/588760860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc524106f00 0x7fc5241092f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc5200098a0 tx=0x7fc520009bb0 comp rx=0 tx=0).stop 2024-02-19T18:51:52.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.655+0000 7fc52c799700 1 -- 172.21.15.114:0/588760860 shutdown_connections 2024-02-19T18:51:52.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.655+0000 7fc52c799700 1 --2- 172.21.15.114:0/588760860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc524106f00 0x7fc5241092f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:52.782 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.655+0000 7fc52c799700 1 -- 172.21.15.114:0/588760860 >> 172.21.15.114:0/588760860 conn(0x7fc5241008d0 msgr2=0x7fc524102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:52.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.655+0000 7fc52c799700 1 -- 172.21.15.114:0/588760860 shutdown_connections 2024-02-19T18:51:52.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.655+0000 7fc52c799700 1 -- 172.21.15.114:0/588760860 wait complete. 2024-02-19T18:51:52.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.656+0000 7fc52c799700 1 Processor -- start 2024-02-19T18:51:52.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.656+0000 7fc52c799700 1 -- start start 2024-02-19T18:51:52.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.656+0000 7fc52c799700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc524106f00 0x7fc52419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:52.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.656+0000 7fc52c799700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52419a610 con 0x7fc524106f00 2024-02-19T18:51:52.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.656+0000 7fc52a535700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc524106f00 0x7fc52419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:52.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.656+0000 7fc52a535700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc524106f00 0x7fc52419a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40760/0 (socket says 172.21.15.114:40760) 2024-02-19T18:51:52.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.656+0000 7fc52a535700 1 -- 172.21.15.114:0/4256409271 learned_addr learned my addr 172.21.15.114:0/4256409271 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:52.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.656+0000 7fc52a535700 1 -- 172.21.15.114:0/4256409271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc520009550 con 0x7fc524106f00 2024-02-19T18:51:52.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.656+0000 7fc52a535700 1 --2- 172.21.15.114:0/4256409271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc524106f00 0x7fc52419a0d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc520003de0 tx=0x7fc520011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:52.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.657+0000 7fc51b7fe700 1 -- 172.21.15.114:0/4256409271 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc520003930 con 0x7fc524106f00 2024-02-19T18:51:52.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.657+0000 7fc52c799700 1 -- 172.21.15.114:0/4256409271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc52419a810 con 0x7fc524106f00 2024-02-19T18:51:52.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.657+0000 7fc51b7fe700 1 -- 172.21.15.114:0/4256409271 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fc520003a90 con 0x7fc524106f00 2024-02-19T18:51:52.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.657+0000 7fc51b7fe700 1 -- 172.21.15.114:0/4256409271 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc520019730 con 0x7fc524106f00 2024-02-19T18:51:52.783 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.657+0000 7fc52c799700 1 -- 172.21.15.114:0/4256409271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc52419acb0 con 0x7fc524106f00 2024-02-19T18:51:52.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.657+0000 7fc51b7fe700 1 -- 172.21.15.114:0/4256409271 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc520019cb0 con 0x7fc524106f00 2024-02-19T18:51:52.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.657+0000 7fc51b7fe700 1 -- 172.21.15.114:0/4256409271 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc52002c3a0 con 0x7fc524106f00 2024-02-19T18:51:52.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.658+0000 7fc52c799700 1 -- 172.21.15.114:0/4256409271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5240623c0 con 0x7fc524106f00 2024-02-19T18:51:52.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.662+0000 7fc51b7fe700 1 -- 172.21.15.114:0/4256409271 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fc52001abc0 con 0x7fc524106f00 2024-02-19T18:51:52.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.732+0000 7fc52c799700 1 -- 172.21.15.114:0/4256409271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc52419d670 con 0x7fc524106f00 2024-02-19T18:51:52.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.733+0000 7fc51b7fe700 1 -- 172.21.15.114:0/4256409271 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1247 (secure 0 0 0) 0x7fc52001a430 con 0x7fc524106f00 2024-02-19T18:51:52.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.734+0000 7fc52c799700 1 -- 172.21.15.114:0/4256409271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc524106f00 msgr2=0x7fc52419a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:52.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.734+0000 7fc52c799700 1 --2- 172.21.15.114:0/4256409271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc524106f00 0x7fc52419a0d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc520003de0 tx=0x7fc520011ab0 comp rx=0 tx=0).stop 2024-02-19T18:51:52.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.734+0000 7fc52c799700 1 -- 172.21.15.114:0/4256409271 shutdown_connections 2024-02-19T18:51:52.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.734+0000 7fc52c799700 1 --2- 172.21.15.114:0/4256409271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc524106f00 0x7fc52419a0d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:52.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.734+0000 7fc52c799700 1 -- 172.21.15.114:0/4256409271 >> 172.21.15.114:0/4256409271 conn(0x7fc5241008d0 msgr2=0x7fc524101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:52.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.734+0000 7fc52c799700 1 -- 172.21.15.114:0/4256409271 shutdown_connections 2024-02-19T18:51:52.784 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:50.734+0000 7fc52c799700 1 -- 172.21.15.114:0/4256409271 wait complete. 2024-02-19T18:51:52.785 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (2/15)... 2024-02-19T18:51:54.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:53 smithi114 ceph-mon[29441]: Activating manager daemon smithi114.pcvtrk 2024-02-19T18:51:54.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:53 smithi114 ceph-mon[29441]: mgrmap e2: smithi114.pcvtrk(active, starting, since 0.00314699s) 2024-02-19T18:51:54.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:53 smithi114 ceph-mon[29441]: from='mgr.14100 172.21.15.114:0/4084155145' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T18:51:54.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:53 smithi114 ceph-mon[29441]: from='mgr.14100 172.21.15.114:0/4084155145' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T18:51:54.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:53 smithi114 ceph-mon[29441]: from='mgr.14100 172.21.15.114:0/4084155145' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T18:51:54.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:53 smithi114 ceph-mon[29441]: from='mgr.14100 172.21.15.114:0/4084155145' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T18:51:54.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:53 smithi114 ceph-mon[29441]: from='mgr.14100 172.21.15.114:0/4084155145' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr metadata", "who": "smithi114.pcvtrk", "id": "smithi114.pcvtrk"}]: dispatch 2024-02-19T18:51:54.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:53 smithi114 ceph-mon[29441]: Manager daemon smithi114.pcvtrk is now available 2024-02-19T18:51:54.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:53 smithi114 ceph-mon[29441]: from='mgr.14100 172.21.15.114:0/4084155145' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T18:51:54.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:53 smithi114 ceph-mon[29441]: from='mgr.14100 172.21.15.114:0/4084155145' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/trash_purge_schedule"}]: dispatch 2024-02-19T18:51:54.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:53 smithi114 ceph-mon[29441]: from='mgr.14100 172.21.15.114:0/4084155145' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:51:54.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:53 smithi114 ceph-mon[29441]: from='mgr.14100 172.21.15.114:0/4084155145' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:51:54.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:53 smithi114 ceph-mon[29441]: from='mgr.14100 172.21.15.114:0/4084155145' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:51:56.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-02-19T18:51:56.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:51:56.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "be02a9a6-cf57-11ee-95bb-87774f69a715", 2024-02-19T18:51:56.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-02-19T18:51:56.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-19T18:51:56.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-19T18:51:56.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-19T18:51:56.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:56.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-19T18:51:56.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-19T18:51:56.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-02-19T18:51:56.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:51:56.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-19T18:51:56.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "smithi114" 2024-02-19T18:51:56.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:51:56.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 17, 2024-02-19T18:51:56.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-19T18:51:56.036 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:51:56.036 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-02-19T18:51:56.036 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-19T18:51:56.036 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:56.036 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-19T18:51:56.036 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:51:56.036 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-19T18:51:56.036 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-19T18:51:56.036 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-19T18:51:56.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-19T18:51:56.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-19T18:51:56.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-19T18:51:56.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:56.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-19T18:51:56.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-19T18:51:56.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-19T18:51:56.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-19T18:51:56.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-19T18:51:56.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-19T18:51:56.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-19T18:51:56.045 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-19T18:51:56.046 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-19T18:51:56.046 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:56.046 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-19T18:51:56.046 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:51:56.046 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-19T18:51:56.046 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-19T18:51:56.046 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:56.046 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-19T18:51:56.046 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-19T18:51:56.046 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-19T18:51:56.046 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-19T18:51:56.046 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-19T18:51:56.046 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-19T18:51:56.046 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-02-19T18:51:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:51:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:51:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-19T18:51:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:51:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-02-19T18:51:33.283529+0000", 2024-02-19T18:51:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:51:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:51:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-19T18:51:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:51:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.209+0000 7f34b6b28700 1 Processor -- start 2024-02-19T18:51:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.209+0000 7f34b6b28700 1 -- start start 2024-02-19T18:51:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.209+0000 7f34b6b28700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34b0106420 0x7f34b0106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.209+0000 7f34b6b28700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b0106e10 con 0x7f34b0106420 2024-02-19T18:51:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.209+0000 7f34b48c4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34b0106420 0x7f34b0106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:56.048 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.209+0000 7f34b48c4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34b0106420 0x7f34b0106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40836/0 (socket says 172.21.15.114:40836) 2024-02-19T18:51:56.048 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.209+0000 7f34b48c4700 1 -- 172.21.15.114:0/3529051187 learned_addr learned my addr 172.21.15.114:0/3529051187 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:56.048 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.210+0000 7f34b48c4700 1 -- 172.21.15.114:0/3529051187 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b0107620 con 0x7f34b0106420 2024-02-19T18:51:56.048 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.210+0000 7f34b48c4700 1 --2- 172.21.15.114:0/3529051187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34b0106420 0x7f34b0106840 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f34a40098a0 tx=0x7f34a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a924aa6527cdaae2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:56.048 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.210+0000 7f34af7fe700 1 -- 172.21.15.114:0/3529051187 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34a4004180 con 0x7f34b0106420 2024-02-19T18:51:56.048 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.210+0000 7f34af7fe700 1 -- 172.21.15.114:0/3529051187 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f34a40042e0 con 0x7f34b0106420 2024-02-19T18:51:56.048 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.210+0000 7f34af7fe700 1 -- 172.21.15.114:0/3529051187 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34a4011a50 con 0x7f34b0106420 2024-02-19T18:51:56.048 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.211+0000 7f34b6b28700 1 -- 172.21.15.114:0/3529051187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34b0106420 msgr2=0x7f34b0106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:56.048 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.211+0000 7f34b6b28700 1 --2- 172.21.15.114:0/3529051187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34b0106420 0x7f34b0106840 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f34a40098a0 tx=0x7f34a4009bb0 comp rx=0 tx=0).stop 2024-02-19T18:51:56.048 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.211+0000 7f34b6b28700 1 -- 172.21.15.114:0/3529051187 shutdown_connections 2024-02-19T18:51:56.048 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.211+0000 7f34b6b28700 1 --2- 172.21.15.114:0/3529051187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34b0106420 0x7f34b0106840 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:56.048 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.211+0000 7f34b6b28700 1 -- 172.21.15.114:0/3529051187 >> 172.21.15.114:0/3529051187 conn(0x7f34b0101a00 msgr2=0x7f34b0103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:56.048 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.211+0000 7f34b6b28700 1 -- 172.21.15.114:0/3529051187 shutdown_connections 2024-02-19T18:51:56.049 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.211+0000 7f34b6b28700 1 -- 172.21.15.114:0/3529051187 wait complete. 2024-02-19T18:51:56.049 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.211+0000 7f34b6b28700 1 Processor -- start 2024-02-19T18:51:56.049 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.211+0000 7f34b6b28700 1 -- start start 2024-02-19T18:51:56.049 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.211+0000 7f34b6b28700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34b0106420 0x7f34b007bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:56.049 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.211+0000 7f34b6b28700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b007c1a0 con 0x7f34b0106420 2024-02-19T18:51:56.049 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.212+0000 7f34b48c4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34b0106420 0x7f34b007bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:56.049 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.212+0000 7f34b48c4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34b0106420 0x7f34b007bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40838/0 (socket says 172.21.15.114:40838) 2024-02-19T18:51:56.049 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.212+0000 7f34b48c4700 1 -- 172.21.15.114:0/269262835 learned_addr learned my addr 172.21.15.114:0/269262835 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:56.049 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.212+0000 7f34b48c4700 1 -- 172.21.15.114:0/269262835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a4009550 con 0x7f34b0106420 2024-02-19T18:51:56.049 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.212+0000 7f34b48c4700 1 --2- 172.21.15.114:0/269262835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34b0106420 0x7f34b007bc60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f34a4000c00 tx=0x7f34a4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:56.049 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.212+0000 7f34adffb700 1 -- 172.21.15.114:0/269262835 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34a4003b90 con 0x7f34b0106420 2024-02-19T18:51:56.049 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.212+0000 7f34adffb700 1 -- 172.21.15.114:0/269262835 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f34a40248b0 con 0x7f34b0106420 2024-02-19T18:51:56.049 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.212+0000 7f34adffb700 1 -- 172.21.15.114:0/269262835 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34a401a8e0 con 0x7f34b0106420 2024-02-19T18:51:56.049 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.212+0000 7f34b6b28700 1 -- 172.21.15.114:0/269262835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b007a2b0 con 0x7f34b0106420 2024-02-19T18:51:56.050 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.212+0000 7f34b6b28700 1 -- 172.21.15.114:0/269262835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b007a750 con 0x7f34b0106420 2024-02-19T18:51:56.050 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.213+0000 7f34adffb700 1 -- 172.21.15.114:0/269262835 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 3) v1 ==== 45166+0+0 (secure 0 0 0) 0x7f34a4024a20 con 0x7f34b0106420 2024-02-19T18:51:56.050 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.213+0000 7f34adffb700 1 --2- 172.21.15.114:0/269262835 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f34a0038200 0x7f34a003a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:56.050 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.213+0000 7f34adffb700 1 -- 172.21.15.114:0/269262835 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f34a404bd90 con 0x7f34b0106420 2024-02-19T18:51:56.050 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.213+0000 7f34b6b28700 1 -- 172.21.15.114:0/269262835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b00623c0 con 0x7f34b0106420 2024-02-19T18:51:56.050 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.213+0000 7f34affff700 1 --2- 172.21.15.114:0/269262835 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f34a0038200 0x7f34a003a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:56.050 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.214+0000 7f34affff700 1 --2- 172.21.15.114:0/269262835 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f34a0038200 0x7f34a003a6c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f349c006fd0 tx=0x7f349c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:56.050 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.218+0000 7f34adffb700 1 -- 172.21.15.114:0/269262835 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f34a4018db0 con 0x7f34b0106420 2024-02-19T18:51:56.050 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.478+0000 7f34b6b28700 1 -- 172.21.15.114:0/269262835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f34b01a0100 con 0x7f34b0106420 2024-02-19T18:51:56.050 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.478+0000 7f34adffb700 1 -- 172.21.15.114:0/269262835 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f34a404a090 con 0x7f34b0106420 2024-02-19T18:51:56.050 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.480+0000 7f34b6b28700 1 -- 172.21.15.114:0/269262835 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f34a0038200 msgr2=0x7f34a003a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:56.050 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.480+0000 7f34b6b28700 1 --2- 172.21.15.114:0/269262835 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f34a0038200 0x7f34a003a6c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f349c006fd0 tx=0x7f349c006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:56.050 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.480+0000 7f34b6b28700 1 -- 172.21.15.114:0/269262835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34b0106420 msgr2=0x7f34b007bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:56.050 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.480+0000 7f34b6b28700 1 --2- 172.21.15.114:0/269262835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34b0106420 0x7f34b007bc60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f34a4000c00 tx=0x7f34a4011ab0 comp rx=0 tx=0).stop 2024-02-19T18:51:56.051 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.481+0000 7f34b6b28700 1 -- 172.21.15.114:0/269262835 shutdown_connections 2024-02-19T18:51:56.051 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.481+0000 7f34b6b28700 1 --2- 172.21.15.114:0/269262835 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f34a0038200 0x7f34a003a6c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:56.051 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.481+0000 7f34b6b28700 1 --2- 172.21.15.114:0/269262835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34b0106420 0x7f34b007bc60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:56.051 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.481+0000 7f34b6b28700 1 -- 172.21.15.114:0/269262835 >> 172.21.15.114:0/269262835 conn(0x7f34b0101a00 msgr2=0x7f34b01027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:56.051 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.481+0000 7f34b6b28700 1 -- 172.21.15.114:0/269262835 shutdown_connections 2024-02-19T18:51:56.051 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:55.481+0000 7f34b6b28700 1 -- 172.21.15.114:0/269262835 wait complete. 2024-02-19T18:51:56.051 INFO:teuthology.orchestra.run.smithi114.stdout:mgr is available 2024-02-19T18:51:56.162 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:55 smithi114 ceph-mon[29441]: mgrmap e3: smithi114.pcvtrk(active, since 1.00687s) 2024-02-19T18:51:56.162 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:55 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/269262835' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T18:51:57.186 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:56 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/711299807' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-19T18:51:57.194 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-02-19T18:51:57.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [global] 2024-02-19T18:51:57.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout fsid = be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T18:51:57.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-19T18:51:57.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-19T18:51:57.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-19T18:51:57.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-19T18:51:57.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-02-19T18:51:57.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-19T18:51:57.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-19T18:51:57.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-02-19T18:51:57.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [osd] 2024-02-19T18:51:57.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-19T18:51:57.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-19T18:51:57.195 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-19T18:51:57.196 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-19T18:51:57.196 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-19T18:51:57.196 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.439+0000 7f47195da700 1 Processor -- start 2024-02-19T18:51:57.196 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.439+0000 7f47195da700 1 -- start start 2024-02-19T18:51:57.196 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.439+0000 7f47195da700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4714105330 0x7f4714105750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:57.196 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.439+0000 7f47195da700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4714105d20 con 0x7f4714105330 2024-02-19T18:51:57.196 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.439+0000 7f4712ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4714105330 0x7f4714105750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:57.196 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.439+0000 7f4712ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4714105330 0x7f4714105750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44740/0 (socket says 172.21.15.114:44740) 2024-02-19T18:51:57.196 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.439+0000 7f4712ffd700 1 -- 172.21.15.114:0/4013467663 learned_addr learned my addr 172.21.15.114:0/4013467663 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:57.196 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.439+0000 7f4712ffd700 1 -- 172.21.15.114:0/4013467663 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4714106530 con 0x7f4714105330 2024-02-19T18:51:57.196 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.439+0000 7f4712ffd700 1 --2- 172.21.15.114:0/4013467663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4714105330 0x7f4714105750 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f47080098a0 tx=0x7f4708009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48230245c449892e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:57.196 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.439+0000 7f4711ffb700 1 -- 172.21.15.114:0/4013467663 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4708004180 con 0x7f4714105330 2024-02-19T18:51:57.196 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.439+0000 7f4711ffb700 1 -- 172.21.15.114:0/4013467663 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f47080042e0 con 0x7f4714105330 2024-02-19T18:51:57.196 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.439+0000 7f4711ffb700 1 -- 172.21.15.114:0/4013467663 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4708011a50 con 0x7f4714105330 2024-02-19T18:51:57.197 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.440+0000 7f47195da700 1 -- 172.21.15.114:0/4013467663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4714105330 msgr2=0x7f4714105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:57.197 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.440+0000 7f47195da700 1 --2- 172.21.15.114:0/4013467663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4714105330 0x7f4714105750 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f47080098a0 tx=0x7f4708009bb0 comp rx=0 tx=0).stop 2024-02-19T18:51:57.197 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.440+0000 7f47195da700 1 -- 172.21.15.114:0/4013467663 shutdown_connections 2024-02-19T18:51:57.197 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.440+0000 7f47195da700 1 --2- 172.21.15.114:0/4013467663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4714105330 0x7f4714105750 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:57.197 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.440+0000 7f47195da700 1 -- 172.21.15.114:0/4013467663 >> 172.21.15.114:0/4013467663 conn(0x7f47141008b0 msgr2=0x7f4714102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:57.197 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.440+0000 7f47195da700 1 -- 172.21.15.114:0/4013467663 shutdown_connections 2024-02-19T18:51:57.197 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.440+0000 7f47195da700 1 -- 172.21.15.114:0/4013467663 wait complete. 2024-02-19T18:51:57.197 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.441+0000 7f47195da700 1 Processor -- start 2024-02-19T18:51:57.197 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.441+0000 7f47195da700 1 -- start start 2024-02-19T18:51:57.197 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.441+0000 7f47195da700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4714105330 0x7f4714190500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:57.197 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.441+0000 7f47195da700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4714190a40 con 0x7f4714105330 2024-02-19T18:51:57.197 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.441+0000 7f4712ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4714105330 0x7f4714190500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:57.197 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.441+0000 7f4712ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4714105330 0x7f4714190500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44748/0 (socket says 172.21.15.114:44748) 2024-02-19T18:51:57.197 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.441+0000 7f4712ffd700 1 -- 172.21.15.114:0/711299807 learned_addr learned my addr 172.21.15.114:0/711299807 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:57.197 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.441+0000 7f4712ffd700 1 -- 172.21.15.114:0/711299807 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4708009550 con 0x7f4714105330 2024-02-19T18:51:57.197 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.441+0000 7f4712ffd700 1 --2- 172.21.15.114:0/711299807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4714105330 0x7f4714190500 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4708009520 tx=0x7f4708011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:57.198 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.442+0000 7f46fbfff700 1 -- 172.21.15.114:0/711299807 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4708003a80 con 0x7f4714105330 2024-02-19T18:51:57.198 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.442+0000 7f46fbfff700 1 -- 172.21.15.114:0/711299807 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4708003be0 con 0x7f4714105330 2024-02-19T18:51:57.198 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.442+0000 7f47195da700 1 -- 172.21.15.114:0/711299807 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471418eb50 con 0x7f4714105330 2024-02-19T18:51:57.198 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.442+0000 7f46fbfff700 1 -- 172.21.15.114:0/711299807 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47080197b0 con 0x7f4714105330 2024-02-19T18:51:57.198 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.442+0000 7f47195da700 1 -- 172.21.15.114:0/711299807 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471418eff0 con 0x7f4714105330 2024-02-19T18:51:57.198 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.442+0000 7f46fbfff700 1 -- 172.21.15.114:0/711299807 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 3) v1 ==== 45166+0+0 (secure 0 0 0) 0x7f4708019a70 con 0x7f4714105330 2024-02-19T18:51:57.198 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.442+0000 7f47195da700 1 -- 172.21.15.114:0/711299807 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47140623c0 con 0x7f4714105330 2024-02-19T18:51:57.198 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.443+0000 7f46fbfff700 1 --2- 172.21.15.114:0/711299807 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f46fc038150 0x7f46fc03a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:57.198 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.443+0000 7f46fbfff700 1 -- 172.21.15.114:0/711299807 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f470804c250 con 0x7f4714105330 2024-02-19T18:51:57.198 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.443+0000 7f47127fc700 1 --2- 172.21.15.114:0/711299807 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f46fc038150 0x7f46fc03a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:57.198 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.443+0000 7f47127fc700 1 --2- 172.21.15.114:0/711299807 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f46fc038150 0x7f46fc03a610 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4704006fd0 tx=0x7f4704006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:57.198 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.447+0000 7f46fbfff700 1 -- 172.21.15.114:0/711299807 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f470802b750 con 0x7f4714105330 2024-02-19T18:51:57.198 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.632+0000 7f47195da700 1 -- 172.21.15.114:0/711299807 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f47141097b0 con 0x7f4714105330 2024-02-19T18:51:57.198 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.633+0000 7f46fbfff700 1 -- 172.21.15.114:0/711299807 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f47080172b0 con 0x7f4714105330 2024-02-19T18:51:57.198 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.634+0000 7f47195da700 1 -- 172.21.15.114:0/711299807 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f46fc038150 msgr2=0x7f46fc03a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:57.199 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.634+0000 7f47195da700 1 --2- 172.21.15.114:0/711299807 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f46fc038150 0x7f46fc03a610 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4704006fd0 tx=0x7f4704006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:57.199 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.634+0000 7f47195da700 1 -- 172.21.15.114:0/711299807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4714105330 msgr2=0x7f4714190500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:57.199 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.634+0000 7f47195da700 1 --2- 172.21.15.114:0/711299807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4714105330 0x7f4714190500 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4708009520 tx=0x7f4708011ab0 comp rx=0 tx=0).stop 2024-02-19T18:51:57.199 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.635+0000 7f47195da700 1 -- 172.21.15.114:0/711299807 shutdown_connections 2024-02-19T18:51:57.199 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.635+0000 7f47195da700 1 --2- 172.21.15.114:0/711299807 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f46fc038150 0x7f46fc03a610 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:57.199 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.635+0000 7f47195da700 1 --2- 172.21.15.114:0/711299807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4714105330 0x7f4714190500 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:57.199 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.635+0000 7f47195da700 1 -- 172.21.15.114:0/711299807 >> 172.21.15.114:0/711299807 conn(0x7f47141008b0 msgr2=0x7f47140798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:57.199 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.635+0000 7f47195da700 1 -- 172.21.15.114:0/711299807 shutdown_connections 2024-02-19T18:51:57.199 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:56.635+0000 7f47195da700 1 -- 172.21.15.114:0/711299807 wait complete. 2024-02-19T18:51:57.199 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling cephadm module... 2024-02-19T18:51:58.284 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:57 smithi114 ceph-mon[29441]: mgrmap e4: smithi114.pcvtrk(active, since 3s) 2024-02-19T18:51:58.284 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:57 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/184436061' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-19T18:51:59.450 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.613+0000 7fe89d0dc700 1 Processor -- start 2024-02-19T18:51:59.451 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.613+0000 7fe89d0dc700 1 -- start start 2024-02-19T18:51:59.451 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.613+0000 7fe89d0dc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe898106ee0 0x7fe8981092d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:59.451 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.613+0000 7fe89d0dc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe898079e80 con 0x7fe898106ee0 2024-02-19T18:51:59.451 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.613+0000 7fe896d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe898106ee0 0x7fe8981092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:59.451 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.613+0000 7fe896d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe898106ee0 0x7fe8981092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44762/0 (socket says 172.21.15.114:44762) 2024-02-19T18:51:59.451 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.613+0000 7fe896d9d700 1 -- 172.21.15.114:0/2171957827 learned_addr learned my addr 172.21.15.114:0/2171957827 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:59.451 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.613+0000 7fe896d9d700 1 -- 172.21.15.114:0/2171957827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe898109810 con 0x7fe898106ee0 2024-02-19T18:51:59.451 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.614+0000 7fe896d9d700 1 --2- 172.21.15.114:0/2171957827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe898106ee0 0x7fe8981092d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe8880098a0 tx=0x7fe888009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2c0b73989a28b5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:59.451 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.614+0000 7fe895d9b700 1 -- 172.21.15.114:0/2171957827 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe888004180 con 0x7fe898106ee0 2024-02-19T18:51:59.451 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.614+0000 7fe895d9b700 1 -- 172.21.15.114:0/2171957827 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe8880042e0 con 0x7fe898106ee0 2024-02-19T18:51:59.451 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.614+0000 7fe895d9b700 1 -- 172.21.15.114:0/2171957827 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe888011a50 con 0x7fe898106ee0 2024-02-19T18:51:59.452 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.614+0000 7fe89d0dc700 1 -- 172.21.15.114:0/2171957827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe898106ee0 msgr2=0x7fe8981092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:59.452 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.614+0000 7fe89d0dc700 1 --2- 172.21.15.114:0/2171957827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe898106ee0 0x7fe8981092d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe8880098a0 tx=0x7fe888009bb0 comp rx=0 tx=0).stop 2024-02-19T18:51:59.452 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.614+0000 7fe89d0dc700 1 -- 172.21.15.114:0/2171957827 shutdown_connections 2024-02-19T18:51:59.452 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.614+0000 7fe89d0dc700 1 --2- 172.21.15.114:0/2171957827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe898106ee0 0x7fe8981092d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:59.452 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.614+0000 7fe89d0dc700 1 -- 172.21.15.114:0/2171957827 >> 172.21.15.114:0/2171957827 conn(0x7fe8981008b0 msgr2=0x7fe898102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:59.452 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.614+0000 7fe89d0dc700 1 -- 172.21.15.114:0/2171957827 shutdown_connections 2024-02-19T18:51:59.452 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.614+0000 7fe89d0dc700 1 -- 172.21.15.114:0/2171957827 wait complete. 2024-02-19T18:51:59.452 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.615+0000 7fe89d0dc700 1 Processor -- start 2024-02-19T18:51:59.452 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.615+0000 7fe89d0dc700 1 -- start start 2024-02-19T18:51:59.452 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.615+0000 7fe89d0dc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe898106ee0 0x7fe89819a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:59.452 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.615+0000 7fe89d0dc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89819a680 con 0x7fe898106ee0 2024-02-19T18:51:59.453 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.616+0000 7fe896d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe898106ee0 0x7fe89819a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:59.453 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.616+0000 7fe896d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe898106ee0 0x7fe89819a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44768/0 (socket says 172.21.15.114:44768) 2024-02-19T18:51:59.453 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.616+0000 7fe896d9d700 1 -- 172.21.15.114:0/184436061 learned_addr learned my addr 172.21.15.114:0/184436061 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:51:59.453 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.616+0000 7fe896d9d700 1 -- 172.21.15.114:0/184436061 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe888009550 con 0x7fe898106ee0 2024-02-19T18:51:59.453 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.616+0000 7fe896d9d700 1 --2- 172.21.15.114:0/184436061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe898106ee0 0x7fe89819a140 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe888003f80 tx=0x7fe888011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:59.453 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.616+0000 7fe87ffff700 1 -- 172.21.15.114:0/184436061 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe888003a80 con 0x7fe898106ee0 2024-02-19T18:51:59.453 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.616+0000 7fe87ffff700 1 -- 172.21.15.114:0/184436061 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe888003be0 con 0x7fe898106ee0 2024-02-19T18:51:59.453 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.616+0000 7fe89d0dc700 1 -- 172.21.15.114:0/184436061 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89819a880 con 0x7fe898106ee0 2024-02-19T18:51:59.453 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.616+0000 7fe87ffff700 1 -- 172.21.15.114:0/184436061 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe888019820 con 0x7fe898106ee0 2024-02-19T18:51:59.453 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.616+0000 7fe89d0dc700 1 -- 172.21.15.114:0/184436061 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89819ad20 con 0x7fe898106ee0 2024-02-19T18:51:59.453 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.617+0000 7fe87ffff700 1 -- 172.21.15.114:0/184436061 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7fe888019a50 con 0x7fe898106ee0 2024-02-19T18:51:59.453 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.617+0000 7fe87ffff700 1 --2- 172.21.15.114:0/184436061 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7fe8800382d0 0x7fe88003a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:59.454 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.617+0000 7fe89d0dc700 1 -- 172.21.15.114:0/184436061 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89810a560 con 0x7fe898106ee0 2024-02-19T18:51:59.454 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.617+0000 7fe87ffff700 1 -- 172.21.15.114:0/184436061 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe88804c570 con 0x7fe898106ee0 2024-02-19T18:51:59.454 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.617+0000 7fe89659c700 1 --2- 172.21.15.114:0/184436061 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7fe8800382d0 0x7fe88003a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:59.454 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.618+0000 7fe89659c700 1 --2- 172.21.15.114:0/184436061 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7fe8800382d0 0x7fe88003a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe88c006fd0 tx=0x7fe88c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:59.454 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.622+0000 7fe87ffff700 1 -- 172.21.15.114:0/184436061 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe88802c3a0 con 0x7fe898106ee0 2024-02-19T18:51:59.454 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:57.862+0000 7fe89d0dc700 1 -- 172.21.15.114:0/184436061 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fe89819db00 con 0x7fe898106ee0 2024-02-19T18:51:59.454 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:58.838+0000 7fe87ffff700 1 -- 172.21.15.114:0/184436061 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fe888016d40 con 0x7fe898106ee0 2024-02-19T18:51:59.454 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:58.839+0000 7fe87ffff700 1 -- 172.21.15.114:0/184436061 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fe88802b340 con 0x7fe898106ee0 2024-02-19T18:51:59.454 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:58.842+0000 7fe89d0dc700 1 -- 172.21.15.114:0/184436061 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7fe8800382d0 msgr2=0x7fe88003a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:59.454 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:58.842+0000 7fe89d0dc700 1 --2- 172.21.15.114:0/184436061 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7fe8800382d0 0x7fe88003a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe88c006fd0 tx=0x7fe88c006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:59.454 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:58.842+0000 7fe89d0dc700 1 -- 172.21.15.114:0/184436061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe898106ee0 msgr2=0x7fe89819a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:59.454 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:58.842+0000 7fe89d0dc700 1 --2- 172.21.15.114:0/184436061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe898106ee0 0x7fe89819a140 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe888003f80 tx=0x7fe888011ab0 comp rx=0 tx=0).stop 2024-02-19T18:51:59.455 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:58.842+0000 7fe89d0dc700 1 -- 172.21.15.114:0/184436061 shutdown_connections 2024-02-19T18:51:59.455 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:58.842+0000 7fe89d0dc700 1 --2- 172.21.15.114:0/184436061 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7fe8800382d0 0x7fe88003a790 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:59.455 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:58.842+0000 7fe89d0dc700 1 --2- 172.21.15.114:0/184436061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe898106ee0 0x7fe89819a140 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:59.455 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:58.842+0000 7fe89d0dc700 1 -- 172.21.15.114:0/184436061 >> 172.21.15.114:0/184436061 conn(0x7fe8981008b0 msgr2=0x7fe898101fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:59.455 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:58.842+0000 7fe89d0dc700 1 -- 172.21.15.114:0/184436061 shutdown_connections 2024-02-19T18:51:59.455 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:58.842+0000 7fe89d0dc700 1 -- 172.21.15.114:0/184436061 wait complete. 2024-02-19T18:52:00.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:59 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/184436061' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-19T18:52:00.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:51:59 smithi114 ceph-mon[29441]: mgrmap e5: smithi114.pcvtrk(active, since 4s) 2024-02-19T18:52:00.724 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:52:00.724 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-19T18:52:00.724 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-19T18:52:00.724 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "active_name": "smithi114.pcvtrk", 2024-02-19T18:52:00.724 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-19T18:52:00.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:52:00.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.862+0000 7f7caa9bc700 1 Processor -- start 2024-02-19T18:52:00.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.862+0000 7f7caa9bc700 1 -- start start 2024-02-19T18:52:00.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.862+0000 7f7caa9bc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ca4106f20 0x7f7ca4109310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:00.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.862+0000 7f7caa9bc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca4079e60 con 0x7f7ca4106f20 2024-02-19T18:52:00.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.862+0000 7f7ca3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ca4106f20 0x7f7ca4109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:00.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.862+0000 7f7ca3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ca4106f20 0x7f7ca4109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44802/0 (socket says 172.21.15.114:44802) 2024-02-19T18:52:00.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.862+0000 7f7ca3fff700 1 -- 172.21.15.114:0/1098006085 learned_addr learned my addr 172.21.15.114:0/1098006085 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:00.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.862+0000 7f7ca3fff700 1 -- 172.21.15.114:0/1098006085 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca4109850 con 0x7f7ca4106f20 2024-02-19T18:52:00.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.863+0000 7f7ca3fff700 1 --2- 172.21.15.114:0/1098006085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ca4106f20 0x7f7ca4109310 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7c940098a0 tx=0x7f7c94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220e797ac0597324 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:00.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.863+0000 7f7ca2ffd700 1 -- 172.21.15.114:0/1098006085 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c94004180 con 0x7f7ca4106f20 2024-02-19T18:52:00.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.863+0000 7f7ca2ffd700 1 -- 172.21.15.114:0/1098006085 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7c940042e0 con 0x7f7ca4106f20 2024-02-19T18:52:00.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.863+0000 7f7ca2ffd700 1 -- 172.21.15.114:0/1098006085 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c94011a50 con 0x7f7ca4106f20 2024-02-19T18:52:00.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.863+0000 7f7caa9bc700 1 -- 172.21.15.114:0/1098006085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ca4106f20 msgr2=0x7f7ca4109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:00.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.863+0000 7f7caa9bc700 1 --2- 172.21.15.114:0/1098006085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ca4106f20 0x7f7ca4109310 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7c940098a0 tx=0x7f7c94009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:00.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.863+0000 7f7caa9bc700 1 -- 172.21.15.114:0/1098006085 shutdown_connections 2024-02-19T18:52:00.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.863+0000 7f7caa9bc700 1 --2- 172.21.15.114:0/1098006085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ca4106f20 0x7f7ca4109310 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:00.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.863+0000 7f7caa9bc700 1 -- 172.21.15.114:0/1098006085 >> 172.21.15.114:0/1098006085 conn(0x7f7ca41008d0 msgr2=0x7f7ca4102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:00.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.864+0000 7f7caa9bc700 1 -- 172.21.15.114:0/1098006085 shutdown_connections 2024-02-19T18:52:00.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.864+0000 7f7caa9bc700 1 -- 172.21.15.114:0/1098006085 wait complete. 2024-02-19T18:52:00.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.864+0000 7f7caa9bc700 1 Processor -- start 2024-02-19T18:52:00.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.864+0000 7f7caa9bc700 1 -- start start 2024-02-19T18:52:00.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.864+0000 7f7caa9bc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ca4106f20 0x7f7ca419c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:00.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.864+0000 7f7caa9bc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca419c830 con 0x7f7ca4106f20 2024-02-19T18:52:00.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.864+0000 7f7ca3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ca4106f20 0x7f7ca419c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:00.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.865+0000 7f7ca3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ca4106f20 0x7f7ca419c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44814/0 (socket says 172.21.15.114:44814) 2024-02-19T18:52:00.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.865+0000 7f7ca3fff700 1 -- 172.21.15.114:0/680480656 learned_addr learned my addr 172.21.15.114:0/680480656 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:00.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.865+0000 7f7ca3fff700 1 -- 172.21.15.114:0/680480656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c94009550 con 0x7f7ca4106f20 2024-02-19T18:52:00.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.865+0000 7f7ca3fff700 1 --2- 172.21.15.114:0/680480656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ca4106f20 0x7f7ca419c2f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7c94003de0 tx=0x7f7c94011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:00.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.865+0000 7f7ca17fa700 1 -- 172.21.15.114:0/680480656 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c94003930 con 0x7f7ca4106f20 2024-02-19T18:52:00.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.865+0000 7f7ca17fa700 1 -- 172.21.15.114:0/680480656 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f7c94003a90 con 0x7f7ca4106f20 2024-02-19T18:52:00.729 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.865+0000 7f7caa9bc700 1 -- 172.21.15.114:0/680480656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca419ca30 con 0x7f7ca4106f20 2024-02-19T18:52:00.729 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.865+0000 7f7ca17fa700 1 -- 172.21.15.114:0/680480656 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c94019730 con 0x7f7ca4106f20 2024-02-19T18:52:00.729 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.865+0000 7f7caa9bc700 1 -- 172.21.15.114:0/680480656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca419ced0 con 0x7f7ca4106f20 2024-02-19T18:52:00.729 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.866+0000 7f7ca17fa700 1 -- 172.21.15.114:0/680480656 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f7c94019980 con 0x7f7ca4106f20 2024-02-19T18:52:00.729 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.866+0000 7f7caa9bc700 1 -- 172.21.15.114:0/680480656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca410a570 con 0x7f7ca4106f20 2024-02-19T18:52:00.730 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.866+0000 7f7ca17fa700 1 --2- 172.21.15.114:0/680480656 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f7c8c038370 0x7f7c8c03a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:00.730 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.866+0000 7f7ca17fa700 1 -- 172.21.15.114:0/680480656 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7c9404c070 con 0x7f7ca4106f20 2024-02-19T18:52:00.730 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.866+0000 7f7ca37fe700 1 -- 172.21.15.114:0/680480656 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f7c8c038370 msgr2=0x7f7c8c03a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/744561179 2024-02-19T18:52:00.730 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.866+0000 7f7ca37fe700 1 --2- 172.21.15.114:0/680480656 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f7c8c038370 0x7f7c8c03a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:52:00.730 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:51:59.871+0000 7f7ca17fa700 1 -- 172.21.15.114:0/680480656 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7c94010b70 con 0x7f7ca4106f20 2024-02-19T18:52:00.730 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:00.067+0000 7f7ca37fe700 1 -- 172.21.15.114:0/680480656 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f7c8c038370 msgr2=0x7f7c8c03a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/744561179 2024-02-19T18:52:00.731 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:00.067+0000 7f7ca37fe700 1 --2- 172.21.15.114:0/680480656 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f7c8c038370 0x7f7c8c03a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T18:52:00.731 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:00.122+0000 7f7caa9bc700 1 -- 172.21.15.114:0/680480656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7ca419fd10 con 0x7f7ca4106f20 2024-02-19T18:52:00.731 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:00.122+0000 7f7ca17fa700 1 -- 172.21.15.114:0/680480656 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f7c94020220 con 0x7f7ca4106f20 2024-02-19T18:52:00.731 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:00.124+0000 7f7caa9bc700 1 -- 172.21.15.114:0/680480656 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f7c8c038370 msgr2=0x7f7c8c03a830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:52:00.731 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:00.125+0000 7f7caa9bc700 1 --2- 172.21.15.114:0/680480656 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f7c8c038370 0x7f7c8c03a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:00.732 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:00.125+0000 7f7caa9bc700 1 -- 172.21.15.114:0/680480656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ca4106f20 msgr2=0x7f7ca419c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:00.732 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:00.125+0000 7f7caa9bc700 1 --2- 172.21.15.114:0/680480656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ca4106f20 0x7f7ca419c2f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7c94003de0 tx=0x7f7c94011ab0 comp rx=0 tx=0).stop 2024-02-19T18:52:00.732 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:00.125+0000 7f7caa9bc700 1 -- 172.21.15.114:0/680480656 shutdown_connections 2024-02-19T18:52:00.732 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:00.125+0000 7f7caa9bc700 1 --2- 172.21.15.114:0/680480656 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f7c8c038370 0x7f7c8c03a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:00.732 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:00.125+0000 7f7caa9bc700 1 --2- 172.21.15.114:0/680480656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ca4106f20 0x7f7ca419c2f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:00.732 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:00.125+0000 7f7caa9bc700 1 -- 172.21.15.114:0/680480656 >> 172.21.15.114:0/680480656 conn(0x7f7ca41008d0 msgr2=0x7f7ca41016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:00.733 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:00.125+0000 7f7caa9bc700 1 -- 172.21.15.114:0/680480656 shutdown_connections 2024-02-19T18:52:00.733 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:00.125+0000 7f7caa9bc700 1 -- 172.21.15.114:0/680480656 wait complete. 2024-02-19T18:52:00.733 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for the mgr to restart... 2024-02-19T18:52:00.733 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr epoch 5... 2024-02-19T18:52:00.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:00 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/680480656' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T18:52:06.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:05 smithi114 ceph-mon[29441]: Active manager daemon smithi114.pcvtrk restarted 2024-02-19T18:52:06.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:05 smithi114 ceph-mon[29441]: Activating manager daemon smithi114.pcvtrk 2024-02-19T18:52:06.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:05 smithi114 ceph-mon[29441]: osdmap e2: 0 total, 0 up, 0 in 2024-02-19T18:52:06.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:05 smithi114 ceph-mon[29441]: mgrmap e6: smithi114.pcvtrk(active, starting, since 0.00321384s) 2024-02-19T18:52:06.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:05 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T18:52:06.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:05 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr metadata", "who": "smithi114.pcvtrk", "id": "smithi114.pcvtrk"}]: dispatch 2024-02-19T18:52:06.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:05 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T18:52:06.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:05 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T18:52:06.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:05 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T18:52:06.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:05 smithi114 ceph-mon[29441]: Manager daemon smithi114.pcvtrk is now available 2024-02-19T18:52:06.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:05 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:06.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:05 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:06.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:05 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:52:06.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:05 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:52:06.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:05 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:52:06.863 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:06 smithi114 ceph-mon[29441]: Found migration_current of "None". Setting to last migration. 2024-02-19T18:52:06.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:06 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T18:52:06.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:06 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/trash_purge_schedule"}]: dispatch 2024-02-19T18:52:06.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:06 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:06.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:06 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:06.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:06 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:52:06.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:06 smithi114 ceph-mon[29441]: mgrmap e7: smithi114.pcvtrk(active, since 1.00572s) 2024-02-19T18:52:07.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:52:07.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-19T18:52:07.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-19T18:52:07.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:52:07.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.136+0000 7f2d9ef60700 1 Processor -- start 2024-02-19T18:52:07.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.136+0000 7f2d9ef60700 1 -- start start 2024-02-19T18:52:07.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.136+0000 7f2d9ef60700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d98106ef0 0x7f2d981092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:07.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.136+0000 7f2d9ef60700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d98074640 con 0x7f2d98106ef0 2024-02-19T18:52:07.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.136+0000 7f2d9ccfc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d98106ef0 0x7f2d981092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:07.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.136+0000 7f2d9ccfc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d98106ef0 0x7f2d981092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44816/0 (socket says 172.21.15.114:44816) 2024-02-19T18:52:07.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.136+0000 7f2d9ccfc700 1 -- 172.21.15.114:0/1264448099 learned_addr learned my addr 172.21.15.114:0/1264448099 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:07.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.136+0000 7f2d9ccfc700 1 -- 172.21.15.114:0/1264448099 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d98109820 con 0x7f2d98106ef0 2024-02-19T18:52:07.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.137+0000 7f2d9ccfc700 1 --2- 172.21.15.114:0/1264448099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d98106ef0 0x7f2d981092e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0098a0 tx=0x7f2d8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ef48b8f5657cd4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:07.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.137+0000 7f2d977fe700 1 -- 172.21.15.114:0/1264448099 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d8c004180 con 0x7f2d98106ef0 2024-02-19T18:52:07.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.137+0000 7f2d977fe700 1 -- 172.21.15.114:0/1264448099 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2d8c0042e0 con 0x7f2d98106ef0 2024-02-19T18:52:07.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.137+0000 7f2d977fe700 1 -- 172.21.15.114:0/1264448099 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d8c011a50 con 0x7f2d98106ef0 2024-02-19T18:52:07.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.137+0000 7f2d9ef60700 1 -- 172.21.15.114:0/1264448099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d98106ef0 msgr2=0x7f2d981092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:07.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.137+0000 7f2d9ef60700 1 --2- 172.21.15.114:0/1264448099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d98106ef0 0x7f2d981092e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0098a0 tx=0x7f2d8c009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:07.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.138+0000 7f2d9ef60700 1 -- 172.21.15.114:0/1264448099 shutdown_connections 2024-02-19T18:52:07.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.138+0000 7f2d9ef60700 1 --2- 172.21.15.114:0/1264448099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d98106ef0 0x7f2d981092e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:07.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.138+0000 7f2d9ef60700 1 -- 172.21.15.114:0/1264448099 >> 172.21.15.114:0/1264448099 conn(0x7f2d981008c0 msgr2=0x7f2d98102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:07.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.138+0000 7f2d9ef60700 1 -- 172.21.15.114:0/1264448099 shutdown_connections 2024-02-19T18:52:07.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.138+0000 7f2d9ef60700 1 -- 172.21.15.114:0/1264448099 wait complete. 2024-02-19T18:52:07.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.138+0000 7f2d9ef60700 1 Processor -- start 2024-02-19T18:52:07.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.138+0000 7f2d9ef60700 1 -- start start 2024-02-19T18:52:07.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.138+0000 7f2d9ef60700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d98106ef0 0x7f2d9819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:07.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.138+0000 7f2d9ef60700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9819a660 con 0x7f2d98106ef0 2024-02-19T18:52:07.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.138+0000 7f2d9ccfc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d98106ef0 0x7f2d9819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:07.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.138+0000 7f2d9ccfc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d98106ef0 0x7f2d9819a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44826/0 (socket says 172.21.15.114:44826) 2024-02-19T18:52:07.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.138+0000 7f2d9ccfc700 1 -- 172.21.15.114:0/4197374937 learned_addr learned my addr 172.21.15.114:0/4197374937 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:07.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.139+0000 7f2d9ccfc700 1 -- 172.21.15.114:0/4197374937 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c009550 con 0x7f2d98106ef0 2024-02-19T18:52:07.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.139+0000 7f2d9ccfc700 1 --2- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d98106ef0 0x7f2d9819a120 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c003f80 tx=0x7f2d8c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:07.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.139+0000 7f2d95ffb700 1 -- 172.21.15.114:0/4197374937 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d8c003a80 con 0x7f2d98106ef0 2024-02-19T18:52:07.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.139+0000 7f2d95ffb700 1 -- 172.21.15.114:0/4197374937 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2d8c003be0 con 0x7f2d98106ef0 2024-02-19T18:52:07.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.139+0000 7f2d9ef60700 1 -- 172.21.15.114:0/4197374937 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d9819a860 con 0x7f2d98106ef0 2024-02-19T18:52:07.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.139+0000 7f2d95ffb700 1 -- 172.21.15.114:0/4197374937 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d8c019820 con 0x7f2d98106ef0 2024-02-19T18:52:07.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.139+0000 7f2d9ef60700 1 -- 172.21.15.114:0/4197374937 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d9819ad00 con 0x7f2d98106ef0 2024-02-19T18:52:07.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.140+0000 7f2d95ffb700 1 -- 172.21.15.114:0/4197374937 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f2d8c01a430 con 0x7f2d98106ef0 2024-02-19T18:52:07.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.141+0000 7f2d95ffb700 1 --2- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f2d88038320 0x7f2d8803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:07.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.141+0000 7f2d95ffb700 1 -- 172.21.15.114:0/4197374937 --> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d8803aef0 con 0x7f2d88038320 2024-02-19T18:52:07.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.141+0000 7f2d95ffb700 1 -- 172.21.15.114:0/4197374937 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2d8c04c540 con 0x7f2d98106ef0 2024-02-19T18:52:07.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.141+0000 7f2d97fff700 1 -- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f2d88038320 msgr2=0x7f2d8803a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/744561179 2024-02-19T18:52:07.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.141+0000 7f2d97fff700 1 --2- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f2d88038320 0x7f2d8803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:52:07.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.342+0000 7f2d97fff700 1 -- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f2d88038320 msgr2=0x7f2d8803a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/744561179 2024-02-19T18:52:07.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.342+0000 7f2d97fff700 1 --2- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f2d88038320 0x7f2d8803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T18:52:07.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.743+0000 7f2d97fff700 1 -- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f2d88038320 msgr2=0x7f2d8803a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/744561179 2024-02-19T18:52:07.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:01.743+0000 7f2d97fff700 1 --2- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f2d88038320 0x7f2d8803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-19T18:52:07.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:02.544+0000 7f2d97fff700 1 -- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f2d88038320 msgr2=0x7f2d8803a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/744561179 2024-02-19T18:52:07.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:02.544+0000 7f2d97fff700 1 --2- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f2d88038320 0x7f2d8803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-19T18:52:07.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:04.145+0000 7f2d97fff700 1 -- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f2d88038320 msgr2=0x7f2d8803a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/744561179 2024-02-19T18:52:07.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:04.145+0000 7f2d97fff700 1 --2- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f2d88038320 0x7f2d8803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-19T18:52:07.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:05.550+0000 7f2d95ffb700 1 -- 172.21.15.114:0/4197374937 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mgrmap(e 6) v1 ==== 45050+0+0 (secure 0 0 0) 0x7f2d8c02d440 con 0x7f2d98106ef0 2024-02-19T18:52:07.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:05.550+0000 7f2d95ffb700 1 -- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f2d88038320 msgr2=0x7f2d8803a7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:52:07.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:05.550+0000 7f2d95ffb700 1 --2- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f2d88038320 0x7f2d8803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:07.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.551+0000 7f2d95ffb700 1 -- 172.21.15.114:0/4197374937 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 7) v1 ==== 45177+0+0 (secure 0 0 0) 0x7f2d8c04de00 con 0x7f2d98106ef0 2024-02-19T18:52:07.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.551+0000 7f2d95ffb700 1 --2- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2d8803bc00 0x7f2d8803dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:07.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.551+0000 7f2d95ffb700 1 -- 172.21.15.114:0/4197374937 --> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d8803aef0 con 0x7f2d8803bc00 2024-02-19T18:52:07.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.551+0000 7f2d97fff700 1 --2- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2d8803bc00 0x7f2d8803dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:07.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.552+0000 7f2d97fff700 1 --2- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2d8803bc00 0x7f2d8803dff0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2d84003a10 tx=0x7f2d840092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:07.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.553+0000 7f2d95ffb700 1 -- 172.21.15.114:0/4197374937 <== mgr.14118 v2:172.21.15.114:6800/971741305 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f2d8803aef0 con 0x7f2d8803bc00 2024-02-19T18:52:07.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.559+0000 7f2d9ef60700 1 -- 172.21.15.114:0/4197374937 --> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f2d9819b460 con 0x7f2d8803bc00 2024-02-19T18:52:07.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.559+0000 7f2d95ffb700 1 -- 172.21.15.114:0/4197374937 <== mgr.14118 v2:172.21.15.114:6800/971741305 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f2d9819b460 con 0x7f2d8803bc00 2024-02-19T18:52:07.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.560+0000 7f2d9ef60700 1 -- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2d8803bc00 msgr2=0x7f2d8803dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:07.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.560+0000 7f2d9ef60700 1 --2- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2d8803bc00 0x7f2d8803dff0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2d84003a10 tx=0x7f2d840092b0 comp rx=0 tx=0).stop 2024-02-19T18:52:07.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.560+0000 7f2d9ef60700 1 -- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d98106ef0 msgr2=0x7f2d9819a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:07.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.560+0000 7f2d9ef60700 1 --2- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d98106ef0 0x7f2d9819a120 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c003f80 tx=0x7f2d8c011ab0 comp rx=0 tx=0).stop 2024-02-19T18:52:07.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.560+0000 7f2d9ef60700 1 -- 172.21.15.114:0/4197374937 shutdown_connections 2024-02-19T18:52:07.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.560+0000 7f2d9ef60700 1 --2- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2d8803bc00 0x7f2d8803dff0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:07.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.560+0000 7f2d9ef60700 1 --2- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d98106ef0 0x7f2d9819a120 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:07.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.560+0000 7f2d9ef60700 1 --2- 172.21.15.114:0/4197374937 >> [v2:172.21.15.114:6800/744561179,v1:172.21.15.114:6801/744561179] conn(0x7f2d88038320 0x7f2d8803a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:07.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.560+0000 7f2d9ef60700 1 -- 172.21.15.114:0/4197374937 >> 172.21.15.114:0/4197374937 conn(0x7f2d981008c0 msgr2=0x7f2d98101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:07.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.560+0000 7f2d9ef60700 1 -- 172.21.15.114:0/4197374937 shutdown_connections 2024-02-19T18:52:07.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:06.560+0000 7f2d9ef60700 1 -- 172.21.15.114:0/4197374937 wait complete. 2024-02-19T18:52:07.156 INFO:teuthology.orchestra.run.smithi114.stdout:mgr epoch 5 is available 2024-02-19T18:52:07.156 INFO:teuthology.orchestra.run.smithi114.stdout:Setting orchestrator backend to cephadm... 2024-02-19T18:52:07.783 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:07 smithi114 ceph-mon[29441]: [19/Feb/2024:18:52:06] ENGINE Bus STARTING 2024-02-19T18:52:07.783 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:07 smithi114 ceph-mon[29441]: [19/Feb/2024:18:52:06] ENGINE Serving on http://172.21.15.114:8765 2024-02-19T18:52:07.783 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:07 smithi114 ceph-mon[29441]: [19/Feb/2024:18:52:06] ENGINE Serving on https://172.21.15.114:7150 2024-02-19T18:52:07.783 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:07 smithi114 ceph-mon[29441]: [19/Feb/2024:18:52:06] ENGINE Bus STARTED 2024-02-19T18:52:07.783 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:07 smithi114 ceph-mon[29441]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T18:52:07.783 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:07 smithi114 ceph-mon[29441]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T18:52:08.353 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.553+0000 7fa8194f0700 1 Processor -- start 2024-02-19T18:52:08.353 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.553+0000 7fa8194f0700 1 -- start start 2024-02-19T18:52:08.354 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.554+0000 7fa8194f0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa814104ac0 0x7fa814104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:08.354 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.554+0000 7fa8194f0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8141054b0 con 0x7fa814104ac0 2024-02-19T18:52:08.354 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.554+0000 7fa812ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa814104ac0 0x7fa814104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:08.354 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.554+0000 7fa812ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa814104ac0 0x7fa814104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40202/0 (socket says 172.21.15.114:40202) 2024-02-19T18:52:08.354 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.554+0000 7fa812ffd700 1 -- 172.21.15.114:0/1127765474 learned_addr learned my addr 172.21.15.114:0/1127765474 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:08.354 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.554+0000 7fa812ffd700 1 -- 172.21.15.114:0/1127765474 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa814105cc0 con 0x7fa814104ac0 2024-02-19T18:52:08.354 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.554+0000 7fa812ffd700 1 --2- 172.21.15.114:0/1127765474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa814104ac0 0x7fa814104ee0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa8040098a0 tx=0x7fa804009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9758ba942acce5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:08.354 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.554+0000 7fa8127fc700 1 -- 172.21.15.114:0/1127765474 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa804004180 con 0x7fa814104ac0 2024-02-19T18:52:08.354 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.554+0000 7fa8127fc700 1 -- 172.21.15.114:0/1127765474 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa8040042e0 con 0x7fa814104ac0 2024-02-19T18:52:08.354 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.555+0000 7fa8127fc700 1 -- 172.21.15.114:0/1127765474 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa804011a50 con 0x7fa814104ac0 2024-02-19T18:52:08.354 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.555+0000 7fa8194f0700 1 -- 172.21.15.114:0/1127765474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa814104ac0 msgr2=0x7fa814104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:08.354 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.555+0000 7fa8194f0700 1 --2- 172.21.15.114:0/1127765474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa814104ac0 0x7fa814104ee0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa8040098a0 tx=0x7fa804009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:08.354 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.555+0000 7fa8194f0700 1 -- 172.21.15.114:0/1127765474 shutdown_connections 2024-02-19T18:52:08.355 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.555+0000 7fa8194f0700 1 --2- 172.21.15.114:0/1127765474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa814104ac0 0x7fa814104ee0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:08.355 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.555+0000 7fa8194f0700 1 -- 172.21.15.114:0/1127765474 >> 172.21.15.114:0/1127765474 conn(0x7fa814100040 msgr2=0x7fa8141024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:08.355 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.555+0000 7fa8194f0700 1 -- 172.21.15.114:0/1127765474 shutdown_connections 2024-02-19T18:52:08.355 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.555+0000 7fa8194f0700 1 -- 172.21.15.114:0/1127765474 wait complete. 2024-02-19T18:52:08.355 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.556+0000 7fa8194f0700 1 Processor -- start 2024-02-19T18:52:08.355 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.556+0000 7fa8194f0700 1 -- start start 2024-02-19T18:52:08.355 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.556+0000 7fa8194f0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa814104ac0 0x7fa81419c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:08.355 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.556+0000 7fa8194f0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81419c890 con 0x7fa814104ac0 2024-02-19T18:52:08.355 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.556+0000 7fa812ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa814104ac0 0x7fa81419c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:08.355 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.556+0000 7fa812ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa814104ac0 0x7fa81419c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40204/0 (socket says 172.21.15.114:40204) 2024-02-19T18:52:08.355 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.556+0000 7fa812ffd700 1 -- 172.21.15.114:0/3941066567 learned_addr learned my addr 172.21.15.114:0/3941066567 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:08.355 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.556+0000 7fa812ffd700 1 -- 172.21.15.114:0/3941066567 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa804009550 con 0x7fa814104ac0 2024-02-19T18:52:08.355 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.556+0000 7fa812ffd700 1 --2- 172.21.15.114:0/3941066567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa814104ac0 0x7fa81419c350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa804009520 tx=0x7fa804011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:08.355 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.557+0000 7fa810ff9700 1 -- 172.21.15.114:0/3941066567 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa804003a80 con 0x7fa814104ac0 2024-02-19T18:52:08.356 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.557+0000 7fa810ff9700 1 -- 172.21.15.114:0/3941066567 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa804003be0 con 0x7fa814104ac0 2024-02-19T18:52:08.356 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.557+0000 7fa810ff9700 1 -- 172.21.15.114:0/3941066567 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8040197b0 con 0x7fa814104ac0 2024-02-19T18:52:08.356 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.557+0000 7fa8194f0700 1 -- 172.21.15.114:0/3941066567 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81419ca90 con 0x7fa814104ac0 2024-02-19T18:52:08.356 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.557+0000 7fa8194f0700 1 -- 172.21.15.114:0/3941066567 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81419ce70 con 0x7fa814104ac0 2024-02-19T18:52:08.356 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.558+0000 7fa810ff9700 1 -- 172.21.15.114:0/3941066567 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 7) v1 ==== 45177+0+0 (secure 0 0 0) 0x7fa804019a20 con 0x7fa814104ac0 2024-02-19T18:52:08.356 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.558+0000 7fa810ff9700 1 --2- 172.21.15.114:0/3941066567 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fa8000381f0 0x7fa80003a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:08.356 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.558+0000 7fa810ff9700 1 -- 172.21.15.114:0/3941066567 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa80404c320 con 0x7fa814104ac0 2024-02-19T18:52:08.356 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.558+0000 7fa8194f0700 1 -- 172.21.15.114:0/3941066567 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa814195c20 con 0x7fa814104ac0 2024-02-19T18:52:08.356 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.558+0000 7fa80bfff700 1 --2- 172.21.15.114:0/3941066567 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fa8000381f0 0x7fa80003a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:08.356 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.558+0000 7fa80bfff700 1 --2- 172.21.15.114:0/3941066567 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fa8000381f0 0x7fa80003a6b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc006fd0 tx=0x7fa7fc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:08.356 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.563+0000 7fa810ff9700 1 -- 172.21.15.114:0/3941066567 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa80402d440 con 0x7fa814104ac0 2024-02-19T18:52:08.356 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.602+0000 7fa810ff9700 1 -- 172.21.15.114:0/3941066567 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fa80401a950 con 0x7fa814104ac0 2024-02-19T18:52:08.357 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.766+0000 7fa8194f0700 1 -- 172.21.15.114:0/3941066567 --> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8140611d0 con 0x7fa8000381f0 2024-02-19T18:52:08.357 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.774+0000 7fa810ff9700 1 -- 172.21.15.114:0/3941066567 <== mgr.14118 v2:172.21.15.114:6800/971741305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa8140611d0 con 0x7fa8000381f0 2024-02-19T18:52:08.357 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.777+0000 7fa8194f0700 1 -- 172.21.15.114:0/3941066567 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fa8000381f0 msgr2=0x7fa80003a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:08.357 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.777+0000 7fa8194f0700 1 --2- 172.21.15.114:0/3941066567 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fa8000381f0 0x7fa80003a6b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc006fd0 tx=0x7fa7fc006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:08.357 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.777+0000 7fa8194f0700 1 -- 172.21.15.114:0/3941066567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa814104ac0 msgr2=0x7fa81419c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:08.357 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.777+0000 7fa8194f0700 1 --2- 172.21.15.114:0/3941066567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa814104ac0 0x7fa81419c350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa804009520 tx=0x7fa804011ab0 comp rx=0 tx=0).stop 2024-02-19T18:52:08.357 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.777+0000 7fa8194f0700 1 -- 172.21.15.114:0/3941066567 shutdown_connections 2024-02-19T18:52:08.357 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.777+0000 7fa8194f0700 1 --2- 172.21.15.114:0/3941066567 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fa8000381f0 0x7fa80003a6b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:08.357 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.777+0000 7fa8194f0700 1 --2- 172.21.15.114:0/3941066567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa814104ac0 0x7fa81419c350 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:08.357 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.777+0000 7fa8194f0700 1 -- 172.21.15.114:0/3941066567 >> 172.21.15.114:0/3941066567 conn(0x7fa814100040 msgr2=0x7fa814190890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:08.357 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.777+0000 7fa8194f0700 1 -- 172.21.15.114:0/3941066567 shutdown_connections 2024-02-19T18:52:08.357 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:07.777+0000 7fa8194f0700 1 -- 172.21.15.114:0/3941066567 wait complete. 2024-02-19T18:52:08.732 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:08 smithi114 ceph-mon[29441]: mgrmap e8: smithi114.pcvtrk(active, since 2s) 2024-02-19T18:52:08.733 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:08 smithi114 ceph-mon[29441]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:52:08.733 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:08 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:08.733 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:08 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:52:09.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-19T18:52:09.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.749+0000 7f7583f90700 1 Processor -- start 2024-02-19T18:52:09.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.749+0000 7f7583f90700 1 -- start start 2024-02-19T18:52:09.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.749+0000 7f7583f90700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757c106420 0x7f757c106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:09.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.749+0000 7f7583f90700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c106e10 con 0x7f757c106420 2024-02-19T18:52:09.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.749+0000 7f7581d2c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757c106420 0x7f757c106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:09.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.749+0000 7f7581d2c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757c106420 0x7f757c106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40208/0 (socket says 172.21.15.114:40208) 2024-02-19T18:52:09.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.749+0000 7f7581d2c700 1 -- 172.21.15.114:0/3621910314 learned_addr learned my addr 172.21.15.114:0/3621910314 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:09.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.750+0000 7f7581d2c700 1 -- 172.21.15.114:0/3621910314 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c107620 con 0x7f757c106420 2024-02-19T18:52:09.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.750+0000 7f7581d2c700 1 --2- 172.21.15.114:0/3621910314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757c106420 0x7f757c106840 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f75780098a0 tx=0x7f7578009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d232320d7222c880 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:09.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.750+0000 7f7580d2a700 1 -- 172.21.15.114:0/3621910314 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7578004180 con 0x7f757c106420 2024-02-19T18:52:09.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.750+0000 7f7580d2a700 1 -- 172.21.15.114:0/3621910314 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f75780042e0 con 0x7f757c106420 2024-02-19T18:52:09.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.750+0000 7f7580d2a700 1 -- 172.21.15.114:0/3621910314 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7578011a50 con 0x7f757c106420 2024-02-19T18:52:09.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.750+0000 7f7583f90700 1 -- 172.21.15.114:0/3621910314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757c106420 msgr2=0x7f757c106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:09.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.750+0000 7f7583f90700 1 --2- 172.21.15.114:0/3621910314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757c106420 0x7f757c106840 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f75780098a0 tx=0x7f7578009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:09.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.751+0000 7f7583f90700 1 -- 172.21.15.114:0/3621910314 shutdown_connections 2024-02-19T18:52:09.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.751+0000 7f7583f90700 1 --2- 172.21.15.114:0/3621910314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757c106420 0x7f757c106840 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:09.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.751+0000 7f7583f90700 1 -- 172.21.15.114:0/3621910314 >> 172.21.15.114:0/3621910314 conn(0x7f757c101a00 msgr2=0x7f757c103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:09.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.751+0000 7f7583f90700 1 -- 172.21.15.114:0/3621910314 shutdown_connections 2024-02-19T18:52:09.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.751+0000 7f7583f90700 1 -- 172.21.15.114:0/3621910314 wait complete. 2024-02-19T18:52:09.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.751+0000 7f7583f90700 1 Processor -- start 2024-02-19T18:52:09.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.751+0000 7f7583f90700 1 -- start start 2024-02-19T18:52:09.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.751+0000 7f7583f90700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757c106420 0x7f757c19a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:09.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.751+0000 7f7583f90700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c19a5f0 con 0x7f757c106420 2024-02-19T18:52:09.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.752+0000 7f7581d2c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757c106420 0x7f757c19a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:09.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.752+0000 7f7581d2c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757c106420 0x7f757c19a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40222/0 (socket says 172.21.15.114:40222) 2024-02-19T18:52:09.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.752+0000 7f7581d2c700 1 -- 172.21.15.114:0/3231249929 learned_addr learned my addr 172.21.15.114:0/3231249929 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:09.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.752+0000 7f7581d2c700 1 -- 172.21.15.114:0/3231249929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7578009550 con 0x7f757c106420 2024-02-19T18:52:09.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.752+0000 7f7581d2c700 1 --2- 172.21.15.114:0/3231249929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757c106420 0x7f757c19a0b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7578000c00 tx=0x7f7578011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:09.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.752+0000 7f7572ffd700 1 -- 172.21.15.114:0/3231249929 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7578003b90 con 0x7f757c106420 2024-02-19T18:52:09.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.752+0000 7f7583f90700 1 -- 172.21.15.114:0/3231249929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f757c19a7f0 con 0x7f757c106420 2024-02-19T18:52:09.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.752+0000 7f7572ffd700 1 -- 172.21.15.114:0/3231249929 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f75780248b0 con 0x7f757c106420 2024-02-19T18:52:09.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.752+0000 7f7572ffd700 1 -- 172.21.15.114:0/3231249929 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f757801a8e0 con 0x7f757c106420 2024-02-19T18:52:09.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.752+0000 7f7583f90700 1 -- 172.21.15.114:0/3231249929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f757c19ac90 con 0x7f757c106420 2024-02-19T18:52:09.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.753+0000 7f7572ffd700 1 -- 172.21.15.114:0/3231249929 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f757801aba0 con 0x7f757c106420 2024-02-19T18:52:09.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.753+0000 7f7572ffd700 1 --2- 172.21.15.114:0/3231249929 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f7568038320 0x7f756803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:09.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.753+0000 7f7572ffd700 1 -- 172.21.15.114:0/3231249929 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f757804c440 con 0x7f757c106420 2024-02-19T18:52:09.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.753+0000 7f7583f90700 1 -- 172.21.15.114:0/3231249929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f757c0623c0 con 0x7f757c106420 2024-02-19T18:52:09.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.753+0000 7f758152b700 1 --2- 172.21.15.114:0/3231249929 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f7568038320 0x7f756803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:09.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.753+0000 7f758152b700 1 --2- 172.21.15.114:0/3231249929 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f7568038320 0x7f756803a7e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f756c006fd0 tx=0x7f756c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:09.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.758+0000 7f7572ffd700 1 -- 172.21.15.114:0/3231249929 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f75780233a0 con 0x7f757c106420 2024-02-19T18:52:09.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.949+0000 7f7583f90700 1 -- 172.21.15.114:0/3231249929 --> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f757c102f00 con 0x7f7568038320 2024-02-19T18:52:09.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.949+0000 7f7572ffd700 1 -- 172.21.15.114:0/3231249929 <== mgr.14118 v2:172.21.15.114:6800/971741305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f757c102f00 con 0x7f7568038320 2024-02-19T18:52:09.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.951+0000 7f7583f90700 1 -- 172.21.15.114:0/3231249929 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f7568038320 msgr2=0x7f756803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:09.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.951+0000 7f7583f90700 1 --2- 172.21.15.114:0/3231249929 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f7568038320 0x7f756803a7e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f756c006fd0 tx=0x7f756c006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:09.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.951+0000 7f7583f90700 1 -- 172.21.15.114:0/3231249929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757c106420 msgr2=0x7f757c19a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:09.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.951+0000 7f7583f90700 1 --2- 172.21.15.114:0/3231249929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757c106420 0x7f757c19a0b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7578000c00 tx=0x7f7578011ab0 comp rx=0 tx=0).stop 2024-02-19T18:52:09.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.951+0000 7f7583f90700 1 -- 172.21.15.114:0/3231249929 shutdown_connections 2024-02-19T18:52:09.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.951+0000 7f7583f90700 1 --2- 172.21.15.114:0/3231249929 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f7568038320 0x7f756803a7e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:09.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.951+0000 7f7583f90700 1 --2- 172.21.15.114:0/3231249929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f757c106420 0x7f757c19a0b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:09.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.951+0000 7f7583f90700 1 -- 172.21.15.114:0/3231249929 >> 172.21.15.114:0/3231249929 conn(0x7f757c101a00 msgr2=0x7f757c1027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:09.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.951+0000 7f7583f90700 1 -- 172.21.15.114:0/3231249929 shutdown_connections 2024-02-19T18:52:09.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:08.952+0000 7f7583f90700 1 -- 172.21.15.114:0/3231249929 wait complete. 2024-02-19T18:52:09.531 INFO:teuthology.orchestra.run.smithi114.stdout:Generating ssh key... 2024-02-19T18:52:10.252 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:09 smithi114 ceph-mon[29441]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:52:10.808 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.931+0000 7eff37b5a700 1 Processor -- start 2024-02-19T18:52:10.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.931+0000 7eff37b5a700 1 -- start start 2024-02-19T18:52:10.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.931+0000 7eff37b5a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff30106420 0x7eff30106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:10.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.931+0000 7eff37b5a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff30106e10 con 0x7eff30106420 2024-02-19T18:52:10.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.931+0000 7eff358f6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff30106420 0x7eff30106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:10.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.931+0000 7eff358f6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff30106420 0x7eff30106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40230/0 (socket says 172.21.15.114:40230) 2024-02-19T18:52:10.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.931+0000 7eff358f6700 1 -- 172.21.15.114:0/3041962691 learned_addr learned my addr 172.21.15.114:0/3041962691 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:10.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.931+0000 7eff358f6700 1 -- 172.21.15.114:0/3041962691 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff30107620 con 0x7eff30106420 2024-02-19T18:52:10.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.932+0000 7eff358f6700 1 --2- 172.21.15.114:0/3041962691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff30106420 0x7eff30106840 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7eff2c0098a0 tx=0x7eff2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90af546f8ae5d6af server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:10.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.932+0000 7eff348f4700 1 -- 172.21.15.114:0/3041962691 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2c004180 con 0x7eff30106420 2024-02-19T18:52:10.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.932+0000 7eff348f4700 1 -- 172.21.15.114:0/3041962691 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7eff2c0042e0 con 0x7eff30106420 2024-02-19T18:52:10.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.932+0000 7eff348f4700 1 -- 172.21.15.114:0/3041962691 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2c011a50 con 0x7eff30106420 2024-02-19T18:52:10.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.932+0000 7eff37b5a700 1 -- 172.21.15.114:0/3041962691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff30106420 msgr2=0x7eff30106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:10.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.932+0000 7eff37b5a700 1 --2- 172.21.15.114:0/3041962691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff30106420 0x7eff30106840 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7eff2c0098a0 tx=0x7eff2c009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:10.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.932+0000 7eff37b5a700 1 -- 172.21.15.114:0/3041962691 shutdown_connections 2024-02-19T18:52:10.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.932+0000 7eff37b5a700 1 --2- 172.21.15.114:0/3041962691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff30106420 0x7eff30106840 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:10.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.932+0000 7eff37b5a700 1 -- 172.21.15.114:0/3041962691 >> 172.21.15.114:0/3041962691 conn(0x7eff30101a00 msgr2=0x7eff30103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:10.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.933+0000 7eff37b5a700 1 -- 172.21.15.114:0/3041962691 shutdown_connections 2024-02-19T18:52:10.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.933+0000 7eff37b5a700 1 -- 172.21.15.114:0/3041962691 wait complete. 2024-02-19T18:52:10.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.933+0000 7eff37b5a700 1 Processor -- start 2024-02-19T18:52:10.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.933+0000 7eff37b5a700 1 -- start start 2024-02-19T18:52:10.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.933+0000 7eff37b5a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff30106420 0x7eff3019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:10.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.933+0000 7eff37b5a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3019a5f0 con 0x7eff30106420 2024-02-19T18:52:10.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.933+0000 7eff358f6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff30106420 0x7eff3019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:10.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.933+0000 7eff358f6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff30106420 0x7eff3019a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40244/0 (socket says 172.21.15.114:40244) 2024-02-19T18:52:10.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.933+0000 7eff358f6700 1 -- 172.21.15.114:0/3708950492 learned_addr learned my addr 172.21.15.114:0/3708950492 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:10.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.934+0000 7eff358f6700 1 -- 172.21.15.114:0/3708950492 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2c009550 con 0x7eff30106420 2024-02-19T18:52:10.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.934+0000 7eff358f6700 1 --2- 172.21.15.114:0/3708950492 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff30106420 0x7eff3019a0b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7eff2c006af0 tx=0x7eff2c011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:10.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.934+0000 7eff26ffd700 1 -- 172.21.15.114:0/3708950492 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2c003b90 con 0x7eff30106420 2024-02-19T18:52:10.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.934+0000 7eff26ffd700 1 -- 172.21.15.114:0/3708950492 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7eff2c01a8a0 con 0x7eff30106420 2024-02-19T18:52:10.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.934+0000 7eff37b5a700 1 -- 172.21.15.114:0/3708950492 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3019a7f0 con 0x7eff30106420 2024-02-19T18:52:10.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.934+0000 7eff26ffd700 1 -- 172.21.15.114:0/3708950492 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2c019880 con 0x7eff30106420 2024-02-19T18:52:10.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.934+0000 7eff37b5a700 1 -- 172.21.15.114:0/3708950492 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3019ac90 con 0x7eff30106420 2024-02-19T18:52:10.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.935+0000 7eff26ffd700 1 -- 172.21.15.114:0/3708950492 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7eff2c01a430 con 0x7eff30106420 2024-02-19T18:52:10.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.935+0000 7eff37b5a700 1 -- 172.21.15.114:0/3708950492 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff300623c0 con 0x7eff30106420 2024-02-19T18:52:10.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.935+0000 7eff26ffd700 1 --2- 172.21.15.114:0/3708950492 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7eff1c038330 0x7eff1c03a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:10.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.935+0000 7eff26ffd700 1 -- 172.21.15.114:0/3708950492 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7eff2c04c640 con 0x7eff30106420 2024-02-19T18:52:10.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.935+0000 7eff350f5700 1 --2- 172.21.15.114:0/3708950492 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7eff1c038330 0x7eff1c03a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:10.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.935+0000 7eff350f5700 1 --2- 172.21.15.114:0/3708950492 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7eff1c038330 0x7eff1c03a7f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7eff20006fd0 tx=0x7eff20006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:10.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:09.940+0000 7eff26ffd700 1 -- 172.21.15.114:0/3708950492 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7eff2c019af0 con 0x7eff30106420 2024-02-19T18:52:10.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:10.128+0000 7eff37b5a700 1 -- 172.21.15.114:0/3708950492 --> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7eff3010a820 con 0x7eff1c038330 2024-02-19T18:52:10.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:10.248+0000 7eff26ffd700 1 -- 172.21.15.114:0/3708950492 <== mgr.14118 v2:172.21.15.114:6800/971741305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7eff3010a820 con 0x7eff1c038330 2024-02-19T18:52:10.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:10.250+0000 7eff37b5a700 1 -- 172.21.15.114:0/3708950492 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7eff1c038330 msgr2=0x7eff1c03a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:10.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:10.250+0000 7eff37b5a700 1 --2- 172.21.15.114:0/3708950492 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7eff1c038330 0x7eff1c03a7f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7eff20006fd0 tx=0x7eff20006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:10.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:10.250+0000 7eff37b5a700 1 -- 172.21.15.114:0/3708950492 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff30106420 msgr2=0x7eff3019a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:10.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:10.250+0000 7eff37b5a700 1 --2- 172.21.15.114:0/3708950492 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff30106420 0x7eff3019a0b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7eff2c006af0 tx=0x7eff2c011a90 comp rx=0 tx=0).stop 2024-02-19T18:52:10.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:10.250+0000 7eff37b5a700 1 -- 172.21.15.114:0/3708950492 shutdown_connections 2024-02-19T18:52:10.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:10.250+0000 7eff37b5a700 1 --2- 172.21.15.114:0/3708950492 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7eff1c038330 0x7eff1c03a7f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:10.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:10.250+0000 7eff37b5a700 1 --2- 172.21.15.114:0/3708950492 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff30106420 0x7eff3019a0b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:10.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:10.250+0000 7eff37b5a700 1 -- 172.21.15.114:0/3708950492 >> 172.21.15.114:0/3708950492 conn(0x7eff30101a00 msgr2=0x7eff301027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:10.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:10.250+0000 7eff37b5a700 1 -- 172.21.15.114:0/3708950492 shutdown_connections 2024-02-19T18:52:10.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:10.250+0000 7eff37b5a700 1 -- 172.21.15.114:0/3708950492 wait complete. 2024-02-19T18:52:11.534 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:11 smithi114 ceph-mon[29441]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:52:11.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:11 smithi114 ceph-mon[29441]: Generating ssh key... 2024-02-19T18:52:11.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:11 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:11.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:11 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:12.031 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDeQI3jKEuC8R3rX7JE6I4AV3TXRjUyihASRChAo/bW1RWDnRWtdfxwxc72Frdm/ks8LyTIfxCDeCsXB/sAPx0mwVGNvTLvAzO/dpJDJ9XOJad7F4Z24xHk6DGBST+u5j9mS7LBY1TVzFX9BLKKBJ4Ewx/qEcZ/WZj2BXKTw1juu3fXuLreJPJ42TeZRMV3JGqZImBoTunDybwjMDcH8iCzSUfd7pVdG30Whz3XjuTD8/LB5d2klUQ37rW0bGwqqClnNsY8x2xDiyOJJrbFmLixtBO4XTB1gofAnwJ9JyzXKPrT9FN6ov0YsloKQR3rOa4QPhpDVigZUERMInhH/xNSwN8tTvfqD7WOmO3seZHF+CIuWqmk6/XDozblVaBAWSfBfd04yszEPCf2t5RL54mdRaQIsos+2+Twe+GpMTgey4rkCPf4QsMeGqiUHV9EQv9tJZFeY0h0h5ua/U+72GVRfyd2he2atyQoUCzpK0ryIPdBdNBSCgdZGAN8O8DBKOs= ceph-be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T18:52:12.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.211+0000 7fec45d83700 1 Processor -- start 2024-02-19T18:52:12.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.211+0000 7fec45d83700 1 -- start start 2024-02-19T18:52:12.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.211+0000 7fec45d83700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec40105330 0x7fec40105750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:12.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.211+0000 7fec45d83700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec40105d20 con 0x7fec40105330 2024-02-19T18:52:12.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.211+0000 7fec3f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec40105330 0x7fec40105750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:12.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.211+0000 7fec3f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec40105330 0x7fec40105750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40258/0 (socket says 172.21.15.114:40258) 2024-02-19T18:52:12.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.211+0000 7fec3f7fe700 1 -- 172.21.15.114:0/2596731641 learned_addr learned my addr 172.21.15.114:0/2596731641 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:12.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.212+0000 7fec3f7fe700 1 -- 172.21.15.114:0/2596731641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec40106530 con 0x7fec40105330 2024-02-19T18:52:12.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.212+0000 7fec3f7fe700 1 --2- 172.21.15.114:0/2596731641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec40105330 0x7fec40105750 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fec300098a0 tx=0x7fec30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f28e51f8f15e61 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:12.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.212+0000 7fec3effd700 1 -- 172.21.15.114:0/2596731641 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec30004180 con 0x7fec40105330 2024-02-19T18:52:12.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.212+0000 7fec3effd700 1 -- 172.21.15.114:0/2596731641 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fec300042e0 con 0x7fec40105330 2024-02-19T18:52:12.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.212+0000 7fec3effd700 1 -- 172.21.15.114:0/2596731641 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec30011a50 con 0x7fec40105330 2024-02-19T18:52:12.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.213+0000 7fec45d83700 1 -- 172.21.15.114:0/2596731641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec40105330 msgr2=0x7fec40105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:12.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.213+0000 7fec45d83700 1 --2- 172.21.15.114:0/2596731641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec40105330 0x7fec40105750 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fec300098a0 tx=0x7fec30009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:12.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.213+0000 7fec45d83700 1 -- 172.21.15.114:0/2596731641 shutdown_connections 2024-02-19T18:52:12.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.213+0000 7fec45d83700 1 --2- 172.21.15.114:0/2596731641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec40105330 0x7fec40105750 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:12.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.213+0000 7fec45d83700 1 -- 172.21.15.114:0/2596731641 >> 172.21.15.114:0/2596731641 conn(0x7fec401008b0 msgr2=0x7fec40102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:12.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.213+0000 7fec45d83700 1 -- 172.21.15.114:0/2596731641 shutdown_connections 2024-02-19T18:52:12.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.213+0000 7fec45d83700 1 -- 172.21.15.114:0/2596731641 wait complete. 2024-02-19T18:52:12.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.213+0000 7fec45d83700 1 Processor -- start 2024-02-19T18:52:12.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.213+0000 7fec45d83700 1 -- start start 2024-02-19T18:52:12.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.213+0000 7fec45d83700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec40105330 0x7fec40199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:12.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.213+0000 7fec45d83700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec40199570 con 0x7fec40105330 2024-02-19T18:52:12.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.214+0000 7fec3f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec40105330 0x7fec40199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:12.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.214+0000 7fec3f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec40105330 0x7fec40199030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40262/0 (socket says 172.21.15.114:40262) 2024-02-19T18:52:12.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.214+0000 7fec3f7fe700 1 -- 172.21.15.114:0/492677987 learned_addr learned my addr 172.21.15.114:0/492677987 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:12.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.214+0000 7fec3f7fe700 1 -- 172.21.15.114:0/492677987 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec30009550 con 0x7fec40105330 2024-02-19T18:52:12.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.214+0000 7fec3f7fe700 1 --2- 172.21.15.114:0/492677987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec40105330 0x7fec40199030 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fec30009520 tx=0x7fec30011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:12.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.214+0000 7fec3d7fa700 1 -- 172.21.15.114:0/492677987 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec30003a80 con 0x7fec40105330 2024-02-19T18:52:12.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.214+0000 7fec3d7fa700 1 -- 172.21.15.114:0/492677987 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fec30003be0 con 0x7fec40105330 2024-02-19T18:52:12.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.214+0000 7fec3d7fa700 1 -- 172.21.15.114:0/492677987 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec300197b0 con 0x7fec40105330 2024-02-19T18:52:12.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.214+0000 7fec45d83700 1 -- 172.21.15.114:0/492677987 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec40199770 con 0x7fec40105330 2024-02-19T18:52:12.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.214+0000 7fec45d83700 1 -- 172.21.15.114:0/492677987 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec40199c10 con 0x7fec40105330 2024-02-19T18:52:12.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.215+0000 7fec3d7fa700 1 -- 172.21.15.114:0/492677987 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fec3001a430 con 0x7fec40105330 2024-02-19T18:52:12.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.215+0000 7fec45d83700 1 -- 172.21.15.114:0/492677987 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec400623c0 con 0x7fec40105330 2024-02-19T18:52:12.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.215+0000 7fec3d7fa700 1 --2- 172.21.15.114:0/492677987 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fec2c0382d0 0x7fec2c03a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:12.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.215+0000 7fec3d7fa700 1 -- 172.21.15.114:0/492677987 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fec3004c570 con 0x7fec40105330 2024-02-19T18:52:12.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.215+0000 7fec36ffd700 1 --2- 172.21.15.114:0/492677987 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fec2c0382d0 0x7fec2c03a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:12.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.216+0000 7fec36ffd700 1 --2- 172.21.15.114:0/492677987 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fec2c0382d0 0x7fec2c03a790 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fec28006fd0 tx=0x7fec28006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:12.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.220+0000 7fec3d7fa700 1 -- 172.21.15.114:0/492677987 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fec3001ac30 con 0x7fec40105330 2024-02-19T18:52:12.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.409+0000 7fec45d83700 1 -- 172.21.15.114:0/492677987 --> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fec40102bc0 con 0x7fec2c0382d0 2024-02-19T18:52:12.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.410+0000 7fec3d7fa700 1 -- 172.21.15.114:0/492677987 <== mgr.14118 v2:172.21.15.114:6800/971741305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fec40102bc0 con 0x7fec2c0382d0 2024-02-19T18:52:12.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.412+0000 7fec45d83700 1 -- 172.21.15.114:0/492677987 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fec2c0382d0 msgr2=0x7fec2c03a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:12.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.412+0000 7fec45d83700 1 --2- 172.21.15.114:0/492677987 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fec2c0382d0 0x7fec2c03a790 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fec28006fd0 tx=0x7fec28006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:12.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.412+0000 7fec45d83700 1 -- 172.21.15.114:0/492677987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec40105330 msgr2=0x7fec40199030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:12.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.412+0000 7fec45d83700 1 --2- 172.21.15.114:0/492677987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec40105330 0x7fec40199030 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fec30009520 tx=0x7fec30011ab0 comp rx=0 tx=0).stop 2024-02-19T18:52:12.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.412+0000 7fec45d83700 1 -- 172.21.15.114:0/492677987 shutdown_connections 2024-02-19T18:52:12.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.412+0000 7fec45d83700 1 --2- 172.21.15.114:0/492677987 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fec2c0382d0 0x7fec2c03a790 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:12.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.412+0000 7fec45d83700 1 --2- 172.21.15.114:0/492677987 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec40105330 0x7fec40199030 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:12.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.412+0000 7fec45d83700 1 -- 172.21.15.114:0/492677987 >> 172.21.15.114:0/492677987 conn(0x7fec401008b0 msgr2=0x7fec40101fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:12.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.412+0000 7fec45d83700 1 -- 172.21.15.114:0/492677987 shutdown_connections 2024-02-19T18:52:12.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:11.412+0000 7fec45d83700 1 -- 172.21.15.114:0/492677987 wait complete. 2024-02-19T18:52:12.035 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-19T18:52:12.035 INFO:teuthology.orchestra.run.smithi114.stdout:Adding key to root@localhost authorized_keys... 2024-02-19T18:52:12.035 INFO:teuthology.orchestra.run.smithi114.stdout:Adding host smithi114... 2024-02-19T18:52:12.534 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:12 smithi114 ceph-mon[29441]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:52:14.284 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:13 smithi114 ceph-mon[29441]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi114", "addr": "172.21.15.114", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:52:15.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:14 smithi114 ceph-mon[29441]: Deploying cephadm binary to smithi114 2024-02-19T18:52:15.815 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Added host 'smithi114' with addr '172.21.15.114' 2024-02-19T18:52:15.815 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.437+0000 7f0e31632700 1 Processor -- start 2024-02-19T18:52:15.815 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.437+0000 7f0e31632700 1 -- start start 2024-02-19T18:52:15.815 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.437+0000 7f0e31632700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e2c105330 0x7f0e2c105750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:15.815 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.437+0000 7f0e31632700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c105d20 con 0x7f0e2c105330 2024-02-19T18:52:15.815 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.437+0000 7f0e2affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e2c105330 0x7f0e2c105750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:15.815 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.437+0000 7f0e2affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e2c105330 0x7f0e2c105750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40268/0 (socket says 172.21.15.114:40268) 2024-02-19T18:52:15.815 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.437+0000 7f0e2affd700 1 -- 172.21.15.114:0/2542388184 learned_addr learned my addr 172.21.15.114:0/2542388184 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:15.815 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.438+0000 7f0e2affd700 1 -- 172.21.15.114:0/2542388184 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c106530 con 0x7f0e2c105330 2024-02-19T18:52:15.815 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.438+0000 7f0e2affd700 1 --2- 172.21.15.114:0/2542388184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e2c105330 0x7f0e2c105750 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c0098a0 tx=0x7f0e1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ea1be9d5f53a3c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:15.816 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.438+0000 7f0e29ffb700 1 -- 172.21.15.114:0/2542388184 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e1c004180 con 0x7f0e2c105330 2024-02-19T18:52:15.816 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.438+0000 7f0e29ffb700 1 -- 172.21.15.114:0/2542388184 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0e1c0042e0 con 0x7f0e2c105330 2024-02-19T18:52:15.816 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.438+0000 7f0e29ffb700 1 -- 172.21.15.114:0/2542388184 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e1c011a50 con 0x7f0e2c105330 2024-02-19T18:52:15.816 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.438+0000 7f0e31632700 1 -- 172.21.15.114:0/2542388184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e2c105330 msgr2=0x7f0e2c105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:15.816 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.438+0000 7f0e31632700 1 --2- 172.21.15.114:0/2542388184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e2c105330 0x7f0e2c105750 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c0098a0 tx=0x7f0e1c009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:15.816 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.438+0000 7f0e31632700 1 -- 172.21.15.114:0/2542388184 shutdown_connections 2024-02-19T18:52:15.816 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.438+0000 7f0e31632700 1 --2- 172.21.15.114:0/2542388184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e2c105330 0x7f0e2c105750 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:15.816 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.438+0000 7f0e31632700 1 -- 172.21.15.114:0/2542388184 >> 172.21.15.114:0/2542388184 conn(0x7f0e2c1008b0 msgr2=0x7f0e2c102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:15.816 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.439+0000 7f0e31632700 1 -- 172.21.15.114:0/2542388184 shutdown_connections 2024-02-19T18:52:15.816 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.439+0000 7f0e31632700 1 -- 172.21.15.114:0/2542388184 wait complete. 2024-02-19T18:52:15.816 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.439+0000 7f0e31632700 1 Processor -- start 2024-02-19T18:52:15.816 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.439+0000 7f0e31632700 1 -- start start 2024-02-19T18:52:15.817 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.439+0000 7f0e31632700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e2c105330 0x7f0e2c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:15.817 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.439+0000 7f0e31632700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c198460 con 0x7f0e2c105330 2024-02-19T18:52:15.817 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.440+0000 7f0e2affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e2c105330 0x7f0e2c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:15.817 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.440+0000 7f0e2affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e2c105330 0x7f0e2c197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40270/0 (socket says 172.21.15.114:40270) 2024-02-19T18:52:15.817 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.440+0000 7f0e2affd700 1 -- 172.21.15.114:0/1250019287 learned_addr learned my addr 172.21.15.114:0/1250019287 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:15.817 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.440+0000 7f0e2affd700 1 -- 172.21.15.114:0/1250019287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c009550 con 0x7f0e2c105330 2024-02-19T18:52:15.817 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.440+0000 7f0e2affd700 1 --2- 172.21.15.114:0/1250019287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e2c105330 0x7f0e2c197f20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c009520 tx=0x7f0e1c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:15.817 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.440+0000 7f0e13fff700 1 -- 172.21.15.114:0/1250019287 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e1c003a80 con 0x7f0e2c105330 2024-02-19T18:52:15.817 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.440+0000 7f0e31632700 1 -- 172.21.15.114:0/1250019287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2c198660 con 0x7f0e2c105330 2024-02-19T18:52:15.817 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.440+0000 7f0e13fff700 1 -- 172.21.15.114:0/1250019287 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f0e1c003be0 con 0x7f0e2c105330 2024-02-19T18:52:15.817 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.440+0000 7f0e13fff700 1 -- 172.21.15.114:0/1250019287 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e1c0197b0 con 0x7f0e2c105330 2024-02-19T18:52:15.817 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.440+0000 7f0e31632700 1 -- 172.21.15.114:0/1250019287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2c198b00 con 0x7f0e2c105330 2024-02-19T18:52:15.817 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.441+0000 7f0e31632700 1 -- 172.21.15.114:0/1250019287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2c191bc0 con 0x7f0e2c105330 2024-02-19T18:52:15.817 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.441+0000 7f0e13fff700 1 -- 172.21.15.114:0/1250019287 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f0e1c019a20 con 0x7f0e2c105330 2024-02-19T18:52:15.817 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.441+0000 7f0e13fff700 1 --2- 172.21.15.114:0/1250019287 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f0e1403a530 0x7f0e1403c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:15.818 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.441+0000 7f0e13fff700 1 -- 172.21.15.114:0/1250019287 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0e1c04c510 con 0x7f0e2c105330 2024-02-19T18:52:15.818 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.441+0000 7f0e2a7fc700 1 --2- 172.21.15.114:0/1250019287 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f0e1403a530 0x7f0e1403c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:15.818 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.442+0000 7f0e2a7fc700 1 --2- 172.21.15.114:0/1250019287 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f0e1403a530 0x7f0e1403c9f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0e20006fd0 tx=0x7f0e20006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:15.818 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.446+0000 7f0e13fff700 1 -- 172.21.15.114:0/1250019287 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0e1c018740 con 0x7f0e2c105330 2024-02-19T18:52:15.818 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:12.639+0000 7f0e31632700 1 -- 172.21.15.114:0/1250019287 --> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi114", "addr": "172.21.15.114", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e2c0611d0 con 0x7f0e1403a530 2024-02-19T18:52:15.818 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:15.258+0000 7f0e13fff700 1 -- 172.21.15.114:0/1250019287 <== mgr.14118 v2:172.21.15.114:6800/971741305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f0e2c0611d0 con 0x7f0e1403a530 2024-02-19T18:52:15.818 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:15.261+0000 7f0e31632700 1 -- 172.21.15.114:0/1250019287 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f0e1403a530 msgr2=0x7f0e1403c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:15.818 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:15.261+0000 7f0e31632700 1 --2- 172.21.15.114:0/1250019287 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f0e1403a530 0x7f0e1403c9f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0e20006fd0 tx=0x7f0e20006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:15.818 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:15.261+0000 7f0e31632700 1 -- 172.21.15.114:0/1250019287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e2c105330 msgr2=0x7f0e2c197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:15.818 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:15.261+0000 7f0e31632700 1 --2- 172.21.15.114:0/1250019287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e2c105330 0x7f0e2c197f20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c009520 tx=0x7f0e1c011ab0 comp rx=0 tx=0).stop 2024-02-19T18:52:15.818 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:15.261+0000 7f0e31632700 1 -- 172.21.15.114:0/1250019287 shutdown_connections 2024-02-19T18:52:15.818 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:15.261+0000 7f0e31632700 1 --2- 172.21.15.114:0/1250019287 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f0e1403a530 0x7f0e1403c9f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:15.818 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:15.261+0000 7f0e31632700 1 --2- 172.21.15.114:0/1250019287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e2c105330 0x7f0e2c197f20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:15.818 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:15.261+0000 7f0e31632700 1 -- 172.21.15.114:0/1250019287 >> 172.21.15.114:0/1250019287 conn(0x7f0e2c1008b0 msgr2=0x7f0e2c18ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:15.818 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:15.261+0000 7f0e31632700 1 -- 172.21.15.114:0/1250019287 shutdown_connections 2024-02-19T18:52:15.818 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:15.261+0000 7f0e31632700 1 -- 172.21.15.114:0/1250019287 wait complete. 2024-02-19T18:52:15.819 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying mon service with default placement... 2024-02-19T18:52:16.437 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:16 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:16.437 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:16 smithi114 ceph-mon[29441]: Added host smithi114 2024-02-19T18:52:16.438 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:16 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:52:17.108 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-19T18:52:17.108 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.204+0000 7f60dd4b3700 1 Processor -- start 2024-02-19T18:52:17.108 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.204+0000 7f60dd4b3700 1 -- start start 2024-02-19T18:52:17.108 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.204+0000 7f60dd4b3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60d8105330 0x7f60d8105750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:17.108 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.204+0000 7f60dd4b3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d8105d20 con 0x7f60d8105330 2024-02-19T18:52:17.108 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.204+0000 7f60d6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60d8105330 0x7f60d8105750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:17.108 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.204+0000 7f60d6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60d8105330 0x7f60d8105750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45042/0 (socket says 172.21.15.114:45042) 2024-02-19T18:52:17.108 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.204+0000 7f60d6ffd700 1 -- 172.21.15.114:0/223158235 learned_addr learned my addr 172.21.15.114:0/223158235 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:17.108 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.205+0000 7f60d6ffd700 1 -- 172.21.15.114:0/223158235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d8106530 con 0x7f60d8105330 2024-02-19T18:52:17.109 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.205+0000 7f60d6ffd700 1 --2- 172.21.15.114:0/223158235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60d8105330 0x7f60d8105750 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f60c80098a0 tx=0x7f60c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a34a51183b10eaf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:17.109 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.205+0000 7f60d5ffb700 1 -- 172.21.15.114:0/223158235 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60c8004180 con 0x7f60d8105330 2024-02-19T18:52:17.109 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.205+0000 7f60d5ffb700 1 -- 172.21.15.114:0/223158235 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f60c80042e0 con 0x7f60d8105330 2024-02-19T18:52:17.109 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.205+0000 7f60d5ffb700 1 -- 172.21.15.114:0/223158235 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60c8011a50 con 0x7f60d8105330 2024-02-19T18:52:17.109 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.205+0000 7f60dd4b3700 1 -- 172.21.15.114:0/223158235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60d8105330 msgr2=0x7f60d8105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:17.109 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.205+0000 7f60dd4b3700 1 --2- 172.21.15.114:0/223158235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60d8105330 0x7f60d8105750 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f60c80098a0 tx=0x7f60c8009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:17.109 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.205+0000 7f60dd4b3700 1 -- 172.21.15.114:0/223158235 shutdown_connections 2024-02-19T18:52:17.109 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.205+0000 7f60dd4b3700 1 --2- 172.21.15.114:0/223158235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60d8105330 0x7f60d8105750 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:17.109 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.206+0000 7f60dd4b3700 1 -- 172.21.15.114:0/223158235 >> 172.21.15.114:0/223158235 conn(0x7f60d81008b0 msgr2=0x7f60d8102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:17.109 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.206+0000 7f60dd4b3700 1 -- 172.21.15.114:0/223158235 shutdown_connections 2024-02-19T18:52:17.109 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.206+0000 7f60dd4b3700 1 -- 172.21.15.114:0/223158235 wait complete. 2024-02-19T18:52:17.109 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.206+0000 7f60dd4b3700 1 Processor -- start 2024-02-19T18:52:17.109 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.206+0000 7f60dd4b3700 1 -- start start 2024-02-19T18:52:17.109 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.206+0000 7f60dd4b3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60d8105330 0x7f60d819a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:17.109 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.206+0000 7f60dd4b3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d819a690 con 0x7f60d8105330 2024-02-19T18:52:17.109 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.207+0000 7f60d6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60d8105330 0x7f60d819a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:17.110 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.207+0000 7f60d6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60d8105330 0x7f60d819a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45044/0 (socket says 172.21.15.114:45044) 2024-02-19T18:52:17.110 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.207+0000 7f60d6ffd700 1 -- 172.21.15.114:0/1790912062 learned_addr learned my addr 172.21.15.114:0/1790912062 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:17.110 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.207+0000 7f60d6ffd700 1 -- 172.21.15.114:0/1790912062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c8009550 con 0x7f60d8105330 2024-02-19T18:52:17.110 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.207+0000 7f60d6ffd700 1 --2- 172.21.15.114:0/1790912062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60d8105330 0x7f60d819a150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f60c8009520 tx=0x7f60c8011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:17.110 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.207+0000 7f60bffff700 1 -- 172.21.15.114:0/1790912062 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60c8003a80 con 0x7f60d8105330 2024-02-19T18:52:17.110 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.207+0000 7f60dd4b3700 1 -- 172.21.15.114:0/1790912062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d819a890 con 0x7f60d8105330 2024-02-19T18:52:17.110 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.207+0000 7f60bffff700 1 -- 172.21.15.114:0/1790912062 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f60c8003be0 con 0x7f60d8105330 2024-02-19T18:52:17.110 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.207+0000 7f60bffff700 1 -- 172.21.15.114:0/1790912062 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60c80197b0 con 0x7f60d8105330 2024-02-19T18:52:17.110 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.207+0000 7f60dd4b3700 1 -- 172.21.15.114:0/1790912062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d819ad30 con 0x7f60d8105330 2024-02-19T18:52:17.110 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.208+0000 7f60bffff700 1 -- 172.21.15.114:0/1790912062 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f60c8019a70 con 0x7f60d8105330 2024-02-19T18:52:17.110 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.208+0000 7f60bffff700 1 --2- 172.21.15.114:0/1790912062 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f60c00382d0 0x7f60c003a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:17.110 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.208+0000 7f60bffff700 1 -- 172.21.15.114:0/1790912062 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f60c801ac30 con 0x7f60d8105330 2024-02-19T18:52:17.110 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.208+0000 7f60d67fc700 1 --2- 172.21.15.114:0/1790912062 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f60c00382d0 0x7f60c003a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:17.110 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.208+0000 7f60dd4b3700 1 -- 172.21.15.114:0/1790912062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d80623c0 con 0x7f60d8105330 2024-02-19T18:52:17.110 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.209+0000 7f60d67fc700 1 --2- 172.21.15.114:0/1790912062 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f60c00382d0 0x7f60c003a790 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f60cc006fd0 tx=0x7f60cc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:17.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.213+0000 7f60bffff700 1 -- 172.21.15.114:0/1790912062 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f60c802c3a0 con 0x7f60d8105330 2024-02-19T18:52:17.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.425+0000 7f60dd4b3700 1 -- 172.21.15.114:0/1790912062 --> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f60d8102ac0 con 0x7f60c00382d0 2024-02-19T18:52:17.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.429+0000 7f60bffff700 1 -- 172.21.15.114:0/1790912062 <== mgr.14118 v2:172.21.15.114:6800/971741305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f60d8102ac0 con 0x7f60c00382d0 2024-02-19T18:52:17.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.431+0000 7f60dd4b3700 1 -- 172.21.15.114:0/1790912062 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f60c00382d0 msgr2=0x7f60c003a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:17.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.431+0000 7f60dd4b3700 1 --2- 172.21.15.114:0/1790912062 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f60c00382d0 0x7f60c003a790 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f60cc006fd0 tx=0x7f60cc006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:17.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.431+0000 7f60dd4b3700 1 -- 172.21.15.114:0/1790912062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60d8105330 msgr2=0x7f60d819a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:17.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.431+0000 7f60dd4b3700 1 --2- 172.21.15.114:0/1790912062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60d8105330 0x7f60d819a150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f60c8009520 tx=0x7f60c8011ab0 comp rx=0 tx=0).stop 2024-02-19T18:52:17.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.431+0000 7f60dd4b3700 1 -- 172.21.15.114:0/1790912062 shutdown_connections 2024-02-19T18:52:17.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.431+0000 7f60dd4b3700 1 --2- 172.21.15.114:0/1790912062 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f60c00382d0 0x7f60c003a790 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:17.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.431+0000 7f60dd4b3700 1 --2- 172.21.15.114:0/1790912062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60d8105330 0x7f60d819a150 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:17.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.431+0000 7f60dd4b3700 1 -- 172.21.15.114:0/1790912062 >> 172.21.15.114:0/1790912062 conn(0x7f60d81008b0 msgr2=0x7f60d8079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:17.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.432+0000 7f60dd4b3700 1 -- 172.21.15.114:0/1790912062 shutdown_connections 2024-02-19T18:52:17.111 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:16.432+0000 7f60dd4b3700 1 -- 172.21.15.114:0/1790912062 wait complete. 2024-02-19T18:52:17.111 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying mgr service with default placement... 2024-02-19T18:52:17.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:17 smithi114 ceph-mon[29441]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:52:17.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:17 smithi114 ceph-mon[29441]: Saving service mon spec with placement count:5 2024-02-19T18:52:17.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:17 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:19.170 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:18 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:19.170 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:18 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:19.170 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:18 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:19.177 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-19T18:52:19.177 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.329+0000 7fd77971f700 1 Processor -- start 2024-02-19T18:52:19.177 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.329+0000 7fd77971f700 1 -- start start 2024-02-19T18:52:19.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.329+0000 7fd77971f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd774106420 0x7fd774106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:19.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.329+0000 7fd77971f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd774106e10 con 0x7fd774106420 2024-02-19T18:52:19.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.329+0000 7fd772ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd774106420 0x7fd774106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:19.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.329+0000 7fd772ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd774106420 0x7fd774106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45048/0 (socket says 172.21.15.114:45048) 2024-02-19T18:52:19.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.329+0000 7fd772ffd700 1 -- 172.21.15.114:0/3024050448 learned_addr learned my addr 172.21.15.114:0/3024050448 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:19.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.330+0000 7fd772ffd700 1 -- 172.21.15.114:0/3024050448 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd774107620 con 0x7fd774106420 2024-02-19T18:52:19.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.330+0000 7fd772ffd700 1 --2- 172.21.15.114:0/3024050448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd774106420 0x7fd774106840 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd7640098a0 tx=0x7fd764009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ef7e5672e2221f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:19.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.330+0000 7fd7727fc700 1 -- 172.21.15.114:0/3024050448 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd764004180 con 0x7fd774106420 2024-02-19T18:52:19.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.330+0000 7fd7727fc700 1 -- 172.21.15.114:0/3024050448 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd7640042e0 con 0x7fd774106420 2024-02-19T18:52:19.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.330+0000 7fd7727fc700 1 -- 172.21.15.114:0/3024050448 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd764011af0 con 0x7fd774106420 2024-02-19T18:52:19.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.330+0000 7fd77971f700 1 -- 172.21.15.114:0/3024050448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd774106420 msgr2=0x7fd774106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:19.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.330+0000 7fd77971f700 1 --2- 172.21.15.114:0/3024050448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd774106420 0x7fd774106840 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd7640098a0 tx=0x7fd764009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:19.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.330+0000 7fd77971f700 1 -- 172.21.15.114:0/3024050448 shutdown_connections 2024-02-19T18:52:19.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.330+0000 7fd77971f700 1 --2- 172.21.15.114:0/3024050448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd774106420 0x7fd774106840 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:19.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.330+0000 7fd77971f700 1 -- 172.21.15.114:0/3024050448 >> 172.21.15.114:0/3024050448 conn(0x7fd774101a00 msgr2=0x7fd774103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:19.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.330+0000 7fd77971f700 1 -- 172.21.15.114:0/3024050448 shutdown_connections 2024-02-19T18:52:19.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.330+0000 7fd77971f700 1 -- 172.21.15.114:0/3024050448 wait complete. 2024-02-19T18:52:19.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.331+0000 7fd77971f700 1 Processor -- start 2024-02-19T18:52:19.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.331+0000 7fd77971f700 1 -- start start 2024-02-19T18:52:19.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.331+0000 7fd77971f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd774106420 0x7fd77419a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:19.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.331+0000 7fd77971f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77419a680 con 0x7fd774106420 2024-02-19T18:52:19.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.331+0000 7fd772ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd774106420 0x7fd77419a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:19.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.331+0000 7fd772ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd774106420 0x7fd77419a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45064/0 (socket says 172.21.15.114:45064) 2024-02-19T18:52:19.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.331+0000 7fd772ffd700 1 -- 172.21.15.114:0/3829856764 learned_addr learned my addr 172.21.15.114:0/3829856764 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:19.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.332+0000 7fd772ffd700 1 -- 172.21.15.114:0/3829856764 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd764009550 con 0x7fd774106420 2024-02-19T18:52:19.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.332+0000 7fd772ffd700 1 --2- 172.21.15.114:0/3829856764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd774106420 0x7fd77419a140 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd764006ba0 tx=0x7fd764011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:19.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.332+0000 7fd770ff9700 1 -- 172.21.15.114:0/3829856764 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd764003b00 con 0x7fd774106420 2024-02-19T18:52:19.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.332+0000 7fd77971f700 1 -- 172.21.15.114:0/3829856764 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd77419a880 con 0x7fd774106420 2024-02-19T18:52:19.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.332+0000 7fd770ff9700 1 -- 172.21.15.114:0/3829856764 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd76401a8e0 con 0x7fd774106420 2024-02-19T18:52:19.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.332+0000 7fd770ff9700 1 -- 172.21.15.114:0/3829856764 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd764019990 con 0x7fd774106420 2024-02-19T18:52:19.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.332+0000 7fd77971f700 1 -- 172.21.15.114:0/3829856764 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd77419ad20 con 0x7fd774106420 2024-02-19T18:52:19.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.333+0000 7fd770ff9700 1 -- 172.21.15.114:0/3829856764 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fd764019c50 con 0x7fd774106420 2024-02-19T18:52:19.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.333+0000 7fd77971f700 1 -- 172.21.15.114:0/3829856764 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7740623c0 con 0x7fd774106420 2024-02-19T18:52:19.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.333+0000 7fd770ff9700 1 --2- 172.21.15.114:0/3829856764 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fd7600382d0 0x7fd76003a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:19.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.333+0000 7fd770ff9700 1 -- 172.21.15.114:0/3829856764 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd76404c740 con 0x7fd774106420 2024-02-19T18:52:19.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.333+0000 7fd76a7fc700 1 --2- 172.21.15.114:0/3829856764 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fd7600382d0 0x7fd76003a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:19.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.333+0000 7fd76a7fc700 1 --2- 172.21.15.114:0/3829856764 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fd7600382d0 0x7fd76003a790 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd75c006fd0 tx=0x7fd75c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:19.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.338+0000 7fd770ff9700 1 -- 172.21.15.114:0/3829856764 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd76402d440 con 0x7fd774106420 2024-02-19T18:52:19.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.557+0000 7fd77971f700 1 -- 172.21.15.114:0/3829856764 --> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fd77410a820 con 0x7fd7600382d0 2024-02-19T18:52:19.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.564+0000 7fd770ff9700 1 -- 172.21.15.114:0/3829856764 <== mgr.14118 v2:172.21.15.114:6800/971741305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd77410a820 con 0x7fd7600382d0 2024-02-19T18:52:19.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.566+0000 7fd77971f700 1 -- 172.21.15.114:0/3829856764 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fd7600382d0 msgr2=0x7fd76003a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:19.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.566+0000 7fd77971f700 1 --2- 172.21.15.114:0/3829856764 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fd7600382d0 0x7fd76003a790 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd75c006fd0 tx=0x7fd75c006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:19.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.566+0000 7fd77971f700 1 -- 172.21.15.114:0/3829856764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd774106420 msgr2=0x7fd77419a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:19.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.566+0000 7fd77971f700 1 --2- 172.21.15.114:0/3829856764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd774106420 0x7fd77419a140 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd764006ba0 tx=0x7fd764011b50 comp rx=0 tx=0).stop 2024-02-19T18:52:19.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.566+0000 7fd77971f700 1 -- 172.21.15.114:0/3829856764 shutdown_connections 2024-02-19T18:52:19.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.566+0000 7fd77971f700 1 --2- 172.21.15.114:0/3829856764 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fd7600382d0 0x7fd76003a790 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:19.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.566+0000 7fd77971f700 1 --2- 172.21.15.114:0/3829856764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd774106420 0x7fd77419a140 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:19.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.566+0000 7fd77971f700 1 -- 172.21.15.114:0/3829856764 >> 172.21.15.114:0/3829856764 conn(0x7fd774101a00 msgr2=0x7fd7741030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:19.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.566+0000 7fd77971f700 1 -- 172.21.15.114:0/3829856764 shutdown_connections 2024-02-19T18:52:19.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:18.566+0000 7fd77971f700 1 -- 172.21.15.114:0/3829856764 wait complete. 2024-02-19T18:52:19.181 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying crash service with default placement... 2024-02-19T18:52:20.140 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:19 smithi114 ceph-mon[29441]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:52:20.140 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:19 smithi114 ceph-mon[29441]: Saving service mgr spec with placement count:2 2024-02-19T18:52:20.141 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:19 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:20.721 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-02-19T18:52:20.721 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.555+0000 7f83c1c46700 1 Processor -- start 2024-02-19T18:52:20.721 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.555+0000 7f83c1c46700 1 -- start start 2024-02-19T18:52:20.721 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.555+0000 7f83c1c46700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83bc106f00 0x7f83bc1092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:20.721 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.555+0000 7f83c1c46700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc079e80 con 0x7f83bc106f00 2024-02-19T18:52:20.721 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.556+0000 7f83bb7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83bc106f00 0x7f83bc1092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:20.721 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.556+0000 7f83bb7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83bc106f00 0x7f83bc1092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45076/0 (socket says 172.21.15.114:45076) 2024-02-19T18:52:20.721 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.556+0000 7f83bb7fe700 1 -- 172.21.15.114:0/349656174 learned_addr learned my addr 172.21.15.114:0/349656174 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:20.722 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.556+0000 7f83bb7fe700 1 -- 172.21.15.114:0/349656174 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83bc109830 con 0x7f83bc106f00 2024-02-19T18:52:20.722 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.556+0000 7f83bb7fe700 1 --2- 172.21.15.114:0/349656174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83bc106f00 0x7f83bc1092f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f83ac0098a0 tx=0x7f83ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ab82535d475e7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:20.722 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.556+0000 7f83ba7fc700 1 -- 172.21.15.114:0/349656174 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83ac004180 con 0x7f83bc106f00 2024-02-19T18:52:20.722 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.556+0000 7f83ba7fc700 1 -- 172.21.15.114:0/349656174 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f83ac0042e0 con 0x7f83bc106f00 2024-02-19T18:52:20.722 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.556+0000 7f83ba7fc700 1 -- 172.21.15.114:0/349656174 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83ac011af0 con 0x7f83bc106f00 2024-02-19T18:52:20.722 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.557+0000 7f83c1c46700 1 -- 172.21.15.114:0/349656174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83bc106f00 msgr2=0x7f83bc1092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:20.722 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.557+0000 7f83c1c46700 1 --2- 172.21.15.114:0/349656174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83bc106f00 0x7f83bc1092f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f83ac0098a0 tx=0x7f83ac009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:20.722 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.557+0000 7f83c1c46700 1 -- 172.21.15.114:0/349656174 shutdown_connections 2024-02-19T18:52:20.722 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.557+0000 7f83c1c46700 1 --2- 172.21.15.114:0/349656174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83bc106f00 0x7f83bc1092f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:20.722 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.557+0000 7f83c1c46700 1 -- 172.21.15.114:0/349656174 >> 172.21.15.114:0/349656174 conn(0x7f83bc1008d0 msgr2=0x7f83bc102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:20.722 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.557+0000 7f83c1c46700 1 -- 172.21.15.114:0/349656174 shutdown_connections 2024-02-19T18:52:20.722 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.557+0000 7f83c1c46700 1 -- 172.21.15.114:0/349656174 wait complete. 2024-02-19T18:52:20.722 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.558+0000 7f83c1c46700 1 Processor -- start 2024-02-19T18:52:20.723 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.558+0000 7f83c1c46700 1 -- start start 2024-02-19T18:52:20.723 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.558+0000 7f83c1c46700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83bc106f00 0x7f83bc19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:20.723 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.558+0000 7f83c1c46700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc19a650 con 0x7f83bc106f00 2024-02-19T18:52:20.723 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.558+0000 7f83bb7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83bc106f00 0x7f83bc19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:20.723 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.558+0000 7f83bb7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83bc106f00 0x7f83bc19a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45088/0 (socket says 172.21.15.114:45088) 2024-02-19T18:52:20.723 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.558+0000 7f83bb7fe700 1 -- 172.21.15.114:0/2863286148 learned_addr learned my addr 172.21.15.114:0/2863286148 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:20.723 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.559+0000 7f83bb7fe700 1 -- 172.21.15.114:0/2863286148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac009550 con 0x7f83bc106f00 2024-02-19T18:52:20.723 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.559+0000 7f83bb7fe700 1 --2- 172.21.15.114:0/2863286148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83bc106f00 0x7f83bc19a110 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f83ac011670 tx=0x7f83ac011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:20.723 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.559+0000 7f83b8ff9700 1 -- 172.21.15.114:0/2863286148 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83ac003a40 con 0x7f83bc106f00 2024-02-19T18:52:20.723 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.559+0000 7f83b8ff9700 1 -- 172.21.15.114:0/2863286148 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f83ac003ba0 con 0x7f83bc106f00 2024-02-19T18:52:20.723 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.559+0000 7f83b8ff9700 1 -- 172.21.15.114:0/2863286148 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83ac024820 con 0x7f83bc106f00 2024-02-19T18:52:20.723 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.559+0000 7f83c1c46700 1 -- 172.21.15.114:0/2863286148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83bc19a850 con 0x7f83bc106f00 2024-02-19T18:52:20.724 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.559+0000 7f83c1c46700 1 -- 172.21.15.114:0/2863286148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83bc19acf0 con 0x7f83bc106f00 2024-02-19T18:52:20.724 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.560+0000 7f83c1c46700 1 -- 172.21.15.114:0/2863286148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83bc10a5c0 con 0x7f83bc106f00 2024-02-19T18:52:20.724 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.560+0000 7f83b8ff9700 1 -- 172.21.15.114:0/2863286148 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f83ac024ae0 con 0x7f83bc106f00 2024-02-19T18:52:20.724 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.560+0000 7f83b8ff9700 1 --2- 172.21.15.114:0/2863286148 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f83a40382e0 0x7f83a403a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:20.724 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.560+0000 7f83b8ff9700 1 -- 172.21.15.114:0/2863286148 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f83ac04c2e0 con 0x7f83bc106f00 2024-02-19T18:52:20.724 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.560+0000 7f83baffd700 1 --2- 172.21.15.114:0/2863286148 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f83a40382e0 0x7f83a403a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:20.724 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.561+0000 7f83baffd700 1 --2- 172.21.15.114:0/2863286148 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f83a40382e0 0x7f83a403a7a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f83b0006fd0 tx=0x7f83b0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:20.724 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.565+0000 7f83b8ff9700 1 -- 172.21.15.114:0/2863286148 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f83ac022310 con 0x7f83bc106f00 2024-02-19T18:52:20.724 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.777+0000 7f83c1c46700 1 -- 172.21.15.114:0/2863286148 --> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f83bc101da0 con 0x7f83a40382e0 2024-02-19T18:52:20.724 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.782+0000 7f83b8ff9700 1 -- 172.21.15.114:0/2863286148 <== mgr.14118 v2:172.21.15.114:6800/971741305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f83bc101da0 con 0x7f83a40382e0 2024-02-19T18:52:20.724 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.784+0000 7f83c1c46700 1 -- 172.21.15.114:0/2863286148 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f83a40382e0 msgr2=0x7f83a403a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:20.724 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.784+0000 7f83c1c46700 1 --2- 172.21.15.114:0/2863286148 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f83a40382e0 0x7f83a403a7a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f83b0006fd0 tx=0x7f83b0006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:20.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.784+0000 7f83c1c46700 1 -- 172.21.15.114:0/2863286148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83bc106f00 msgr2=0x7f83bc19a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:20.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.784+0000 7f83c1c46700 1 --2- 172.21.15.114:0/2863286148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83bc106f00 0x7f83bc19a110 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f83ac011670 tx=0x7f83ac011aa0 comp rx=0 tx=0).stop 2024-02-19T18:52:20.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.784+0000 7f83c1c46700 1 -- 172.21.15.114:0/2863286148 shutdown_connections 2024-02-19T18:52:20.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.784+0000 7f83c1c46700 1 --2- 172.21.15.114:0/2863286148 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f83a40382e0 0x7f83a403a7a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:20.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.784+0000 7f83c1c46700 1 --2- 172.21.15.114:0/2863286148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83bc106f00 0x7f83bc19a110 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:20.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.784+0000 7f83c1c46700 1 -- 172.21.15.114:0/2863286148 >> 172.21.15.114:0/2863286148 conn(0x7f83bc1008d0 msgr2=0x7f83bc101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:20.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.784+0000 7f83c1c46700 1 -- 172.21.15.114:0/2863286148 shutdown_connections 2024-02-19T18:52:20.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:19.784+0000 7f83c1c46700 1 -- 172.21.15.114:0/2863286148 wait complete. 2024-02-19T18:52:20.725 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying ceph-exporter service with default placement... 2024-02-19T18:52:21.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:20 smithi114 ceph-mon[29441]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:52:21.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:20 smithi114 ceph-mon[29441]: Saving service crash spec with placement * 2024-02-19T18:52:21.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:20 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:21.970 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-02-19T18:52:21.970 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.122+0000 7f2ff62f2700 1 Processor -- start 2024-02-19T18:52:21.970 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.122+0000 7f2ff62f2700 1 -- start start 2024-02-19T18:52:21.970 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.122+0000 7f2ff62f2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ff0106f20 0x7f2ff0109310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:21.970 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.122+0000 7f2ff62f2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff0079e60 con 0x7f2ff0106f20 2024-02-19T18:52:21.970 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.123+0000 7f2feffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ff0106f20 0x7f2ff0109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:21.970 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.123+0000 7f2feffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ff0106f20 0x7f2ff0109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45090/0 (socket says 172.21.15.114:45090) 2024-02-19T18:52:21.970 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.123+0000 7f2feffff700 1 -- 172.21.15.114:0/2049036448 learned_addr learned my addr 172.21.15.114:0/2049036448 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:21.970 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.123+0000 7f2feffff700 1 -- 172.21.15.114:0/2049036448 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff0109850 con 0x7f2ff0106f20 2024-02-19T18:52:21.970 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.123+0000 7f2feffff700 1 --2- 172.21.15.114:0/2049036448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ff0106f20 0x7f2ff0109310 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80098a0 tx=0x7f2fd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a654aa386796d5a3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:21.970 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.123+0000 7f2feeffd700 1 -- 172.21.15.114:0/2049036448 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fd8004180 con 0x7f2ff0106f20 2024-02-19T18:52:21.970 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.123+0000 7f2feeffd700 1 -- 172.21.15.114:0/2049036448 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2fd80042e0 con 0x7f2ff0106f20 2024-02-19T18:52:21.971 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.123+0000 7f2feeffd700 1 -- 172.21.15.114:0/2049036448 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fd8011af0 con 0x7f2ff0106f20 2024-02-19T18:52:21.971 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.124+0000 7f2ff62f2700 1 -- 172.21.15.114:0/2049036448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ff0106f20 msgr2=0x7f2ff0109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:21.971 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.124+0000 7f2ff62f2700 1 --2- 172.21.15.114:0/2049036448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ff0106f20 0x7f2ff0109310 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80098a0 tx=0x7f2fd8009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:21.971 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.124+0000 7f2ff62f2700 1 -- 172.21.15.114:0/2049036448 shutdown_connections 2024-02-19T18:52:21.971 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.124+0000 7f2ff62f2700 1 --2- 172.21.15.114:0/2049036448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ff0106f20 0x7f2ff0109310 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:21.971 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.124+0000 7f2ff62f2700 1 -- 172.21.15.114:0/2049036448 >> 172.21.15.114:0/2049036448 conn(0x7f2ff01008d0 msgr2=0x7f2ff0102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:21.971 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.124+0000 7f2ff62f2700 1 -- 172.21.15.114:0/2049036448 shutdown_connections 2024-02-19T18:52:21.971 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.124+0000 7f2ff62f2700 1 -- 172.21.15.114:0/2049036448 wait complete. 2024-02-19T18:52:21.971 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.125+0000 7f2ff62f2700 1 Processor -- start 2024-02-19T18:52:21.971 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.125+0000 7f2ff62f2700 1 -- start start 2024-02-19T18:52:21.971 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.125+0000 7f2ff62f2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ff0106f20 0x7f2ff019a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:21.971 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.125+0000 7f2ff62f2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff019a650 con 0x7f2ff0106f20 2024-02-19T18:52:21.971 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.125+0000 7f2feffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ff0106f20 0x7f2ff019a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:21.971 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.125+0000 7f2feffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ff0106f20 0x7f2ff019a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45092/0 (socket says 172.21.15.114:45092) 2024-02-19T18:52:21.972 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.125+0000 7f2feffff700 1 -- 172.21.15.114:0/458682066 learned_addr learned my addr 172.21.15.114:0/458682066 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:21.972 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.125+0000 7f2feffff700 1 -- 172.21.15.114:0/458682066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd8009550 con 0x7f2ff0106f20 2024-02-19T18:52:21.972 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.125+0000 7f2feffff700 1 --2- 172.21.15.114:0/458682066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ff0106f20 0x7f2ff019a110 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8011670 tx=0x7f2fd8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:21.972 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.126+0000 7f2fed7fa700 1 -- 172.21.15.114:0/458682066 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fd8003a40 con 0x7f2ff0106f20 2024-02-19T18:52:21.972 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.126+0000 7f2fed7fa700 1 -- 172.21.15.114:0/458682066 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2fd8003ba0 con 0x7f2ff0106f20 2024-02-19T18:52:21.972 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.126+0000 7f2ff62f2700 1 -- 172.21.15.114:0/458682066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff019a850 con 0x7f2ff0106f20 2024-02-19T18:52:21.972 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.126+0000 7f2fed7fa700 1 -- 172.21.15.114:0/458682066 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fd8024790 con 0x7f2ff0106f20 2024-02-19T18:52:21.972 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.126+0000 7f2ff62f2700 1 -- 172.21.15.114:0/458682066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff019acf0 con 0x7f2ff0106f20 2024-02-19T18:52:21.972 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.126+0000 7f2fed7fa700 1 -- 172.21.15.114:0/458682066 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f2fd802d3f0 con 0x7f2ff0106f20 2024-02-19T18:52:21.972 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.127+0000 7f2fed7fa700 1 --2- 172.21.15.114:0/458682066 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2fdc038320 0x7f2fdc03a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:21.972 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.127+0000 7f2fed7fa700 1 -- 172.21.15.114:0/458682066 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2fd804d4e0 con 0x7f2ff0106f20 2024-02-19T18:52:21.972 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.127+0000 7f2fef7fe700 1 --2- 172.21.15.114:0/458682066 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2fdc038320 0x7f2fdc03a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:21.972 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.127+0000 7f2ff62f2700 1 -- 172.21.15.114:0/458682066 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff00623c0 con 0x7f2ff0106f20 2024-02-19T18:52:21.973 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.127+0000 7f2fef7fe700 1 --2- 172.21.15.114:0/458682066 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2fdc038320 0x7f2fdc03a7e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0006fd0 tx=0x7f2fe0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:21.973 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.132+0000 7f2fed7fa700 1 -- 172.21.15.114:0/458682066 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2fd8010970 con 0x7f2ff0106f20 2024-02-19T18:52:21.973 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.344+0000 7f2ff62f2700 1 -- 172.21.15.114:0/458682066 --> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ff0101db0 con 0x7f2fdc038320 2024-02-19T18:52:21.973 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.351+0000 7f2fed7fa700 1 -- 172.21.15.114:0/458682066 <== mgr.14118 v2:172.21.15.114:6800/971741305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f2ff0101db0 con 0x7f2fdc038320 2024-02-19T18:52:21.973 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.353+0000 7f2ff62f2700 1 -- 172.21.15.114:0/458682066 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2fdc038320 msgr2=0x7f2fdc03a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:21.973 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.353+0000 7f2ff62f2700 1 --2- 172.21.15.114:0/458682066 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2fdc038320 0x7f2fdc03a7e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0006fd0 tx=0x7f2fe0006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:21.973 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.353+0000 7f2ff62f2700 1 -- 172.21.15.114:0/458682066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ff0106f20 msgr2=0x7f2ff019a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:21.973 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.353+0000 7f2ff62f2700 1 --2- 172.21.15.114:0/458682066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ff0106f20 0x7f2ff019a110 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8011670 tx=0x7f2fd8011b50 comp rx=0 tx=0).stop 2024-02-19T18:52:21.973 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.353+0000 7f2ff62f2700 1 -- 172.21.15.114:0/458682066 shutdown_connections 2024-02-19T18:52:21.973 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.353+0000 7f2ff62f2700 1 --2- 172.21.15.114:0/458682066 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2fdc038320 0x7f2fdc03a7e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:21.973 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.353+0000 7f2ff62f2700 1 --2- 172.21.15.114:0/458682066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ff0106f20 0x7f2ff019a110 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:21.973 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.353+0000 7f2ff62f2700 1 -- 172.21.15.114:0/458682066 >> 172.21.15.114:0/458682066 conn(0x7f2ff01008d0 msgr2=0x7f2ff01016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:21.973 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.353+0000 7f2ff62f2700 1 -- 172.21.15.114:0/458682066 shutdown_connections 2024-02-19T18:52:21.973 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:21.353+0000 7f2ff62f2700 1 -- 172.21.15.114:0/458682066 wait complete. 2024-02-19T18:52:21.974 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying prometheus service with default placement... 2024-02-19T18:52:22.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:22 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:22.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:22 smithi114 ceph-mon[29441]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:52:22.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:22 smithi114 ceph-mon[29441]: Saving service ceph-exporter spec with placement * 2024-02-19T18:52:22.466 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:22 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:23.784 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:23 smithi114 ceph-mon[29441]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:52:23.784 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:23 smithi114 ceph-mon[29441]: Saving service prometheus spec with placement count:1 2024-02-19T18:52:23.784 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:23 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:23.933 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-02-19T18:52:23.933 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.440+0000 7fc609c99700 1 Processor -- start 2024-02-19T18:52:23.933 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.440+0000 7fc609c99700 1 -- start start 2024-02-19T18:52:23.933 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.440+0000 7fc609c99700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc604106420 0x7fc604106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:23.933 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.440+0000 7fc609c99700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc604106e10 con 0x7fc604106420 2024-02-19T18:52:23.933 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.441+0000 7fc6037fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc604106420 0x7fc604106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:23.933 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.441+0000 7fc6037fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc604106420 0x7fc604106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45096/0 (socket says 172.21.15.114:45096) 2024-02-19T18:52:23.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.441+0000 7fc6037fe700 1 -- 172.21.15.114:0/1774732899 learned_addr learned my addr 172.21.15.114:0/1774732899 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:23.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.441+0000 7fc6037fe700 1 -- 172.21.15.114:0/1774732899 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc604107620 con 0x7fc604106420 2024-02-19T18:52:23.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.441+0000 7fc6037fe700 1 --2- 172.21.15.114:0/1774732899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc604106420 0x7fc604106840 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc5f40098a0 tx=0x7fc5f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f161bb91c93fb7f0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:23.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.441+0000 7fc602ffd700 1 -- 172.21.15.114:0/1774732899 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5f4004180 con 0x7fc604106420 2024-02-19T18:52:23.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.441+0000 7fc602ffd700 1 -- 172.21.15.114:0/1774732899 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc5f40042e0 con 0x7fc604106420 2024-02-19T18:52:23.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.441+0000 7fc602ffd700 1 -- 172.21.15.114:0/1774732899 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5f4011af0 con 0x7fc604106420 2024-02-19T18:52:23.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.442+0000 7fc609c99700 1 -- 172.21.15.114:0/1774732899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc604106420 msgr2=0x7fc604106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:23.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.442+0000 7fc609c99700 1 --2- 172.21.15.114:0/1774732899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc604106420 0x7fc604106840 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc5f40098a0 tx=0x7fc5f4009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:23.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.442+0000 7fc609c99700 1 -- 172.21.15.114:0/1774732899 shutdown_connections 2024-02-19T18:52:23.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.442+0000 7fc609c99700 1 --2- 172.21.15.114:0/1774732899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc604106420 0x7fc604106840 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:23.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.442+0000 7fc609c99700 1 -- 172.21.15.114:0/1774732899 >> 172.21.15.114:0/1774732899 conn(0x7fc604101a00 msgr2=0x7fc604103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:23.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.442+0000 7fc609c99700 1 -- 172.21.15.114:0/1774732899 shutdown_connections 2024-02-19T18:52:23.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.442+0000 7fc609c99700 1 -- 172.21.15.114:0/1774732899 wait complete. 2024-02-19T18:52:23.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.443+0000 7fc609c99700 1 Processor -- start 2024-02-19T18:52:23.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.443+0000 7fc609c99700 1 -- start start 2024-02-19T18:52:23.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.443+0000 7fc609c99700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc604106420 0x7fc60407bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:23.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.443+0000 7fc609c99700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60407c1a0 con 0x7fc604106420 2024-02-19T18:52:23.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.443+0000 7fc6037fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc604106420 0x7fc60407bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:23.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.443+0000 7fc6037fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc604106420 0x7fc60407bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45110/0 (socket says 172.21.15.114:45110) 2024-02-19T18:52:23.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.443+0000 7fc6037fe700 1 -- 172.21.15.114:0/865365351 learned_addr learned my addr 172.21.15.114:0/865365351 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:23.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.443+0000 7fc6037fe700 1 -- 172.21.15.114:0/865365351 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f4009550 con 0x7fc604106420 2024-02-19T18:52:23.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.443+0000 7fc6037fe700 1 --2- 172.21.15.114:0/865365351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc604106420 0x7fc60407bc60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4011670 tx=0x7fc5f4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:23.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.444+0000 7fc6017fa700 1 -- 172.21.15.114:0/865365351 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5f40039c0 con 0x7fc604106420 2024-02-19T18:52:23.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.444+0000 7fc6017fa700 1 -- 172.21.15.114:0/865365351 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc5f4003b20 con 0x7fc604106420 2024-02-19T18:52:23.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.444+0000 7fc609c99700 1 -- 172.21.15.114:0/865365351 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc60407a2b0 con 0x7fc604106420 2024-02-19T18:52:23.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.444+0000 7fc6017fa700 1 -- 172.21.15.114:0/865365351 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5f40247f0 con 0x7fc604106420 2024-02-19T18:52:23.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.444+0000 7fc609c99700 1 -- 172.21.15.114:0/865365351 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc60407a750 con 0x7fc604106420 2024-02-19T18:52:23.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.444+0000 7fc6017fa700 1 -- 172.21.15.114:0/865365351 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fc5f4024a60 con 0x7fc604106420 2024-02-19T18:52:23.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.445+0000 7fc609c99700 1 -- 172.21.15.114:0/865365351 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6040623c0 con 0x7fc604106420 2024-02-19T18:52:23.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.445+0000 7fc6017fa700 1 --2- 172.21.15.114:0/865365351 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fc5f0038330 0x7fc5f003a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:23.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.445+0000 7fc6017fa700 1 -- 172.21.15.114:0/865365351 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc5f404c270 con 0x7fc604106420 2024-02-19T18:52:23.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.445+0000 7fc5faffd700 1 --2- 172.21.15.114:0/865365351 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fc5f0038330 0x7fc5f003a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:23.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.445+0000 7fc5faffd700 1 --2- 172.21.15.114:0/865365351 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fc5f0038330 0x7fc5f003a7f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec006fd0 tx=0x7fc5ec006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:23.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.449+0000 7fc6017fa700 1 -- 172.21.15.114:0/865365351 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc5f4024d10 con 0x7fc604106420 2024-02-19T18:52:23.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.659+0000 7fc609c99700 1 -- 172.21.15.114:0/865365351 --> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fc604103d10 con 0x7fc5f0038330 2024-02-19T18:52:23.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.665+0000 7fc6017fa700 1 -- 172.21.15.114:0/865365351 <== mgr.14118 v2:172.21.15.114:6800/971741305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fc604103d10 con 0x7fc5f0038330 2024-02-19T18:52:23.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.668+0000 7fc609c99700 1 -- 172.21.15.114:0/865365351 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fc5f0038330 msgr2=0x7fc5f003a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:23.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.668+0000 7fc609c99700 1 --2- 172.21.15.114:0/865365351 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fc5f0038330 0x7fc5f003a7f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec006fd0 tx=0x7fc5ec006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:23.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.668+0000 7fc609c99700 1 -- 172.21.15.114:0/865365351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc604106420 msgr2=0x7fc60407bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:23.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.668+0000 7fc609c99700 1 --2- 172.21.15.114:0/865365351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc604106420 0x7fc60407bc60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4011670 tx=0x7fc5f4011aa0 comp rx=0 tx=0).stop 2024-02-19T18:52:23.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.668+0000 7fc609c99700 1 -- 172.21.15.114:0/865365351 shutdown_connections 2024-02-19T18:52:23.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.668+0000 7fc609c99700 1 --2- 172.21.15.114:0/865365351 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fc5f0038330 0x7fc5f003a7f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:23.937 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.668+0000 7fc609c99700 1 --2- 172.21.15.114:0/865365351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc604106420 0x7fc60407bc60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:23.937 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.668+0000 7fc609c99700 1 -- 172.21.15.114:0/865365351 >> 172.21.15.114:0/865365351 conn(0x7fc604101a00 msgr2=0x7fc6041030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:23.937 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.668+0000 7fc609c99700 1 -- 172.21.15.114:0/865365351 shutdown_connections 2024-02-19T18:52:23.937 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:22.668+0000 7fc609c99700 1 -- 172.21.15.114:0/865365351 wait complete. 2024-02-19T18:52:23.937 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying grafana service with default placement... 2024-02-19T18:52:25.148 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-02-19T18:52:25.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.329+0000 7f2bc008d700 1 Processor -- start 2024-02-19T18:52:25.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.329+0000 7f2bc008d700 1 -- start start 2024-02-19T18:52:25.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.329+0000 7f2bc008d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bb8104cf0 0x7f2bb8105110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:25.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.329+0000 7f2bc008d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb8079e80 con 0x7f2bb8104cf0 2024-02-19T18:52:25.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.329+0000 7f2bbde29700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bb8104cf0 0x7f2bb8105110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:25.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.329+0000 7f2bbde29700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bb8104cf0 0x7f2bb8105110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45116/0 (socket says 172.21.15.114:45116) 2024-02-19T18:52:25.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.329+0000 7f2bbde29700 1 -- 172.21.15.114:0/802157064 learned_addr learned my addr 172.21.15.114:0/802157064 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:25.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.330+0000 7f2bbde29700 1 -- 172.21.15.114:0/802157064 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb8105650 con 0x7f2bb8104cf0 2024-02-19T18:52:25.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.330+0000 7f2bbde29700 1 --2- 172.21.15.114:0/802157064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bb8104cf0 0x7f2bb8105110 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2bb40098a0 tx=0x7f2bb4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a15f55adee693f5e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:25.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.330+0000 7f2bbce27700 1 -- 172.21.15.114:0/802157064 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb4004180 con 0x7f2bb8104cf0 2024-02-19T18:52:25.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.330+0000 7f2bbce27700 1 -- 172.21.15.114:0/802157064 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2bb40042e0 con 0x7f2bb8104cf0 2024-02-19T18:52:25.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.330+0000 7f2bbce27700 1 -- 172.21.15.114:0/802157064 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb4011af0 con 0x7f2bb8104cf0 2024-02-19T18:52:25.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.331+0000 7f2bc008d700 1 -- 172.21.15.114:0/802157064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bb8104cf0 msgr2=0x7f2bb8105110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:25.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.331+0000 7f2bc008d700 1 --2- 172.21.15.114:0/802157064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bb8104cf0 0x7f2bb8105110 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2bb40098a0 tx=0x7f2bb4009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:25.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.331+0000 7f2bc008d700 1 -- 172.21.15.114:0/802157064 shutdown_connections 2024-02-19T18:52:25.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.331+0000 7f2bc008d700 1 --2- 172.21.15.114:0/802157064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bb8104cf0 0x7f2bb8105110 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:25.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.331+0000 7f2bc008d700 1 -- 172.21.15.114:0/802157064 >> 172.21.15.114:0/802157064 conn(0x7f2bb81008d0 msgr2=0x7f2bb8102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:25.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.331+0000 7f2bc008d700 1 -- 172.21.15.114:0/802157064 shutdown_connections 2024-02-19T18:52:25.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.331+0000 7f2bc008d700 1 -- 172.21.15.114:0/802157064 wait complete. 2024-02-19T18:52:25.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.331+0000 7f2bc008d700 1 Processor -- start 2024-02-19T18:52:25.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.332+0000 7f2bc008d700 1 -- start start 2024-02-19T18:52:25.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.332+0000 7f2bc008d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bb8104cf0 0x7f2bb819c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:25.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.332+0000 7f2bc008d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb819c850 con 0x7f2bb8104cf0 2024-02-19T18:52:25.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.332+0000 7f2bbde29700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bb8104cf0 0x7f2bb819c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:25.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.332+0000 7f2bbde29700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bb8104cf0 0x7f2bb819c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45128/0 (socket says 172.21.15.114:45128) 2024-02-19T18:52:25.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.332+0000 7f2bbde29700 1 -- 172.21.15.114:0/464430667 learned_addr learned my addr 172.21.15.114:0/464430667 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:25.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.332+0000 7f2bbde29700 1 -- 172.21.15.114:0/464430667 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb4009550 con 0x7f2bb8104cf0 2024-02-19T18:52:25.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.332+0000 7f2bbde29700 1 --2- 172.21.15.114:0/464430667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bb8104cf0 0x7f2bb819c310 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4011670 tx=0x7f2bb4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:25.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.332+0000 7f2baeffd700 1 -- 172.21.15.114:0/464430667 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb4003a40 con 0x7f2bb8104cf0 2024-02-19T18:52:25.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.332+0000 7f2bc008d700 1 -- 172.21.15.114:0/464430667 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb819ca50 con 0x7f2bb8104cf0 2024-02-19T18:52:25.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.332+0000 7f2baeffd700 1 -- 172.21.15.114:0/464430667 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2bb4003ba0 con 0x7f2bb8104cf0 2024-02-19T18:52:25.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.332+0000 7f2baeffd700 1 -- 172.21.15.114:0/464430667 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bb4024820 con 0x7f2bb8104cf0 2024-02-19T18:52:25.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.332+0000 7f2bc008d700 1 -- 172.21.15.114:0/464430667 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb819cef0 con 0x7f2bb8104cf0 2024-02-19T18:52:25.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.333+0000 7f2baeffd700 1 -- 172.21.15.114:0/464430667 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f2bb401a430 con 0x7f2bb8104cf0 2024-02-19T18:52:25.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.333+0000 7f2bc008d700 1 -- 172.21.15.114:0/464430667 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb8195be0 con 0x7f2bb8104cf0 2024-02-19T18:52:25.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.333+0000 7f2baeffd700 1 --2- 172.21.15.114:0/464430667 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2ba40382e0 0x7f2ba403a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:25.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.334+0000 7f2baeffd700 1 -- 172.21.15.114:0/464430667 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2bb404c310 con 0x7f2bb8104cf0 2024-02-19T18:52:25.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.334+0000 7f2bbd628700 1 --2- 172.21.15.114:0/464430667 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2ba40382e0 0x7f2ba403a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:25.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.334+0000 7f2bbd628700 1 --2- 172.21.15.114:0/464430667 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2ba40382e0 0x7f2ba403a7a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8006fd0 tx=0x7f2ba8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:25.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.338+0000 7f2baeffd700 1 -- 172.21.15.114:0/464430667 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2bb4018530 con 0x7f2bb8104cf0 2024-02-19T18:52:25.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.545+0000 7f2bc008d700 1 -- 172.21.15.114:0/464430667 --> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f2bb80611d0 con 0x7f2ba40382e0 2024-02-19T18:52:25.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.549+0000 7f2baeffd700 1 -- 172.21.15.114:0/464430667 <== mgr.14118 v2:172.21.15.114:6800/971741305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f2bb80611d0 con 0x7f2ba40382e0 2024-02-19T18:52:25.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.551+0000 7f2bc008d700 1 -- 172.21.15.114:0/464430667 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2ba40382e0 msgr2=0x7f2ba403a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:25.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.551+0000 7f2bc008d700 1 --2- 172.21.15.114:0/464430667 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2ba40382e0 0x7f2ba403a7a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8006fd0 tx=0x7f2ba8006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:25.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.551+0000 7f2bc008d700 1 -- 172.21.15.114:0/464430667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bb8104cf0 msgr2=0x7f2bb819c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:25.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.551+0000 7f2bc008d700 1 --2- 172.21.15.114:0/464430667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bb8104cf0 0x7f2bb819c310 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4011670 tx=0x7f2bb4011aa0 comp rx=0 tx=0).stop 2024-02-19T18:52:25.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.552+0000 7f2bc008d700 1 -- 172.21.15.114:0/464430667 shutdown_connections 2024-02-19T18:52:25.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.552+0000 7f2bc008d700 1 --2- 172.21.15.114:0/464430667 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f2ba40382e0 0x7f2ba403a7a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:25.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.552+0000 7f2bc008d700 1 --2- 172.21.15.114:0/464430667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bb8104cf0 0x7f2bb819c310 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:25.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.552+0000 7f2bc008d700 1 -- 172.21.15.114:0/464430667 >> 172.21.15.114:0/464430667 conn(0x7f2bb81008d0 msgr2=0x7f2bb81908b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:25.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.552+0000 7f2bc008d700 1 -- 172.21.15.114:0/464430667 shutdown_connections 2024-02-19T18:52:25.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:24.552+0000 7f2bc008d700 1 -- 172.21.15.114:0/464430667 wait complete. 2024-02-19T18:52:25.153 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying node-exporter service with default placement... 2024-02-19T18:52:25.775 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:25 smithi114 ceph-mon[29441]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:52:25.775 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:25 smithi114 ceph-mon[29441]: Saving service grafana spec with placement count:1 2024-02-19T18:52:25.775 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:25 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:26.390 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-02-19T18:52:26.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.544+0000 7eff5f571700 1 Processor -- start 2024-02-19T18:52:26.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.544+0000 7eff5f571700 1 -- start start 2024-02-19T18:52:26.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.544+0000 7eff5f571700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5807beb0 0x7eff5807a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:26.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.544+0000 7eff5f571700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5807a8e0 con 0x7eff5807beb0 2024-02-19T18:52:26.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.545+0000 7eff5d30d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5807beb0 0x7eff5807a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:26.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.545+0000 7eff5d30d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5807beb0 0x7eff5807a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57250/0 (socket says 172.21.15.114:57250) 2024-02-19T18:52:26.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.545+0000 7eff5d30d700 1 -- 172.21.15.114:0/2945059443 learned_addr learned my addr 172.21.15.114:0/2945059443 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:26.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.545+0000 7eff5d30d700 1 -- 172.21.15.114:0/2945059443 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5807aa20 con 0x7eff5807beb0 2024-02-19T18:52:26.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.545+0000 7eff5d30d700 1 --2- 172.21.15.114:0/2945059443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5807beb0 0x7eff5807a310 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7eff50009e80 tx=0x7eff5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a74672336a1a7e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:26.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.545+0000 7eff4ffff700 1 -- 172.21.15.114:0/2945059443 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff5000e070 con 0x7eff5807beb0 2024-02-19T18:52:26.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.545+0000 7eff4ffff700 1 -- 172.21.15.114:0/2945059443 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7eff5000b8c0 con 0x7eff5807beb0 2024-02-19T18:52:26.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.545+0000 7eff4ffff700 1 -- 172.21.15.114:0/2945059443 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff500044d0 con 0x7eff5807beb0 2024-02-19T18:52:26.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.546+0000 7eff5f571700 1 -- 172.21.15.114:0/2945059443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5807beb0 msgr2=0x7eff5807a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:26.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.546+0000 7eff5f571700 1 --2- 172.21.15.114:0/2945059443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5807beb0 0x7eff5807a310 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7eff50009e80 tx=0x7eff5000b330 comp rx=0 tx=0).stop 2024-02-19T18:52:26.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.546+0000 7eff5f571700 1 -- 172.21.15.114:0/2945059443 shutdown_connections 2024-02-19T18:52:26.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.546+0000 7eff5f571700 1 --2- 172.21.15.114:0/2945059443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5807beb0 0x7eff5807a310 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:26.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.546+0000 7eff5f571700 1 -- 172.21.15.114:0/2945059443 >> 172.21.15.114:0/2945059443 conn(0x7eff58101a00 msgr2=0x7eff58103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:26.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.546+0000 7eff5f571700 1 -- 172.21.15.114:0/2945059443 shutdown_connections 2024-02-19T18:52:26.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.546+0000 7eff5f571700 1 -- 172.21.15.114:0/2945059443 wait complete. 2024-02-19T18:52:26.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.546+0000 7eff5f571700 1 Processor -- start 2024-02-19T18:52:26.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.546+0000 7eff5f571700 1 -- start start 2024-02-19T18:52:26.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.547+0000 7eff5f571700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5807beb0 0x7eff5819c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:26.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.547+0000 7eff5f571700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5819c810 con 0x7eff5807beb0 2024-02-19T18:52:26.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.547+0000 7eff5d30d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5807beb0 0x7eff5819c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:26.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.547+0000 7eff5d30d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5807beb0 0x7eff5819c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57256/0 (socket says 172.21.15.114:57256) 2024-02-19T18:52:26.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.547+0000 7eff5d30d700 1 -- 172.21.15.114:0/2435374174 learned_addr learned my addr 172.21.15.114:0/2435374174 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:26.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.547+0000 7eff5d30d700 1 -- 172.21.15.114:0/2435374174 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff50009b30 con 0x7eff5807beb0 2024-02-19T18:52:26.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.547+0000 7eff5d30d700 1 --2- 172.21.15.114:0/2435374174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5807beb0 0x7eff5819c2d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7eff500072a0 tx=0x7eff500049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:26.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.547+0000 7eff4e7fc700 1 -- 172.21.15.114:0/2435374174 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff5000e040 con 0x7eff5807beb0 2024-02-19T18:52:26.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.547+0000 7eff4e7fc700 1 -- 172.21.15.114:0/2435374174 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7eff5000bd80 con 0x7eff5807beb0 2024-02-19T18:52:26.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.547+0000 7eff5f571700 1 -- 172.21.15.114:0/2435374174 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff5819ca10 con 0x7eff5807beb0 2024-02-19T18:52:26.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.547+0000 7eff4e7fc700 1 -- 172.21.15.114:0/2435374174 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff500126c0 con 0x7eff5807beb0 2024-02-19T18:52:26.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.547+0000 7eff5f571700 1 -- 172.21.15.114:0/2435374174 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff5819cf10 con 0x7eff5807beb0 2024-02-19T18:52:26.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.548+0000 7eff4e7fc700 1 -- 172.21.15.114:0/2435374174 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7eff50019040 con 0x7eff5807beb0 2024-02-19T18:52:26.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.548+0000 7eff5f571700 1 -- 172.21.15.114:0/2435374174 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff580623c0 con 0x7eff5807beb0 2024-02-19T18:52:26.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.549+0000 7eff4e7fc700 1 --2- 172.21.15.114:0/2435374174 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7eff44038330 0x7eff4403a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:26.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.549+0000 7eff4e7fc700 1 -- 172.21.15.114:0/2435374174 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7eff5004be40 con 0x7eff5807beb0 2024-02-19T18:52:26.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.549+0000 7eff5cb0c700 1 --2- 172.21.15.114:0/2435374174 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7eff44038330 0x7eff4403a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:26.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.549+0000 7eff5cb0c700 1 --2- 172.21.15.114:0/2435374174 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7eff44038330 0x7eff4403a7f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7eff48006fd0 tx=0x7eff48006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:26.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.555+0000 7eff4e7fc700 1 -- 172.21.15.114:0/2435374174 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7eff500244a0 con 0x7eff5807beb0 2024-02-19T18:52:26.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.758+0000 7eff5f571700 1 -- 172.21.15.114:0/2435374174 --> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7eff58102eb0 con 0x7eff44038330 2024-02-19T18:52:26.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.764+0000 7eff4e7fc700 1 -- 172.21.15.114:0/2435374174 <== mgr.14118 v2:172.21.15.114:6800/971741305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7eff58102eb0 con 0x7eff44038330 2024-02-19T18:52:26.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.766+0000 7eff5f571700 1 -- 172.21.15.114:0/2435374174 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7eff44038330 msgr2=0x7eff4403a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:26.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.766+0000 7eff5f571700 1 --2- 172.21.15.114:0/2435374174 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7eff44038330 0x7eff4403a7f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7eff48006fd0 tx=0x7eff48006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:26.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.766+0000 7eff5f571700 1 -- 172.21.15.114:0/2435374174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5807beb0 msgr2=0x7eff5819c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:26.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.766+0000 7eff5f571700 1 --2- 172.21.15.114:0/2435374174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5807beb0 0x7eff5819c2d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7eff500072a0 tx=0x7eff500049f0 comp rx=0 tx=0).stop 2024-02-19T18:52:26.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.766+0000 7eff5f571700 1 -- 172.21.15.114:0/2435374174 shutdown_connections 2024-02-19T18:52:26.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.766+0000 7eff5f571700 1 --2- 172.21.15.114:0/2435374174 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7eff44038330 0x7eff4403a7f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:26.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.766+0000 7eff5f571700 1 --2- 172.21.15.114:0/2435374174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff5807beb0 0x7eff5819c2d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:26.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.766+0000 7eff5f571700 1 -- 172.21.15.114:0/2435374174 >> 172.21.15.114:0/2435374174 conn(0x7eff58101a00 msgr2=0x7eff581027a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:26.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.766+0000 7eff5f571700 1 -- 172.21.15.114:0/2435374174 shutdown_connections 2024-02-19T18:52:26.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:25.766+0000 7eff5f571700 1 -- 172.21.15.114:0/2435374174 wait complete. 2024-02-19T18:52:26.394 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying alertmanager service with default placement... 2024-02-19T18:52:27.011 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:26 smithi114 ceph-mon[29441]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:27.011 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:26 smithi114 ceph-mon[29441]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:52:27.011 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:26 smithi114 ceph-mon[29441]: Saving service node-exporter spec with placement * 2024-02-19T18:52:27.011 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:26 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:27.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-02-19T18:52:27.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.784+0000 7f0764f97700 1 Processor -- start 2024-02-19T18:52:27.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.784+0000 7f0764f97700 1 -- start start 2024-02-19T18:52:27.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.784+0000 7f0764f97700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0760106420 0x7f0760106840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:27.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.784+0000 7f0764f97700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0760106e10 con 0x7f0760106420 2024-02-19T18:52:27.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.784+0000 7f075e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0760106420 0x7f0760106840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:27.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.784+0000 7f075e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0760106420 0x7f0760106840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57270/0 (socket says 172.21.15.114:57270) 2024-02-19T18:52:27.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.784+0000 7f075e59c700 1 -- 172.21.15.114:0/10541661 learned_addr learned my addr 172.21.15.114:0/10541661 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:27.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.785+0000 7f075e59c700 1 -- 172.21.15.114:0/10541661 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0760107620 con 0x7f0760106420 2024-02-19T18:52:27.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.785+0000 7f075e59c700 1 --2- 172.21.15.114:0/10541661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0760106420 0x7f0760106840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f07500098a0 tx=0x7f0750009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3a6011a80903a56 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:27.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.785+0000 7f075dd9b700 1 -- 172.21.15.114:0/10541661 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0750004180 con 0x7f0760106420 2024-02-19T18:52:27.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.785+0000 7f075dd9b700 1 -- 172.21.15.114:0/10541661 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f07500042e0 con 0x7f0760106420 2024-02-19T18:52:27.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.785+0000 7f075dd9b700 1 -- 172.21.15.114:0/10541661 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0750011af0 con 0x7f0760106420 2024-02-19T18:52:27.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.785+0000 7f0764f97700 1 -- 172.21.15.114:0/10541661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0760106420 msgr2=0x7f0760106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:27.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.785+0000 7f0764f97700 1 --2- 172.21.15.114:0/10541661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0760106420 0x7f0760106840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f07500098a0 tx=0x7f0750009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:27.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.786+0000 7f0764f97700 1 -- 172.21.15.114:0/10541661 shutdown_connections 2024-02-19T18:52:27.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.786+0000 7f0764f97700 1 --2- 172.21.15.114:0/10541661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0760106420 0x7f0760106840 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:27.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.786+0000 7f0764f97700 1 -- 172.21.15.114:0/10541661 >> 172.21.15.114:0/10541661 conn(0x7f0760101a00 msgr2=0x7f0760103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:27.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.786+0000 7f0764f97700 1 -- 172.21.15.114:0/10541661 shutdown_connections 2024-02-19T18:52:27.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.786+0000 7f0764f97700 1 -- 172.21.15.114:0/10541661 wait complete. 2024-02-19T18:52:27.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.786+0000 7f0764f97700 1 Processor -- start 2024-02-19T18:52:27.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.786+0000 7f0764f97700 1 -- start start 2024-02-19T18:52:27.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.787+0000 7f0764f97700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0760106420 0x7f076019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:27.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.787+0000 7f0764f97700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076019a6c0 con 0x7f0760106420 2024-02-19T18:52:27.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.787+0000 7f075e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0760106420 0x7f076019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:27.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.787+0000 7f075e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0760106420 0x7f076019a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57282/0 (socket says 172.21.15.114:57282) 2024-02-19T18:52:27.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.787+0000 7f075e59c700 1 -- 172.21.15.114:0/2863683686 learned_addr learned my addr 172.21.15.114:0/2863683686 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:27.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.787+0000 7f075e59c700 1 -- 172.21.15.114:0/2863683686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0750009550 con 0x7f0760106420 2024-02-19T18:52:27.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.787+0000 7f075e59c700 1 --2- 172.21.15.114:0/2863683686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0760106420 0x7f076019a180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0750011670 tx=0x7f0750011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:27.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.787+0000 7f0757fff700 1 -- 172.21.15.114:0/2863683686 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07500039c0 con 0x7f0760106420 2024-02-19T18:52:27.530 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.787+0000 7f0757fff700 1 -- 172.21.15.114:0/2863683686 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0750003b20 con 0x7f0760106420 2024-02-19T18:52:27.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.787+0000 7f0764f97700 1 -- 172.21.15.114:0/2863683686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076019a8c0 con 0x7f0760106420 2024-02-19T18:52:27.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.787+0000 7f0757fff700 1 -- 172.21.15.114:0/2863683686 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07500247f0 con 0x7f0760106420 2024-02-19T18:52:27.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.787+0000 7f0764f97700 1 -- 172.21.15.114:0/2863683686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076019ad60 con 0x7f0760106420 2024-02-19T18:52:27.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.788+0000 7f0757fff700 1 -- 172.21.15.114:0/2863683686 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f0750024ab0 con 0x7f0760106420 2024-02-19T18:52:27.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.789+0000 7f0764f97700 1 -- 172.21.15.114:0/2863683686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07600623c0 con 0x7f0760106420 2024-02-19T18:52:27.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.789+0000 7f0757fff700 1 --2- 172.21.15.114:0/2863683686 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f074c0382e0 0x7f074c03a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:27.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.789+0000 7f0757fff700 1 -- 172.21.15.114:0/2863683686 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f075004c5a0 con 0x7f0760106420 2024-02-19T18:52:27.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.789+0000 7f0755d9b700 1 --2- 172.21.15.114:0/2863683686 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f074c0382e0 0x7f074c03a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:27.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.789+0000 7f0755d9b700 1 --2- 172.21.15.114:0/2863683686 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f074c0382e0 0x7f074c03a7a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0748006fd0 tx=0x7f0748006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:27.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.793+0000 7f0757fff700 1 -- 172.21.15.114:0/2863683686 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0750024d60 con 0x7f0760106420 2024-02-19T18:52:27.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:26.995+0000 7f0764f97700 1 -- 172.21.15.114:0/2863683686 --> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f0760103d10 con 0x7f074c0382e0 2024-02-19T18:52:27.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.000+0000 7f0757fff700 1 -- 172.21.15.114:0/2863683686 <== mgr.14118 v2:172.21.15.114:6800/971741305 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f0760103d10 con 0x7f074c0382e0 2024-02-19T18:52:27.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.002+0000 7f0764f97700 1 -- 172.21.15.114:0/2863683686 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f074c0382e0 msgr2=0x7f074c03a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:27.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.002+0000 7f0764f97700 1 --2- 172.21.15.114:0/2863683686 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f074c0382e0 0x7f074c03a7a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0748006fd0 tx=0x7f0748006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:27.531 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.002+0000 7f0764f97700 1 -- 172.21.15.114:0/2863683686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0760106420 msgr2=0x7f076019a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:27.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.002+0000 7f0764f97700 1 --2- 172.21.15.114:0/2863683686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0760106420 0x7f076019a180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0750011670 tx=0x7f0750011aa0 comp rx=0 tx=0).stop 2024-02-19T18:52:27.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.002+0000 7f0764f97700 1 -- 172.21.15.114:0/2863683686 shutdown_connections 2024-02-19T18:52:27.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.002+0000 7f0764f97700 1 --2- 172.21.15.114:0/2863683686 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f074c0382e0 0x7f074c03a7a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:27.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.002+0000 7f0764f97700 1 --2- 172.21.15.114:0/2863683686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0760106420 0x7f076019a180 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:27.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.002+0000 7f0764f97700 1 -- 172.21.15.114:0/2863683686 >> 172.21.15.114:0/2863683686 conn(0x7f0760101a00 msgr2=0x7f07601030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:27.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.002+0000 7f0764f97700 1 -- 172.21.15.114:0/2863683686 shutdown_connections 2024-02-19T18:52:27.532 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.002+0000 7f0764f97700 1 -- 172.21.15.114:0/2863683686 wait complete. 2024-02-19T18:52:28.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:28 smithi114 ceph-mon[29441]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:52:28.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:28 smithi114 ceph-mon[29441]: Saving service alertmanager spec with placement count:1 2024-02-19T18:52:28.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:28 smithi114 ceph-mon[29441]: from='mgr.14118 172.21.15.114:0/2974511756' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:28.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:28 smithi114 ceph-mon[29441]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:28.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.968+0000 7f41f319f700 1 Processor -- start 2024-02-19T18:52:28.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.968+0000 7f41f319f700 1 -- start start 2024-02-19T18:52:28.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.969+0000 7f41f319f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec104550 0x7f41ec104970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:28.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.969+0000 7f41f319f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec104f40 con 0x7f41ec104550 2024-02-19T18:52:28.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.969+0000 7f41f219d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec104550 0x7f41ec104970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:28.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.969+0000 7f41f219d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec104550 0x7f41ec104970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57298/0 (socket says 172.21.15.114:57298) 2024-02-19T18:52:28.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.969+0000 7f41f219d700 1 -- 172.21.15.114:0/854824978 learned_addr learned my addr 172.21.15.114:0/854824978 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:28.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.969+0000 7f41f219d700 1 -- 172.21.15.114:0/854824978 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ec1074e0 con 0x7f41ec104550 2024-02-19T18:52:28.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.969+0000 7f41f219d700 1 --2- 172.21.15.114:0/854824978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec104550 0x7f41ec104970 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f41e80098a0 tx=0x7f41e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c78ba68a24a0c9f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:28.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.969+0000 7f41f119b700 1 -- 172.21.15.114:0/854824978 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41e8004180 con 0x7f41ec104550 2024-02-19T18:52:28.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.969+0000 7f41f119b700 1 -- 172.21.15.114:0/854824978 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f41e80042e0 con 0x7f41ec104550 2024-02-19T18:52:28.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.969+0000 7f41f119b700 1 -- 172.21.15.114:0/854824978 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41e8011af0 con 0x7f41ec104550 2024-02-19T18:52:28.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.970+0000 7f41f319f700 1 -- 172.21.15.114:0/854824978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec104550 msgr2=0x7f41ec104970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:28.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.970+0000 7f41f319f700 1 --2- 172.21.15.114:0/854824978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec104550 0x7f41ec104970 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f41e80098a0 tx=0x7f41e8009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:28.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.970+0000 7f41f319f700 1 -- 172.21.15.114:0/854824978 shutdown_connections 2024-02-19T18:52:28.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.970+0000 7f41f319f700 1 --2- 172.21.15.114:0/854824978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec104550 0x7f41ec104970 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:28.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.970+0000 7f41f319f700 1 -- 172.21.15.114:0/854824978 >> 172.21.15.114:0/854824978 conn(0x7f41ec0ffe40 msgr2=0x7f41ec1022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:28.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.970+0000 7f41f319f700 1 -- 172.21.15.114:0/854824978 shutdown_connections 2024-02-19T18:52:28.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.970+0000 7f41f319f700 1 -- 172.21.15.114:0/854824978 wait complete. 2024-02-19T18:52:28.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.971+0000 7f41f319f700 1 Processor -- start 2024-02-19T18:52:28.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.971+0000 7f41f319f700 1 -- start start 2024-02-19T18:52:28.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.971+0000 7f41f319f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec104550 0x7f41ec106e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:28.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.971+0000 7f41f319f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec107380 con 0x7f41ec104550 2024-02-19T18:52:28.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.971+0000 7f41f219d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec104550 0x7f41ec106e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:28.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.971+0000 7f41f219d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec104550 0x7f41ec106e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57310/0 (socket says 172.21.15.114:57310) 2024-02-19T18:52:28.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.971+0000 7f41f219d700 1 -- 172.21.15.114:0/2456776233 learned_addr learned my addr 172.21.15.114:0/2456776233 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:28.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.971+0000 7f41f219d700 1 -- 172.21.15.114:0/2456776233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e8009550 con 0x7f41ec104550 2024-02-19T18:52:28.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.971+0000 7f41f219d700 1 --2- 172.21.15.114:0/2456776233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec104550 0x7f41ec106e40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f41e8011670 tx=0x7f41e8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:28.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.971+0000 7f41e37fe700 1 -- 172.21.15.114:0/2456776233 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41e80039f0 con 0x7f41ec104550 2024-02-19T18:52:28.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.971+0000 7f41e37fe700 1 -- 172.21.15.114:0/2456776233 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f41e8003b50 con 0x7f41ec104550 2024-02-19T18:52:28.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.972+0000 7f41f319f700 1 -- 172.21.15.114:0/2456776233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41ec105490 con 0x7f41ec104550 2024-02-19T18:52:28.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.972+0000 7f41e37fe700 1 -- 172.21.15.114:0/2456776233 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41e80247c0 con 0x7f41ec104550 2024-02-19T18:52:28.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.972+0000 7f41f319f700 1 -- 172.21.15.114:0/2456776233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ec105930 con 0x7f41ec104550 2024-02-19T18:52:28.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.972+0000 7f41e37fe700 1 -- 172.21.15.114:0/2456776233 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f41e802d3f0 con 0x7f41ec104550 2024-02-19T18:52:28.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.972+0000 7f41f319f700 1 -- 172.21.15.114:0/2456776233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41ec0623c0 con 0x7f41ec104550 2024-02-19T18:52:28.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.973+0000 7f41e37fe700 1 --2- 172.21.15.114:0/2456776233 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f41d8038320 0x7f41d803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:28.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.973+0000 7f41e37fe700 1 -- 172.21.15.114:0/2456776233 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f41e804c500 con 0x7f41ec104550 2024-02-19T18:52:28.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.973+0000 7f41f199c700 1 --2- 172.21.15.114:0/2456776233 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f41d8038320 0x7f41d803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:28.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.973+0000 7f41f199c700 1 --2- 172.21.15.114:0/2456776233 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f41d8038320 0x7f41d803a7e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f41dc006fd0 tx=0x7f41dc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:28.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:27.977+0000 7f41e37fe700 1 -- 172.21.15.114:0/2456776233 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f41e8010970 con 0x7f41ec104550 2024-02-19T18:52:28.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:28.172+0000 7f41f319f700 1 -- 172.21.15.114:0/2456776233 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f41ec10a6e0 con 0x7f41ec104550 2024-02-19T18:52:28.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:28.175+0000 7f41e37fe700 1 -- 172.21.15.114:0/2456776233 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f41e8010380 con 0x7f41ec104550 2024-02-19T18:52:28.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:28.178+0000 7f41f319f700 1 -- 172.21.15.114:0/2456776233 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f41d8038320 msgr2=0x7f41d803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:28.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:28.178+0000 7f41f319f700 1 --2- 172.21.15.114:0/2456776233 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f41d8038320 0x7f41d803a7e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f41dc006fd0 tx=0x7f41dc006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:28.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:28.178+0000 7f41f319f700 1 -- 172.21.15.114:0/2456776233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec104550 msgr2=0x7f41ec106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:28.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:28.178+0000 7f41f319f700 1 --2- 172.21.15.114:0/2456776233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec104550 0x7f41ec106e40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f41e8011670 tx=0x7f41e8011b50 comp rx=0 tx=0).stop 2024-02-19T18:52:28.729 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:28.178+0000 7f41f319f700 1 -- 172.21.15.114:0/2456776233 shutdown_connections 2024-02-19T18:52:28.729 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:28.178+0000 7f41f319f700 1 --2- 172.21.15.114:0/2456776233 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f41d8038320 0x7f41d803a7e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:28.729 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:28.178+0000 7f41f319f700 1 --2- 172.21.15.114:0/2456776233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41ec104550 0x7f41ec106e40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:28.729 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:28.178+0000 7f41f319f700 1 -- 172.21.15.114:0/2456776233 >> 172.21.15.114:0/2456776233 conn(0x7f41ec0ffe40 msgr2=0x7f41ec1015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:28.729 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:28.178+0000 7f41f319f700 1 -- 172.21.15.114:0/2456776233 shutdown_connections 2024-02-19T18:52:28.729 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:28.178+0000 7f41f319f700 1 -- 172.21.15.114:0/2456776233 wait complete. 2024-02-19T18:52:29.534 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:29 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/2456776233' entity='client.admin' 2024-02-19T18:52:29.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.121+0000 7ff0fad9f700 1 Processor -- start 2024-02-19T18:52:29.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.121+0000 7ff0fad9f700 1 -- start start 2024-02-19T18:52:29.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.121+0000 7ff0fad9f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f4105340 0x7ff0f4105760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:29.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.121+0000 7ff0fad9f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f4105d30 con 0x7ff0f4105340 2024-02-19T18:52:29.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.121+0000 7ff0f8b3b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f4105340 0x7ff0f4105760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:29.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.122+0000 7ff0f8b3b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f4105340 0x7ff0f4105760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57312/0 (socket says 172.21.15.114:57312) 2024-02-19T18:52:29.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.122+0000 7ff0f8b3b700 1 -- 172.21.15.114:0/2218014762 learned_addr learned my addr 172.21.15.114:0/2218014762 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:29.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.122+0000 7ff0f8b3b700 1 -- 172.21.15.114:0/2218014762 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f4106540 con 0x7ff0f4105340 2024-02-19T18:52:29.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.122+0000 7ff0f8b3b700 1 --2- 172.21.15.114:0/2218014762 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f4105340 0x7ff0f4105760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff0f00098a0 tx=0x7ff0f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d464c5c6f66ce83 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:29.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.122+0000 7ff0eb7fe700 1 -- 172.21.15.114:0/2218014762 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0f0004180 con 0x7ff0f4105340 2024-02-19T18:52:29.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.122+0000 7ff0eb7fe700 1 -- 172.21.15.114:0/2218014762 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff0f00042e0 con 0x7ff0f4105340 2024-02-19T18:52:29.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.122+0000 7ff0eb7fe700 1 -- 172.21.15.114:0/2218014762 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0f0011af0 con 0x7ff0f4105340 2024-02-19T18:52:29.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.123+0000 7ff0fad9f700 1 -- 172.21.15.114:0/2218014762 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f4105340 msgr2=0x7ff0f4105760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:29.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.123+0000 7ff0fad9f700 1 --2- 172.21.15.114:0/2218014762 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f4105340 0x7ff0f4105760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff0f00098a0 tx=0x7ff0f0009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:29.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.123+0000 7ff0fad9f700 1 -- 172.21.15.114:0/2218014762 shutdown_connections 2024-02-19T18:52:29.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.123+0000 7ff0fad9f700 1 --2- 172.21.15.114:0/2218014762 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f4105340 0x7ff0f4105760 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:29.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.123+0000 7ff0fad9f700 1 -- 172.21.15.114:0/2218014762 >> 172.21.15.114:0/2218014762 conn(0x7ff0f41008c0 msgr2=0x7ff0f4102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:29.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.123+0000 7ff0fad9f700 1 -- 172.21.15.114:0/2218014762 shutdown_connections 2024-02-19T18:52:29.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.123+0000 7ff0fad9f700 1 -- 172.21.15.114:0/2218014762 wait complete. 2024-02-19T18:52:29.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.123+0000 7ff0fad9f700 1 Processor -- start 2024-02-19T18:52:29.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.123+0000 7ff0fad9f700 1 -- start start 2024-02-19T18:52:29.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.123+0000 7ff0fad9f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f4105340 0x7ff0f4074360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:29.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.123+0000 7ff0fad9f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f419ce70 con 0x7ff0f4105340 2024-02-19T18:52:29.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.124+0000 7ff0f8b3b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f4105340 0x7ff0f4074360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:29.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.124+0000 7ff0f8b3b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f4105340 0x7ff0f4074360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57320/0 (socket says 172.21.15.114:57320) 2024-02-19T18:52:29.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.124+0000 7ff0f8b3b700 1 -- 172.21.15.114:0/3307704883 learned_addr learned my addr 172.21.15.114:0/3307704883 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:29.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.124+0000 7ff0f8b3b700 1 -- 172.21.15.114:0/3307704883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f0009550 con 0x7ff0f4105340 2024-02-19T18:52:29.951 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.124+0000 7ff0f8b3b700 1 --2- 172.21.15.114:0/3307704883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f4105340 0x7ff0f4074360 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0011670 tx=0x7ff0f0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:29.951 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.124+0000 7ff0e9ffb700 1 -- 172.21.15.114:0/3307704883 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0f00039c0 con 0x7ff0f4105340 2024-02-19T18:52:29.951 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.124+0000 7ff0e9ffb700 1 -- 172.21.15.114:0/3307704883 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff0f0003b20 con 0x7ff0f4105340 2024-02-19T18:52:29.951 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.124+0000 7ff0fad9f700 1 -- 172.21.15.114:0/3307704883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f419cfb0 con 0x7ff0f4105340 2024-02-19T18:52:29.951 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.124+0000 7ff0e9ffb700 1 -- 172.21.15.114:0/3307704883 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0f00247f0 con 0x7ff0f4105340 2024-02-19T18:52:29.951 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.124+0000 7ff0fad9f700 1 -- 172.21.15.114:0/3307704883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f419d450 con 0x7ff0f4105340 2024-02-19T18:52:29.951 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.125+0000 7ff0e9ffb700 1 -- 172.21.15.114:0/3307704883 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7ff0f001ac70 con 0x7ff0f4105340 2024-02-19T18:52:29.951 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.125+0000 7ff0fad9f700 1 -- 172.21.15.114:0/3307704883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f40623c0 con 0x7ff0f4105340 2024-02-19T18:52:29.951 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.125+0000 7ff0e9ffb700 1 --2- 172.21.15.114:0/3307704883 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7ff0dc038330 0x7ff0dc03a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:29.951 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.126+0000 7ff0e9ffb700 1 -- 172.21.15.114:0/3307704883 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff0f004c270 con 0x7ff0f4105340 2024-02-19T18:52:29.951 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.126+0000 7ff0ebfff700 1 --2- 172.21.15.114:0/3307704883 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7ff0dc038330 0x7ff0dc03a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:29.951 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.126+0000 7ff0ebfff700 1 --2- 172.21.15.114:0/3307704883 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7ff0dc038330 0x7ff0dc03a7f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4006fd0 tx=0x7ff0e4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:29.951 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.130+0000 7ff0e9ffb700 1 -- 172.21.15.114:0/3307704883 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff0f0018740 con 0x7ff0f4105340 2024-02-19T18:52:29.951 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.318+0000 7ff0fad9f700 1 -- 172.21.15.114:0/3307704883 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7ff0f4109770 con 0x7ff0f4105340 2024-02-19T18:52:29.951 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.322+0000 7ff0e9ffb700 1 -- 172.21.15.114:0/3307704883 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7ff0f0018350 con 0x7ff0f4105340 2024-02-19T18:52:29.952 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.324+0000 7ff0fad9f700 1 -- 172.21.15.114:0/3307704883 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7ff0dc038330 msgr2=0x7ff0dc03a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:29.952 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.324+0000 7ff0fad9f700 1 --2- 172.21.15.114:0/3307704883 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7ff0dc038330 0x7ff0dc03a7f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4006fd0 tx=0x7ff0e4006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:29.952 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.324+0000 7ff0fad9f700 1 -- 172.21.15.114:0/3307704883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f4105340 msgr2=0x7ff0f4074360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:29.952 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.324+0000 7ff0fad9f700 1 --2- 172.21.15.114:0/3307704883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f4105340 0x7ff0f4074360 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0011670 tx=0x7ff0f0011aa0 comp rx=0 tx=0).stop 2024-02-19T18:52:29.952 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.324+0000 7ff0fad9f700 1 -- 172.21.15.114:0/3307704883 shutdown_connections 2024-02-19T18:52:29.952 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.324+0000 7ff0fad9f700 1 --2- 172.21.15.114:0/3307704883 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7ff0dc038330 0x7ff0dc03a7f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:29.952 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.324+0000 7ff0fad9f700 1 --2- 172.21.15.114:0/3307704883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f4105340 0x7ff0f4074360 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:29.952 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.324+0000 7ff0fad9f700 1 -- 172.21.15.114:0/3307704883 >> 172.21.15.114:0/3307704883 conn(0x7ff0f41008c0 msgr2=0x7ff0f4101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:29.952 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.324+0000 7ff0fad9f700 1 -- 172.21.15.114:0/3307704883 shutdown_connections 2024-02-19T18:52:29.952 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:29.324+0000 7ff0fad9f700 1 -- 172.21.15.114:0/3307704883 wait complete. 2024-02-19T18:52:29.952 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling the dashboard module... 2024-02-19T18:52:30.534 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:30 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/3307704883' entity='client.admin' 2024-02-19T18:52:30.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:30 smithi114 ceph-mon[29441]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:31.655 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:31 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/4264241275' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-19T18:52:31.998 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.344+0000 7fb73bd3a700 1 Processor -- start 2024-02-19T18:52:31.998 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.344+0000 7fb73bd3a700 1 -- start start 2024-02-19T18:52:31.998 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.344+0000 7fb73bd3a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb73407bef0 0x7fb73407a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:31.998 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.344+0000 7fb73bd3a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73407a920 con 0x7fb73407bef0 2024-02-19T18:52:31.998 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.344+0000 7fb739ad6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb73407bef0 0x7fb73407a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:31.998 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.344+0000 7fb739ad6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb73407bef0 0x7fb73407a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57334/0 (socket says 172.21.15.114:57334) 2024-02-19T18:52:31.998 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.344+0000 7fb739ad6700 1 -- 172.21.15.114:0/375082751 learned_addr learned my addr 172.21.15.114:0/375082751 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:31.998 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.344+0000 7fb739ad6700 1 -- 172.21.15.114:0/375082751 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73407aa60 con 0x7fb73407bef0 2024-02-19T18:52:31.998 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.345+0000 7fb739ad6700 1 --2- 172.21.15.114:0/375082751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb73407bef0 0x7fb73407a350 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb7300098a0 tx=0x7fb730009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2ce72f95485ad39 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:31.998 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.345+0000 7fb738ad4700 1 -- 172.21.15.114:0/375082751 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb730004180 con 0x7fb73407bef0 2024-02-19T18:52:31.998 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.345+0000 7fb738ad4700 1 -- 172.21.15.114:0/375082751 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb7300042e0 con 0x7fb73407bef0 2024-02-19T18:52:31.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.345+0000 7fb738ad4700 1 -- 172.21.15.114:0/375082751 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb730011af0 con 0x7fb73407bef0 2024-02-19T18:52:31.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.345+0000 7fb73bd3a700 1 -- 172.21.15.114:0/375082751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb73407bef0 msgr2=0x7fb73407a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:31.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.345+0000 7fb73bd3a700 1 --2- 172.21.15.114:0/375082751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb73407bef0 0x7fb73407a350 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb7300098a0 tx=0x7fb730009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:31.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.345+0000 7fb73bd3a700 1 -- 172.21.15.114:0/375082751 shutdown_connections 2024-02-19T18:52:31.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.345+0000 7fb73bd3a700 1 --2- 172.21.15.114:0/375082751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb73407bef0 0x7fb73407a350 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:31.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.345+0000 7fb73bd3a700 1 -- 172.21.15.114:0/375082751 >> 172.21.15.114:0/375082751 conn(0x7fb734101a00 msgr2=0x7fb734103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:31.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.345+0000 7fb73bd3a700 1 -- 172.21.15.114:0/375082751 shutdown_connections 2024-02-19T18:52:31.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.345+0000 7fb73bd3a700 1 -- 172.21.15.114:0/375082751 wait complete. 2024-02-19T18:52:31.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.346+0000 7fb73bd3a700 1 Processor -- start 2024-02-19T18:52:31.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.346+0000 7fb73bd3a700 1 -- start start 2024-02-19T18:52:31.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.346+0000 7fb73bd3a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb73407bef0 0x7fb73419c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:31.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.346+0000 7fb73bd3a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73419c880 con 0x7fb73407bef0 2024-02-19T18:52:31.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.346+0000 7fb739ad6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb73407bef0 0x7fb73419c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:31.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.346+0000 7fb739ad6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb73407bef0 0x7fb73419c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57348/0 (socket says 172.21.15.114:57348) 2024-02-19T18:52:32.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.346+0000 7fb739ad6700 1 -- 172.21.15.114:0/4264241275 learned_addr learned my addr 172.21.15.114:0/4264241275 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:32.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.346+0000 7fb739ad6700 1 -- 172.21.15.114:0/4264241275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb730009550 con 0x7fb73407bef0 2024-02-19T18:52:32.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.347+0000 7fb739ad6700 1 --2- 172.21.15.114:0/4264241275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb73407bef0 0x7fb73419c340 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb730006ba0 tx=0x7fb730011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:32.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.347+0000 7fb72affd700 1 -- 172.21.15.114:0/4264241275 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb730003b00 con 0x7fb73407bef0 2024-02-19T18:52:32.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.347+0000 7fb72affd700 1 -- 172.21.15.114:0/4264241275 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb73001a8e0 con 0x7fb73407bef0 2024-02-19T18:52:32.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.347+0000 7fb72affd700 1 -- 172.21.15.114:0/4264241275 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb730019990 con 0x7fb73407bef0 2024-02-19T18:52:32.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.347+0000 7fb73bd3a700 1 -- 172.21.15.114:0/4264241275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73419ca80 con 0x7fb73407bef0 2024-02-19T18:52:32.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.347+0000 7fb73bd3a700 1 -- 172.21.15.114:0/4264241275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73419cf20 con 0x7fb73407bef0 2024-02-19T18:52:32.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.348+0000 7fb72affd700 1 -- 172.21.15.114:0/4264241275 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fb730019c00 con 0x7fb73407bef0 2024-02-19T18:52:32.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.348+0000 7fb73bd3a700 1 -- 172.21.15.114:0/4264241275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73410c7f0 con 0x7fb73407bef0 2024-02-19T18:52:32.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.348+0000 7fb72affd700 1 --2- 172.21.15.114:0/4264241275 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fb720038320 0x7fb72003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:32.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.348+0000 7fb72affd700 1 -- 172.21.15.114:0/4264241275 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb73004c6e0 con 0x7fb73407bef0 2024-02-19T18:52:32.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.348+0000 7fb7392d5700 1 --2- 172.21.15.114:0/4264241275 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fb720038320 0x7fb72003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:32.001 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.348+0000 7fb7392d5700 1 --2- 172.21.15.114:0/4264241275 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fb720038320 0x7fb72003a7e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb724006fd0 tx=0x7fb724006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:32.001 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.353+0000 7fb72affd700 1 -- 172.21.15.114:0/4264241275 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb73001a430 con 0x7fb73407bef0 2024-02-19T18:52:32.001 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:30.593+0000 7fb73bd3a700 1 -- 172.21.15.114:0/4264241275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fb73410ca10 con 0x7fb73407bef0 2024-02-19T18:52:32.001 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:31.326+0000 7fb72affd700 1 -- 172.21.15.114:0/4264241275 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fb730023390 con 0x7fb73407bef0 2024-02-19T18:52:32.001 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:31.328+0000 7fb72affd700 1 -- 172.21.15.114:0/4264241275 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fb730024c20 con 0x7fb73407bef0 2024-02-19T18:52:32.001 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:31.330+0000 7fb73bd3a700 1 -- 172.21.15.114:0/4264241275 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fb720038320 msgr2=0x7fb72003a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:32.001 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:31.330+0000 7fb73bd3a700 1 --2- 172.21.15.114:0/4264241275 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fb720038320 0x7fb72003a7e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb724006fd0 tx=0x7fb724006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:32.001 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:31.330+0000 7fb73bd3a700 1 -- 172.21.15.114:0/4264241275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb73407bef0 msgr2=0x7fb73419c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:32.001 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:31.330+0000 7fb73bd3a700 1 --2- 172.21.15.114:0/4264241275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb73407bef0 0x7fb73419c340 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb730006ba0 tx=0x7fb730011b50 comp rx=0 tx=0).stop 2024-02-19T18:52:32.001 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:31.330+0000 7fb73bd3a700 1 -- 172.21.15.114:0/4264241275 shutdown_connections 2024-02-19T18:52:32.001 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:31.330+0000 7fb73bd3a700 1 --2- 172.21.15.114:0/4264241275 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7fb720038320 0x7fb72003a7e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:32.001 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:31.330+0000 7fb73bd3a700 1 --2- 172.21.15.114:0/4264241275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb73407bef0 0x7fb73419c340 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:32.001 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:31.330+0000 7fb73bd3a700 1 -- 172.21.15.114:0/4264241275 >> 172.21.15.114:0/4264241275 conn(0x7fb734101a00 msgr2=0x7fb7341030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:32.001 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:31.331+0000 7fb73bd3a700 1 -- 172.21.15.114:0/4264241275 shutdown_connections 2024-02-19T18:52:32.001 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:31.331+0000 7fb73bd3a700 1 -- 172.21.15.114:0/4264241275 wait complete. 2024-02-19T18:52:32.621 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:32 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/4264241275' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-19T18:52:32.621 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:32 smithi114 ceph-mon[29441]: mgrmap e9: smithi114.pcvtrk(active, since 25s) 2024-02-19T18:52:33.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:52:33.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-19T18:52:33.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-19T18:52:33.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "active_name": "smithi114.pcvtrk", 2024-02-19T18:52:33.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-19T18:52:33.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:52:33.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.399+0000 7f744c775700 1 Processor -- start 2024-02-19T18:52:33.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.400+0000 7f744c775700 1 -- start start 2024-02-19T18:52:33.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.400+0000 7f744c775700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7444106ef0 0x7f74441092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:33.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.400+0000 7f744c775700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7444074640 con 0x7f7444106ef0 2024-02-19T18:52:33.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.400+0000 7f744a511700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7444106ef0 0x7f74441092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:33.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.400+0000 7f744a511700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7444106ef0 0x7f74441092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57366/0 (socket says 172.21.15.114:57366) 2024-02-19T18:52:33.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.400+0000 7f744a511700 1 -- 172.21.15.114:0/291239449 learned_addr learned my addr 172.21.15.114:0/291239449 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:33.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.400+0000 7f744a511700 1 -- 172.21.15.114:0/291239449 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7444109820 con 0x7f7444106ef0 2024-02-19T18:52:33.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.400+0000 7f744a511700 1 --2- 172.21.15.114:0/291239449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7444106ef0 0x7f74441092e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f74400098a0 tx=0x7f7440009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537a49d7b527508 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:33.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.401+0000 7f744950f700 1 -- 172.21.15.114:0/291239449 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7440004180 con 0x7f7444106ef0 2024-02-19T18:52:33.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.401+0000 7f744950f700 1 -- 172.21.15.114:0/291239449 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f74400042e0 con 0x7f7444106ef0 2024-02-19T18:52:33.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.401+0000 7f744950f700 1 -- 172.21.15.114:0/291239449 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7440011af0 con 0x7f7444106ef0 2024-02-19T18:52:33.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.401+0000 7f744c775700 1 -- 172.21.15.114:0/291239449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7444106ef0 msgr2=0x7f74441092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:33.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.401+0000 7f744c775700 1 --2- 172.21.15.114:0/291239449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7444106ef0 0x7f74441092e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f74400098a0 tx=0x7f7440009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:33.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.401+0000 7f744c775700 1 -- 172.21.15.114:0/291239449 shutdown_connections 2024-02-19T18:52:33.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.401+0000 7f744c775700 1 --2- 172.21.15.114:0/291239449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7444106ef0 0x7f74441092e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:33.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.401+0000 7f744c775700 1 -- 172.21.15.114:0/291239449 >> 172.21.15.114:0/291239449 conn(0x7f74441008c0 msgr2=0x7f7444102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:33.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.402+0000 7f744c775700 1 -- 172.21.15.114:0/291239449 shutdown_connections 2024-02-19T18:52:33.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.402+0000 7f744c775700 1 -- 172.21.15.114:0/291239449 wait complete. 2024-02-19T18:52:33.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.402+0000 7f744c775700 1 Processor -- start 2024-02-19T18:52:33.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.402+0000 7f744c775700 1 -- start start 2024-02-19T18:52:33.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.402+0000 7f744c775700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7444106ef0 0x7f7444106850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:33.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.402+0000 7f744c775700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7444106d90 con 0x7f7444106ef0 2024-02-19T18:52:33.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.402+0000 7f744a511700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7444106ef0 0x7f7444106850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:33.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.402+0000 7f744a511700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7444106ef0 0x7f7444106850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57378/0 (socket says 172.21.15.114:57378) 2024-02-19T18:52:33.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.402+0000 7f744a511700 1 -- 172.21.15.114:0/876842695 learned_addr learned my addr 172.21.15.114:0/876842695 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:33.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.403+0000 7f744a511700 1 -- 172.21.15.114:0/876842695 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7440009550 con 0x7f7444106ef0 2024-02-19T18:52:33.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.403+0000 7f744a511700 1 --2- 172.21.15.114:0/876842695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7444106ef0 0x7f7444106850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7440011670 tx=0x7f7440011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:33.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.403+0000 7f743b7fe700 1 -- 172.21.15.114:0/876842695 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7440003a40 con 0x7f7444106ef0 2024-02-19T18:52:33.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.403+0000 7f743b7fe700 1 -- 172.21.15.114:0/876842695 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7440003ba0 con 0x7f7444106ef0 2024-02-19T18:52:33.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.403+0000 7f743b7fe700 1 -- 172.21.15.114:0/876842695 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7440024820 con 0x7f7444106ef0 2024-02-19T18:52:33.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.403+0000 7f744c775700 1 -- 172.21.15.114:0/876842695 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7444104ea0 con 0x7f7444106ef0 2024-02-19T18:52:33.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.403+0000 7f744c775700 1 -- 172.21.15.114:0/876842695 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7444105340 con 0x7f7444106ef0 2024-02-19T18:52:33.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.404+0000 7f743b7fe700 1 -- 172.21.15.114:0/876842695 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f7440024ae0 con 0x7f7444106ef0 2024-02-19T18:52:33.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.404+0000 7f744c775700 1 -- 172.21.15.114:0/876842695 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7444062430 con 0x7f7444106ef0 2024-02-19T18:52:33.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.404+0000 7f743b7fe700 1 --2- 172.21.15.114:0/876842695 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f7430038330 0x7f743003a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:33.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.404+0000 7f743b7fe700 1 -- 172.21.15.114:0/876842695 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f744004d380 con 0x7f7444106ef0 2024-02-19T18:52:33.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.404+0000 7f7449d10700 1 -- 172.21.15.114:0/876842695 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f7430038330 msgr2=0x7f743003a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/971741305 2024-02-19T18:52:33.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.404+0000 7f7449d10700 1 --2- 172.21.15.114:0/876842695 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f7430038330 0x7f743003a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:52:33.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.409+0000 7f743b7fe700 1 -- 172.21.15.114:0/876842695 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7444062430 con 0x7f7444106ef0 2024-02-19T18:52:33.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.605+0000 7f7449d10700 1 -- 172.21.15.114:0/876842695 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f7430038330 msgr2=0x7f743003a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/971741305 2024-02-19T18:52:33.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.605+0000 7f7449d10700 1 --2- 172.21.15.114:0/876842695 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f7430038330 0x7f743003a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T18:52:33.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.658+0000 7f744c775700 1 -- 172.21.15.114:0/876842695 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7444105690 con 0x7f7444106ef0 2024-02-19T18:52:33.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.659+0000 7f743b7fe700 1 -- 172.21.15.114:0/876842695 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f7440022310 con 0x7f7444106ef0 2024-02-19T18:52:33.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.661+0000 7f744c775700 1 -- 172.21.15.114:0/876842695 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f7430038330 msgr2=0x7f743003a7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:52:33.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.661+0000 7f744c775700 1 --2- 172.21.15.114:0/876842695 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f7430038330 0x7f743003a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:33.259 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.661+0000 7f744c775700 1 -- 172.21.15.114:0/876842695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7444106ef0 msgr2=0x7f7444106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:33.259 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.661+0000 7f744c775700 1 --2- 172.21.15.114:0/876842695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7444106ef0 0x7f7444106850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7440011670 tx=0x7f7440011aa0 comp rx=0 tx=0).stop 2024-02-19T18:52:33.259 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.661+0000 7f744c775700 1 -- 172.21.15.114:0/876842695 shutdown_connections 2024-02-19T18:52:33.259 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.661+0000 7f744c775700 1 --2- 172.21.15.114:0/876842695 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f7430038330 0x7f743003a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:33.259 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.661+0000 7f744c775700 1 --2- 172.21.15.114:0/876842695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7444106ef0 0x7f7444106850 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:33.259 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.661+0000 7f744c775700 1 -- 172.21.15.114:0/876842695 >> 172.21.15.114:0/876842695 conn(0x7f74441008c0 msgr2=0x7f7444101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:33.259 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.661+0000 7f744c775700 1 -- 172.21.15.114:0/876842695 shutdown_connections 2024-02-19T18:52:33.259 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:32.661+0000 7f744c775700 1 -- 172.21.15.114:0/876842695 wait complete. 2024-02-19T18:52:33.259 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for the mgr to restart... 2024-02-19T18:52:33.259 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr epoch 9... 2024-02-19T18:52:33.505 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:33 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/876842695' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T18:52:39.108 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:38 smithi114 ceph-mon[29441]: Active manager daemon smithi114.pcvtrk restarted 2024-02-19T18:52:39.108 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:38 smithi114 ceph-mon[29441]: Activating manager daemon smithi114.pcvtrk 2024-02-19T18:52:39.108 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:38 smithi114 ceph-mon[29441]: osdmap e3: 0 total, 0 up, 0 in 2024-02-19T18:52:39.108 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:38 smithi114 ceph-mon[29441]: mgrmap e10: smithi114.pcvtrk(active, starting, since 0.00369879s) 2024-02-19T18:52:39.108 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:38 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T18:52:39.108 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:38 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr metadata", "who": "smithi114.pcvtrk", "id": "smithi114.pcvtrk"}]: dispatch 2024-02-19T18:52:39.108 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:38 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T18:52:39.108 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:38 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T18:52:39.108 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:38 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T18:52:39.108 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:38 smithi114 ceph-mon[29441]: Manager daemon smithi114.pcvtrk is now available 2024-02-19T18:52:39.109 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:38 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:52:39.109 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:38 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:52:39.109 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:38 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T18:52:39.109 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:38 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/trash_purge_schedule"}]: dispatch 2024-02-19T18:52:39.832 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:52:39.832 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-19T18:52:39.832 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-19T18:52:39.832 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:52:39.832 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.727+0000 7f0306d4e700 1 Processor -- start 2024-02-19T18:52:39.832 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.727+0000 7f0306d4e700 1 -- start start 2024-02-19T18:52:39.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.727+0000 7f0306d4e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0300106f00 0x7f03001092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:39.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.727+0000 7f0306d4e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0300079e80 con 0x7f0300106f00 2024-02-19T18:52:39.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.727+0000 7f0304aea700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0300106f00 0x7f03001092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:39.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.727+0000 7f0304aea700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0300106f00 0x7f03001092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57380/0 (socket says 172.21.15.114:57380) 2024-02-19T18:52:39.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.727+0000 7f0304aea700 1 -- 172.21.15.114:0/2159055302 learned_addr learned my addr 172.21.15.114:0/2159055302 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:39.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.727+0000 7f0304aea700 1 -- 172.21.15.114:0/2159055302 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0300109830 con 0x7f0300106f00 2024-02-19T18:52:39.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.728+0000 7f0304aea700 1 --2- 172.21.15.114:0/2159055302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0300106f00 0x7f03001092f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f02f40098a0 tx=0x7f02f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c205a11df7160288 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:39.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.728+0000 7f02ff7fe700 1 -- 172.21.15.114:0/2159055302 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02f4004180 con 0x7f0300106f00 2024-02-19T18:52:39.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.728+0000 7f02ff7fe700 1 -- 172.21.15.114:0/2159055302 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f02f40042e0 con 0x7f0300106f00 2024-02-19T18:52:39.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.728+0000 7f02ff7fe700 1 -- 172.21.15.114:0/2159055302 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02f4011af0 con 0x7f0300106f00 2024-02-19T18:52:39.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.728+0000 7f0306d4e700 1 -- 172.21.15.114:0/2159055302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0300106f00 msgr2=0x7f03001092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:39.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.728+0000 7f0306d4e700 1 --2- 172.21.15.114:0/2159055302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0300106f00 0x7f03001092f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f02f40098a0 tx=0x7f02f4009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:39.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.729+0000 7f0306d4e700 1 -- 172.21.15.114:0/2159055302 shutdown_connections 2024-02-19T18:52:39.834 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.729+0000 7f0306d4e700 1 --2- 172.21.15.114:0/2159055302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0300106f00 0x7f03001092f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:39.834 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.729+0000 7f0306d4e700 1 -- 172.21.15.114:0/2159055302 >> 172.21.15.114:0/2159055302 conn(0x7f03001008d0 msgr2=0x7f0300102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:39.834 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.729+0000 7f0306d4e700 1 -- 172.21.15.114:0/2159055302 shutdown_connections 2024-02-19T18:52:39.834 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.729+0000 7f0306d4e700 1 -- 172.21.15.114:0/2159055302 wait complete. 2024-02-19T18:52:39.834 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.729+0000 7f0306d4e700 1 Processor -- start 2024-02-19T18:52:39.834 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.729+0000 7f0306d4e700 1 -- start start 2024-02-19T18:52:39.834 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.729+0000 7f0306d4e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0300106f00 0x7f0300106860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:39.834 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.729+0000 7f0306d4e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0300106da0 con 0x7f0300106f00 2024-02-19T18:52:39.834 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.729+0000 7f0304aea700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0300106f00 0x7f0300106860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:39.834 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.729+0000 7f0304aea700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0300106f00 0x7f0300106860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57396/0 (socket says 172.21.15.114:57396) 2024-02-19T18:52:39.834 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.729+0000 7f0304aea700 1 -- 172.21.15.114:0/1129856136 learned_addr learned my addr 172.21.15.114:0/1129856136 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:39.834 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.730+0000 7f0304aea700 1 -- 172.21.15.114:0/1129856136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f4009550 con 0x7f0300106f00 2024-02-19T18:52:39.834 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.730+0000 7f0304aea700 1 --2- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0300106f00 0x7f0300106860 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f02f4011670 tx=0x7f02f4011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:39.834 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.730+0000 7f02fdffb700 1 -- 172.21.15.114:0/1129856136 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02f4003a40 con 0x7f0300106f00 2024-02-19T18:52:39.834 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.730+0000 7f02fdffb700 1 -- 172.21.15.114:0/1129856136 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f02f4003ba0 con 0x7f0300106f00 2024-02-19T18:52:39.835 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.730+0000 7f0306d4e700 1 -- 172.21.15.114:0/1129856136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0300104eb0 con 0x7f0300106f00 2024-02-19T18:52:39.835 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.730+0000 7f02fdffb700 1 -- 172.21.15.114:0/1129856136 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02f4024790 con 0x7f0300106f00 2024-02-19T18:52:39.835 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.730+0000 7f0306d4e700 1 -- 172.21.15.114:0/1129856136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0300105350 con 0x7f0300106f00 2024-02-19T18:52:39.835 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.731+0000 7f02fdffb700 1 -- 172.21.15.114:0/1129856136 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f02f402d3f0 con 0x7f0300106f00 2024-02-19T18:52:39.835 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.731+0000 7f02fdffb700 1 --2- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f02f0038370 0x7f02f003a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:39.835 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.731+0000 7f0306d4e700 1 -- 172.21.15.114:0/1129856136 --> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f030010a5c0 con 0x7f02f0038370 2024-02-19T18:52:39.835 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.731+0000 7f02fdffb700 1 -- 172.21.15.114:0/1129856136 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f02f404d500 con 0x7f0300106f00 2024-02-19T18:52:39.835 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.731+0000 7f02fffff700 1 -- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f02f0038370 msgr2=0x7f02f003a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/971741305 2024-02-19T18:52:39.835 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.731+0000 7f02fffff700 1 --2- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f02f0038370 0x7f02f003a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:52:39.835 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.932+0000 7f02fffff700 1 -- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f02f0038370 msgr2=0x7f02f003a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/971741305 2024-02-19T18:52:39.835 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:33.932+0000 7f02fffff700 1 --2- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f02f0038370 0x7f02f003a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T18:52:39.835 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:34.333+0000 7f02fffff700 1 -- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f02f0038370 msgr2=0x7f02f003a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/971741305 2024-02-19T18:52:39.835 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:34.333+0000 7f02fffff700 1 --2- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f02f0038370 0x7f02f003a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-19T18:52:39.836 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:35.134+0000 7f02fffff700 1 -- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f02f0038370 msgr2=0x7f02f003a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/971741305 2024-02-19T18:52:39.836 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:35.134+0000 7f02fffff700 1 --2- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f02f0038370 0x7f02f003a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-19T18:52:39.836 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:36.735+0000 7f02fffff700 1 -- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f02f0038370 msgr2=0x7f02f003a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/971741305 2024-02-19T18:52:39.836 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:36.735+0000 7f02fffff700 1 --2- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f02f0038370 0x7f02f003a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-19T18:52:39.836 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:38.089+0000 7f02fdffb700 1 -- 172.21.15.114:0/1129856136 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mgrmap(e 10) v1 ==== 45063+0+0 (secure 0 0 0) 0x7f02f4021d60 con 0x7f0300106f00 2024-02-19T18:52:39.836 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:38.089+0000 7f02fdffb700 1 -- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f02f0038370 msgr2=0x7f02f003a830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:52:39.836 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:38.089+0000 7f02fdffb700 1 --2- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f02f0038370 0x7f02f003a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:39.836 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.090+0000 7f02fdffb700 1 -- 172.21.15.114:0/1129856136 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 11) v1 ==== 45190+0+0 (secure 0 0 0) 0x7f02f404e090 con 0x7f0300106f00 2024-02-19T18:52:39.836 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.090+0000 7f02fdffb700 1 --2- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f02f003bc30 0x7f02f003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:39.836 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.090+0000 7f02fdffb700 1 -- 172.21.15.114:0/1129856136 --> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f030010a5c0 con 0x7f02f003bc30 2024-02-19T18:52:39.836 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.091+0000 7f02fffff700 1 --2- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f02f003bc30 0x7f02f003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:39.836 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.091+0000 7f02fffff700 1 --2- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f02f003bc30 0x7f02f003e020 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f02ec003a10 tx=0x7f02ec0092b0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:39.837 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.092+0000 7f02fdffb700 1 -- 172.21.15.114:0/1129856136 <== mgr.14162 v2:172.21.15.114:6800/1719431365 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f030010a5c0 con 0x7f02f003bc30 2024-02-19T18:52:39.837 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.098+0000 7f0306d4e700 1 -- 172.21.15.114:0/1129856136 --> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f0300106400 con 0x7f02f003bc30 2024-02-19T18:52:39.837 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.098+0000 7f02fdffb700 1 -- 172.21.15.114:0/1129856136 <== mgr.14162 v2:172.21.15.114:6800/1719431365 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f0300106400 con 0x7f02f003bc30 2024-02-19T18:52:39.837 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.099+0000 7f0306d4e700 1 -- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f02f003bc30 msgr2=0x7f02f003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:39.837 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.099+0000 7f0306d4e700 1 --2- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f02f003bc30 0x7f02f003e020 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f02ec003a10 tx=0x7f02ec0092b0 comp rx=0 tx=0).stop 2024-02-19T18:52:39.837 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.099+0000 7f0306d4e700 1 -- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0300106f00 msgr2=0x7f0300106860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:39.837 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.099+0000 7f0306d4e700 1 --2- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0300106f00 0x7f0300106860 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f02f4011670 tx=0x7f02f4011b50 comp rx=0 tx=0).stop 2024-02-19T18:52:39.837 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.099+0000 7f0306d4e700 1 -- 172.21.15.114:0/1129856136 shutdown_connections 2024-02-19T18:52:39.837 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.099+0000 7f0306d4e700 1 --2- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f02f003bc30 0x7f02f003e020 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:39.837 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.099+0000 7f0306d4e700 1 --2- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0300106f00 0x7f0300106860 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:39.837 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.099+0000 7f0306d4e700 1 --2- 172.21.15.114:0/1129856136 >> [v2:172.21.15.114:6800/971741305,v1:172.21.15.114:6801/971741305] conn(0x7f02f0038370 0x7f02f003a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:39.837 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.099+0000 7f0306d4e700 1 -- 172.21.15.114:0/1129856136 >> 172.21.15.114:0/1129856136 conn(0x7f03001008d0 msgr2=0x7f0300101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:39.837 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.099+0000 7f0306d4e700 1 -- 172.21.15.114:0/1129856136 shutdown_connections 2024-02-19T18:52:39.838 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:39.099+0000 7f0306d4e700 1 -- 172.21.15.114:0/1129856136 wait complete. 2024-02-19T18:52:39.838 INFO:teuthology.orchestra.run.smithi114.stdout:mgr epoch 9 is available 2024-02-19T18:52:39.838 INFO:teuthology.orchestra.run.smithi114.stdout:Generating a dashboard self-signed certificate... 2024-02-19T18:52:40.501 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:40 smithi114 ceph-mon[29441]: [19/Feb/2024:18:52:38] ENGINE Bus STARTING 2024-02-19T18:52:40.501 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:40 smithi114 ceph-mon[29441]: [19/Feb/2024:18:52:38] ENGINE Serving on http://172.21.15.114:8765 2024-02-19T18:52:40.502 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:40 smithi114 ceph-mon[29441]: [19/Feb/2024:18:52:38] ENGINE Serving on https://172.21.15.114:7150 2024-02-19T18:52:40.502 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:40 smithi114 ceph-mon[29441]: [19/Feb/2024:18:52:38] ENGINE Bus STARTED 2024-02-19T18:52:40.502 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:40 smithi114 ceph-mon[29441]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T18:52:40.502 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:40 smithi114 ceph-mon[29441]: mgrmap e11: smithi114.pcvtrk(active, since 1.00589s) 2024-02-19T18:52:40.502 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:40 smithi114 ceph-mon[29441]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T18:52:40.502 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:40 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:41.490 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:41 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:41.490 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:41 smithi114 ceph-mon[29441]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:52:41.490 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:41 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:41.490 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:41 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:41.490 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:41 smithi114 ceph-mon[29441]: mgrmap e12: smithi114.pcvtrk(active, since 3s) 2024-02-19T18:52:41.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-19T18:52:41.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.270+0000 7f1b88bd8700 1 Processor -- start 2024-02-19T18:52:41.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.270+0000 7f1b88bd8700 1 -- start start 2024-02-19T18:52:41.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.270+0000 7f1b88bd8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b84104d50 0x7f1b84105170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:41.497 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.270+0000 7f1b88bd8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b84079e20 con 0x7f1b84104d50 2024-02-19T18:52:41.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.270+0000 7f1b8259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b84104d50 0x7f1b84105170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:41.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.270+0000 7f1b8259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b84104d50 0x7f1b84105170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60236/0 (socket says 172.21.15.114:60236) 2024-02-19T18:52:41.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.270+0000 7f1b8259c700 1 -- 172.21.15.114:0/647435797 learned_addr learned my addr 172.21.15.114:0/647435797 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:41.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.271+0000 7f1b8259c700 1 -- 172.21.15.114:0/647435797 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b84079f60 con 0x7f1b84104d50 2024-02-19T18:52:41.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.271+0000 7f1b8259c700 1 --2- 172.21.15.114:0/647435797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b84104d50 0x7f1b84105170 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1b740098a0 tx=0x7f1b74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6d9d79654812c7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:41.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.271+0000 7f1b8159a700 1 -- 172.21.15.114:0/647435797 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b74004180 con 0x7f1b84104d50 2024-02-19T18:52:41.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.271+0000 7f1b8159a700 1 -- 172.21.15.114:0/647435797 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1b740042e0 con 0x7f1b84104d50 2024-02-19T18:52:41.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.271+0000 7f1b8159a700 1 -- 172.21.15.114:0/647435797 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b74011af0 con 0x7f1b84104d50 2024-02-19T18:52:41.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.272+0000 7f1b88bd8700 1 -- 172.21.15.114:0/647435797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b84104d50 msgr2=0x7f1b84105170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:41.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.272+0000 7f1b88bd8700 1 --2- 172.21.15.114:0/647435797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b84104d50 0x7f1b84105170 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1b740098a0 tx=0x7f1b74009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:41.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.272+0000 7f1b88bd8700 1 -- 172.21.15.114:0/647435797 shutdown_connections 2024-02-19T18:52:41.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.272+0000 7f1b88bd8700 1 --2- 172.21.15.114:0/647435797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b84104d50 0x7f1b84105170 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:41.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.272+0000 7f1b88bd8700 1 -- 172.21.15.114:0/647435797 >> 172.21.15.114:0/647435797 conn(0x7f1b841008d0 msgr2=0x7f1b84102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:41.498 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.272+0000 7f1b88bd8700 1 -- 172.21.15.114:0/647435797 shutdown_connections 2024-02-19T18:52:41.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.272+0000 7f1b88bd8700 1 -- 172.21.15.114:0/647435797 wait complete. 2024-02-19T18:52:41.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.273+0000 7f1b88bd8700 1 Processor -- start 2024-02-19T18:52:41.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.273+0000 7f1b88bd8700 1 -- start start 2024-02-19T18:52:41.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.273+0000 7f1b88bd8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b84104d50 0x7f1b8419c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:41.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.273+0000 7f1b88bd8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8419c890 con 0x7f1b84104d50 2024-02-19T18:52:41.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.273+0000 7f1b8259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b84104d50 0x7f1b8419c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:41.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.273+0000 7f1b8259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b84104d50 0x7f1b8419c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60244/0 (socket says 172.21.15.114:60244) 2024-02-19T18:52:41.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.273+0000 7f1b8259c700 1 -- 172.21.15.114:0/3706486263 learned_addr learned my addr 172.21.15.114:0/3706486263 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:41.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.273+0000 7f1b8259c700 1 -- 172.21.15.114:0/3706486263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b74009550 con 0x7f1b84104d50 2024-02-19T18:52:41.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.273+0000 7f1b8259c700 1 --2- 172.21.15.114:0/3706486263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b84104d50 0x7f1b8419c350 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1b74011670 tx=0x7f1b74011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:41.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.273+0000 7f1b737fe700 1 -- 172.21.15.114:0/3706486263 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b74003a40 con 0x7f1b84104d50 2024-02-19T18:52:41.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.274+0000 7f1b88bd8700 1 -- 172.21.15.114:0/3706486263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8419ca90 con 0x7f1b84104d50 2024-02-19T18:52:41.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.274+0000 7f1b88bd8700 1 -- 172.21.15.114:0/3706486263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8419cf30 con 0x7f1b84104d50 2024-02-19T18:52:41.499 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.274+0000 7f1b737fe700 1 -- 172.21.15.114:0/3706486263 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1b74003ba0 con 0x7f1b84104d50 2024-02-19T18:52:41.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.274+0000 7f1b737fe700 1 -- 172.21.15.114:0/3706486263 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b740246b0 con 0x7f1b84104d50 2024-02-19T18:52:41.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.274+0000 7f1b88bd8700 1 -- 172.21.15.114:0/3706486263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b64005320 con 0x7f1b84104d50 2024-02-19T18:52:41.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.275+0000 7f1b737fe700 1 -- 172.21.15.114:0/3706486263 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 11) v1 ==== 45190+0+0 (secure 0 0 0) 0x7f1b7402d3f0 con 0x7f1b84104d50 2024-02-19T18:52:41.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.275+0000 7f1b737fe700 1 --2- 172.21.15.114:0/3706486263 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f1b6c038250 0x7f1b6c03a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:41.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.275+0000 7f1b737fe700 1 -- 172.21.15.114:0/3706486263 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1b7404c490 con 0x7f1b84104d50 2024-02-19T18:52:41.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.275+0000 7f1b81d9b700 1 --2- 172.21.15.114:0/3706486263 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f1b6c038250 0x7f1b6c03a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:41.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.276+0000 7f1b81d9b700 1 --2- 172.21.15.114:0/3706486263 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f1b6c038250 0x7f1b6c03a710 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1b78006fd0 tx=0x7f1b78006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:41.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.280+0000 7f1b737fe700 1 -- 172.21.15.114:0/3706486263 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1b74010c00 con 0x7f1b84104d50 2024-02-19T18:52:41.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.485+0000 7f1b88bd8700 1 -- 172.21.15.114:0/3706486263 --> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b64000bf0 con 0x7f1b6c038250 2024-02-19T18:52:41.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.841+0000 7f1b737fe700 1 -- 172.21.15.114:0/3706486263 <== mgr.14162 v2:172.21.15.114:6800/1719431365 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f1b64000bf0 con 0x7f1b6c038250 2024-02-19T18:52:41.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.843+0000 7f1b88bd8700 1 -- 172.21.15.114:0/3706486263 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f1b6c038250 msgr2=0x7f1b6c03a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:41.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.843+0000 7f1b88bd8700 1 --2- 172.21.15.114:0/3706486263 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f1b6c038250 0x7f1b6c03a710 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1b78006fd0 tx=0x7f1b78006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:41.500 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.843+0000 7f1b88bd8700 1 -- 172.21.15.114:0/3706486263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b84104d50 msgr2=0x7f1b8419c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:41.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.843+0000 7f1b88bd8700 1 --2- 172.21.15.114:0/3706486263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b84104d50 0x7f1b8419c350 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1b74011670 tx=0x7f1b74011b50 comp rx=0 tx=0).stop 2024-02-19T18:52:41.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.843+0000 7f1b88bd8700 1 -- 172.21.15.114:0/3706486263 shutdown_connections 2024-02-19T18:52:41.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.843+0000 7f1b88bd8700 1 --2- 172.21.15.114:0/3706486263 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f1b6c038250 0x7f1b6c03a710 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:41.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.843+0000 7f1b88bd8700 1 --2- 172.21.15.114:0/3706486263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b84104d50 0x7f1b8419c350 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:41.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.843+0000 7f1b88bd8700 1 -- 172.21.15.114:0/3706486263 >> 172.21.15.114:0/3706486263 conn(0x7f1b841008d0 msgr2=0x7f1b84190910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:41.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.843+0000 7f1b88bd8700 1 -- 172.21.15.114:0/3706486263 shutdown_connections 2024-02-19T18:52:41.501 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:40.843+0000 7f1b88bd8700 1 -- 172.21.15.114:0/3706486263 wait complete. 2024-02-19T18:52:41.501 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial admin user... 2024-02-19T18:52:43.667 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$t/MBWDmEWEH8fKtQ2y1E2OhjLmjt3Lc2bZYFzhDGD42TfFBdSQdu2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708368763, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-19T18:52:43.668 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.676+0000 7f0c21128700 1 Processor -- start 2024-02-19T18:52:43.668 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.676+0000 7f0c21128700 1 -- start start 2024-02-19T18:52:43.668 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.676+0000 7f0c21128700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c104730 0x7f0c1c104b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:43.668 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.676+0000 7f0c21128700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c105120 con 0x7f0c1c104730 2024-02-19T18:52:43.668 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.676+0000 7f0c1ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c104730 0x7f0c1c104b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:43.668 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.676+0000 7f0c1ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c104730 0x7f0c1c104b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60258/0 (socket says 172.21.15.114:60258) 2024-02-19T18:52:43.668 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.676+0000 7f0c1ad9d700 1 -- 172.21.15.114:0/2216106532 learned_addr learned my addr 172.21.15.114:0/2216106532 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:43.668 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.676+0000 7f0c1ad9d700 1 -- 172.21.15.114:0/2216106532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c1c1076c0 con 0x7f0c1c104730 2024-02-19T18:52:43.668 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.677+0000 7f0c1ad9d700 1 --2- 172.21.15.114:0/2216106532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c104730 0x7f0c1c104b50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c0098a0 tx=0x7f0c0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f427caf0705d8799 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:43.668 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.677+0000 7f0c19d9b700 1 -- 172.21.15.114:0/2216106532 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c0c004180 con 0x7f0c1c104730 2024-02-19T18:52:43.668 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.677+0000 7f0c19d9b700 1 -- 172.21.15.114:0/2216106532 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0c0c0042e0 con 0x7f0c1c104730 2024-02-19T18:52:43.668 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.677+0000 7f0c19d9b700 1 -- 172.21.15.114:0/2216106532 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c0c011af0 con 0x7f0c1c104730 2024-02-19T18:52:43.668 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.677+0000 7f0c21128700 1 -- 172.21.15.114:0/2216106532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c104730 msgr2=0x7f0c1c104b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:43.669 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.677+0000 7f0c21128700 1 --2- 172.21.15.114:0/2216106532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c104730 0x7f0c1c104b50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c0098a0 tx=0x7f0c0c009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:43.669 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.677+0000 7f0c21128700 1 -- 172.21.15.114:0/2216106532 shutdown_connections 2024-02-19T18:52:43.669 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.677+0000 7f0c21128700 1 --2- 172.21.15.114:0/2216106532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c104730 0x7f0c1c104b50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:43.669 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.677+0000 7f0c21128700 1 -- 172.21.15.114:0/2216106532 >> 172.21.15.114:0/2216106532 conn(0x7f0c1c100040 msgr2=0x7f0c1c1024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:43.669 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.677+0000 7f0c21128700 1 -- 172.21.15.114:0/2216106532 shutdown_connections 2024-02-19T18:52:43.669 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.677+0000 7f0c21128700 1 -- 172.21.15.114:0/2216106532 wait complete. 2024-02-19T18:52:43.669 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.678+0000 7f0c21128700 1 Processor -- start 2024-02-19T18:52:43.669 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.678+0000 7f0c21128700 1 -- start start 2024-02-19T18:52:43.669 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.678+0000 7f0c21128700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c104730 0x7f0c1c19c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:43.669 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.678+0000 7f0c21128700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c19c900 con 0x7f0c1c104730 2024-02-19T18:52:43.669 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.679+0000 7f0c1ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c104730 0x7f0c1c19c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:43.669 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.679+0000 7f0c1ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c104730 0x7f0c1c19c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60262/0 (socket says 172.21.15.114:60262) 2024-02-19T18:52:43.669 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.679+0000 7f0c1ad9d700 1 -- 172.21.15.114:0/3857520329 learned_addr learned my addr 172.21.15.114:0/3857520329 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:43.669 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.679+0000 7f0c1ad9d700 1 -- 172.21.15.114:0/3857520329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c009550 con 0x7f0c1c104730 2024-02-19T18:52:43.669 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.679+0000 7f0c1ad9d700 1 --2- 172.21.15.114:0/3857520329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c104730 0x7f0c1c19c3c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c011670 tx=0x7f0c0c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:43.670 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.679+0000 7f0c03fff700 1 -- 172.21.15.114:0/3857520329 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c0c0039f0 con 0x7f0c1c104730 2024-02-19T18:52:43.670 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.679+0000 7f0c03fff700 1 -- 172.21.15.114:0/3857520329 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0c0c003b50 con 0x7f0c1c104730 2024-02-19T18:52:43.670 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.679+0000 7f0c21128700 1 -- 172.21.15.114:0/3857520329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c1c19cb00 con 0x7f0c1c104730 2024-02-19T18:52:43.670 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.679+0000 7f0c03fff700 1 -- 172.21.15.114:0/3857520329 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c0c0247c0 con 0x7f0c1c104730 2024-02-19T18:52:43.670 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.679+0000 7f0c21128700 1 -- 172.21.15.114:0/3857520329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c1c19cfa0 con 0x7f0c1c104730 2024-02-19T18:52:43.670 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.680+0000 7f0c03fff700 1 -- 172.21.15.114:0/3857520329 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f0c0c02d3f0 con 0x7f0c1c104730 2024-02-19T18:52:43.670 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.680+0000 7f0c21128700 1 -- 172.21.15.114:0/3857520329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c1c0623c0 con 0x7f0c1c104730 2024-02-19T18:52:43.670 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.680+0000 7f0c03fff700 1 --2- 172.21.15.114:0/3857520329 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0c04038370 0x7f0c0403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:43.670 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.680+0000 7f0c03fff700 1 -- 172.21.15.114:0/3857520329 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0c0c04c600 con 0x7f0c1c104730 2024-02-19T18:52:43.670 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.680+0000 7f0c1a59c700 1 --2- 172.21.15.114:0/3857520329 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0c04038370 0x7f0c0403a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:43.670 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.681+0000 7f0c1a59c700 1 --2- 172.21.15.114:0/3857520329 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0c04038370 0x7f0c0403a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0c10006fd0 tx=0x7f0c10006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:43.670 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.686+0000 7f0c03fff700 1 -- 172.21.15.114:0/3857520329 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0c0c010560 con 0x7f0c1c104730 2024-02-19T18:52:43.670 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:42.889+0000 7f0c21128700 1 -- 172.21.15.114:0/3857520329 --> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f0c1c0621e0 con 0x7f0c04038370 2024-02-19T18:52:43.670 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:43.111+0000 7f0c03fff700 1 -- 172.21.15.114:0/3857520329 <== mgr.14162 v2:172.21.15.114:6800/1719431365 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f0c1c0621e0 con 0x7f0c04038370 2024-02-19T18:52:43.670 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:43.112+0000 7f0c21128700 1 -- 172.21.15.114:0/3857520329 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0c04038370 msgr2=0x7f0c0403a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:43.671 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:43.112+0000 7f0c21128700 1 --2- 172.21.15.114:0/3857520329 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0c04038370 0x7f0c0403a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0c10006fd0 tx=0x7f0c10006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:43.671 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:43.112+0000 7f0c21128700 1 -- 172.21.15.114:0/3857520329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c104730 msgr2=0x7f0c1c19c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:43.671 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:43.114+0000 7f0c21128700 1 --2- 172.21.15.114:0/3857520329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c104730 0x7f0c1c19c3c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c011670 tx=0x7f0c0c011b50 comp rx=0 tx=0).stop 2024-02-19T18:52:43.671 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:43.114+0000 7f0c21128700 1 -- 172.21.15.114:0/3857520329 shutdown_connections 2024-02-19T18:52:43.671 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:43.114+0000 7f0c21128700 1 --2- 172.21.15.114:0/3857520329 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0c04038370 0x7f0c0403a830 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:43.671 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:43.114+0000 7f0c21128700 1 --2- 172.21.15.114:0/3857520329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c1c104730 0x7f0c1c19c3c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:43.671 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:43.114+0000 7f0c21128700 1 -- 172.21.15.114:0/3857520329 >> 172.21.15.114:0/3857520329 conn(0x7f0c1c100040 msgr2=0x7f0c1c101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:43.671 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:43.114+0000 7f0c21128700 1 -- 172.21.15.114:0/3857520329 shutdown_connections 2024-02-19T18:52:43.671 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:43.114+0000 7f0c21128700 1 -- 172.21.15.114:0/3857520329 wait complete. 2024-02-19T18:52:43.671 INFO:teuthology.orchestra.run.smithi114.stdout:Fetching dashboard port number... 2024-02-19T18:52:44.277 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:44 smithi114 ceph-mon[29441]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:52:44.277 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:44 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:44.824 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 8443 2024-02-19T18:52:44.824 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.065+0000 7f5089f55700 1 Processor -- start 2024-02-19T18:52:44.824 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.065+0000 7f5089f55700 1 -- start start 2024-02-19T18:52:44.824 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.065+0000 7f5089f55700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5084104d30 0x7f5084105150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:44.824 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.065+0000 7f5089f55700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5084079e20 con 0x7f5084104d30 2024-02-19T18:52:44.824 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.065+0000 7f50837fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5084104d30 0x7f5084105150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:44.824 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.065+0000 7f50837fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5084104d30 0x7f5084105150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60264/0 (socket says 172.21.15.114:60264) 2024-02-19T18:52:44.824 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.065+0000 7f50837fe700 1 -- 172.21.15.114:0/1977144166 learned_addr learned my addr 172.21.15.114:0/1977144166 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:44.824 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.066+0000 7f50837fe700 1 -- 172.21.15.114:0/1977144166 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5084079f60 con 0x7f5084104d30 2024-02-19T18:52:44.824 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.066+0000 7f50837fe700 1 --2- 172.21.15.114:0/1977144166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5084104d30 0x7f5084105150 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f50740098a0 tx=0x7f5074009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4950467d4547a8ee server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:44.824 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.066+0000 7f50827fc700 1 -- 172.21.15.114:0/1977144166 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5074004180 con 0x7f5084104d30 2024-02-19T18:52:44.825 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.066+0000 7f50827fc700 1 -- 172.21.15.114:0/1977144166 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f50740042e0 con 0x7f5084104d30 2024-02-19T18:52:44.825 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.066+0000 7f50827fc700 1 -- 172.21.15.114:0/1977144166 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5074011af0 con 0x7f5084104d30 2024-02-19T18:52:44.825 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.067+0000 7f5089f55700 1 -- 172.21.15.114:0/1977144166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5084104d30 msgr2=0x7f5084105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:44.825 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.067+0000 7f5089f55700 1 --2- 172.21.15.114:0/1977144166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5084104d30 0x7f5084105150 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f50740098a0 tx=0x7f5074009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:44.825 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.067+0000 7f5089f55700 1 -- 172.21.15.114:0/1977144166 shutdown_connections 2024-02-19T18:52:44.825 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.067+0000 7f5089f55700 1 --2- 172.21.15.114:0/1977144166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5084104d30 0x7f5084105150 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:44.825 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.067+0000 7f5089f55700 1 -- 172.21.15.114:0/1977144166 >> 172.21.15.114:0/1977144166 conn(0x7f50841008b0 msgr2=0x7f5084102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:44.825 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.067+0000 7f5089f55700 1 -- 172.21.15.114:0/1977144166 shutdown_connections 2024-02-19T18:52:44.825 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.067+0000 7f5089f55700 1 -- 172.21.15.114:0/1977144166 wait complete. 2024-02-19T18:52:44.825 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.067+0000 7f5089f55700 1 Processor -- start 2024-02-19T18:52:44.825 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.067+0000 7f5089f55700 1 -- start start 2024-02-19T18:52:44.825 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.067+0000 7f5089f55700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5084104d30 0x7f508419c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:44.825 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.067+0000 7f5089f55700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508419c8b0 con 0x7f5084104d30 2024-02-19T18:52:44.825 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.068+0000 7f50837fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5084104d30 0x7f508419c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:44.825 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.068+0000 7f50837fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5084104d30 0x7f508419c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60274/0 (socket says 172.21.15.114:60274) 2024-02-19T18:52:44.826 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.068+0000 7f50837fe700 1 -- 172.21.15.114:0/2812439709 learned_addr learned my addr 172.21.15.114:0/2812439709 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:44.826 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.068+0000 7f50837fe700 1 -- 172.21.15.114:0/2812439709 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5074009550 con 0x7f5084104d30 2024-02-19T18:52:44.826 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.068+0000 7f50837fe700 1 --2- 172.21.15.114:0/2812439709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5084104d30 0x7f508419c370 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5074011670 tx=0x7f5074011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:44.826 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.068+0000 7f5080ff9700 1 -- 172.21.15.114:0/2812439709 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5074003a40 con 0x7f5084104d30 2024-02-19T18:52:44.826 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.068+0000 7f5089f55700 1 -- 172.21.15.114:0/2812439709 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f508419cab0 con 0x7f5084104d30 2024-02-19T18:52:44.826 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.068+0000 7f5080ff9700 1 -- 172.21.15.114:0/2812439709 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5074003ba0 con 0x7f5084104d30 2024-02-19T18:52:44.826 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.068+0000 7f5080ff9700 1 -- 172.21.15.114:0/2812439709 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5074024820 con 0x7f5084104d30 2024-02-19T18:52:44.826 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.068+0000 7f5089f55700 1 -- 172.21.15.114:0/2812439709 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f508419cf50 con 0x7f5084104d30 2024-02-19T18:52:44.826 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.069+0000 7f5080ff9700 1 -- 172.21.15.114:0/2812439709 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f5074024ae0 con 0x7f5084104d30 2024-02-19T18:52:44.826 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.069+0000 7f5080ff9700 1 --2- 172.21.15.114:0/2812439709 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f506c038330 0x7f506c03a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:44.826 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.069+0000 7f5080ff9700 1 -- 172.21.15.114:0/2812439709 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f507404d570 con 0x7f5084104d30 2024-02-19T18:52:44.826 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.069+0000 7f5082ffd700 1 --2- 172.21.15.114:0/2812439709 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f506c038330 0x7f506c03a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:44.826 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.069+0000 7f5089f55700 1 -- 172.21.15.114:0/2812439709 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5084195c20 con 0x7f5084104d30 2024-02-19T18:52:44.826 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.070+0000 7f5082ffd700 1 --2- 172.21.15.114:0/2812439709 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f506c038330 0x7f506c03a7f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5078006fd0 tx=0x7f5078006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:44.826 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.074+0000 7f5080ff9700 1 -- 172.21.15.114:0/2812439709 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5074024de0 con 0x7f5084104d30 2024-02-19T18:52:44.826 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.260+0000 7f5089f55700 1 -- 172.21.15.114:0/2812439709 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f508402cef0 con 0x7f5084104d30 2024-02-19T18:52:44.827 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.261+0000 7f5080ff9700 1 -- 172.21.15.114:0/2812439709 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f507404a030 con 0x7f5084104d30 2024-02-19T18:52:44.827 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.263+0000 7f5089f55700 1 -- 172.21.15.114:0/2812439709 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f506c038330 msgr2=0x7f506c03a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:44.827 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.263+0000 7f5089f55700 1 --2- 172.21.15.114:0/2812439709 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f506c038330 0x7f506c03a7f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5078006fd0 tx=0x7f5078006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:44.827 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.263+0000 7f5089f55700 1 -- 172.21.15.114:0/2812439709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5084104d30 msgr2=0x7f508419c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:44.827 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.263+0000 7f5089f55700 1 --2- 172.21.15.114:0/2812439709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5084104d30 0x7f508419c370 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5074011670 tx=0x7f5074011aa0 comp rx=0 tx=0).stop 2024-02-19T18:52:44.827 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.263+0000 7f5089f55700 1 -- 172.21.15.114:0/2812439709 shutdown_connections 2024-02-19T18:52:44.827 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.263+0000 7f5089f55700 1 --2- 172.21.15.114:0/2812439709 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f506c038330 0x7f506c03a7f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:44.827 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.263+0000 7f5089f55700 1 --2- 172.21.15.114:0/2812439709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5084104d30 0x7f508419c370 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:44.827 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.263+0000 7f5089f55700 1 -- 172.21.15.114:0/2812439709 >> 172.21.15.114:0/2812439709 conn(0x7f50841008b0 msgr2=0x7f50841908f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:44.827 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.263+0000 7f5089f55700 1 -- 172.21.15.114:0/2812439709 shutdown_connections 2024-02-19T18:52:44.827 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:44.263+0000 7f5089f55700 1 -- 172.21.15.114:0/2812439709 wait complete. 2024-02-19T18:52:44.830 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-19T18:52:44.830 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-02-19T18:52:44.837 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-19T18:52:44.837 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-02-19T18:52:44.837 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-02-19T18:52:44.838 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-19T18:52:44.839 INFO:teuthology.orchestra.run.smithi114.stdout:Ceph Dashboard is now available at: 2024-02-19T18:52:44.839 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:52:44.839 INFO:teuthology.orchestra.run.smithi114.stdout: URL: https://smithi114.front.sepia.ceph.com:8443/ 2024-02-19T18:52:44.839 INFO:teuthology.orchestra.run.smithi114.stdout: User: admin 2024-02-19T18:52:44.839 INFO:teuthology.orchestra.run.smithi114.stdout: Password: f12sgmunb2 2024-02-19T18:52:44.839 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:52:44.839 INFO:teuthology.orchestra.run.smithi114.stdout:Saving cluster configuration to /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config directory 2024-02-19T18:52:44.840 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling autotune for osd_memory_target 2024-02-19T18:52:45.236 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:45 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/2812439709' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-19T18:52:46.055 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.237+0000 7fcdd36cf700 1 Processor -- start 2024-02-19T18:52:46.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.237+0000 7fcdd36cf700 1 -- start start 2024-02-19T18:52:46.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.238+0000 7fcdd36cf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdcc106ea0 0x7fcdcc109290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:46.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.238+0000 7fcdd36cf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc079ec0 con 0x7fcdcc106ea0 2024-02-19T18:52:46.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.238+0000 7fcdd146b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdcc106ea0 0x7fcdcc109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:46.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.238+0000 7fcdd146b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdcc106ea0 0x7fcdcc109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60284/0 (socket says 172.21.15.114:60284) 2024-02-19T18:52:46.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.238+0000 7fcdd146b700 1 -- 172.21.15.114:0/2491671598 learned_addr learned my addr 172.21.15.114:0/2491671598 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:46.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.238+0000 7fcdd146b700 1 -- 172.21.15.114:0/2491671598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdcc1097d0 con 0x7fcdcc106ea0 2024-02-19T18:52:46.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.238+0000 7fcdd146b700 1 --2- 172.21.15.114:0/2491671598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdcc106ea0 0x7fcdcc109290 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc0098a0 tx=0x7fcdbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78c19e5dfc34a03 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:46.056 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.238+0000 7fcdc3fff700 1 -- 172.21.15.114:0/2491671598 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdbc004180 con 0x7fcdcc106ea0 2024-02-19T18:52:46.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.238+0000 7fcdc3fff700 1 -- 172.21.15.114:0/2491671598 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcdbc0042e0 con 0x7fcdcc106ea0 2024-02-19T18:52:46.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.238+0000 7fcdc3fff700 1 -- 172.21.15.114:0/2491671598 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdbc011af0 con 0x7fcdcc106ea0 2024-02-19T18:52:46.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.239+0000 7fcdd36cf700 1 -- 172.21.15.114:0/2491671598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdcc106ea0 msgr2=0x7fcdcc109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:46.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.239+0000 7fcdd36cf700 1 --2- 172.21.15.114:0/2491671598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdcc106ea0 0x7fcdcc109290 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc0098a0 tx=0x7fcdbc009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:46.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.239+0000 7fcdd36cf700 1 -- 172.21.15.114:0/2491671598 shutdown_connections 2024-02-19T18:52:46.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.239+0000 7fcdd36cf700 1 --2- 172.21.15.114:0/2491671598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdcc106ea0 0x7fcdcc109290 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:46.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.239+0000 7fcdd36cf700 1 -- 172.21.15.114:0/2491671598 >> 172.21.15.114:0/2491671598 conn(0x7fcdcc1008b0 msgr2=0x7fcdcc102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:46.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.239+0000 7fcdd36cf700 1 -- 172.21.15.114:0/2491671598 shutdown_connections 2024-02-19T18:52:46.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.239+0000 7fcdd36cf700 1 -- 172.21.15.114:0/2491671598 wait complete. 2024-02-19T18:52:46.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.240+0000 7fcdd36cf700 1 Processor -- start 2024-02-19T18:52:46.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.240+0000 7fcdd36cf700 1 -- start start 2024-02-19T18:52:46.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.240+0000 7fcdd36cf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdcc106ea0 0x7fcdcc19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:46.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.240+0000 7fcdd36cf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc19a6b0 con 0x7fcdcc106ea0 2024-02-19T18:52:46.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.240+0000 7fcdd146b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdcc106ea0 0x7fcdcc19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:46.057 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.240+0000 7fcdd146b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdcc106ea0 0x7fcdcc19a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60296/0 (socket says 172.21.15.114:60296) 2024-02-19T18:52:46.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.240+0000 7fcdd146b700 1 -- 172.21.15.114:0/739136703 learned_addr learned my addr 172.21.15.114:0/739136703 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:46.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.240+0000 7fcdd146b700 1 -- 172.21.15.114:0/739136703 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdbc009550 con 0x7fcdcc106ea0 2024-02-19T18:52:46.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.240+0000 7fcdd146b700 1 --2- 172.21.15.114:0/739136703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdcc106ea0 0x7fcdcc19a170 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc011670 tx=0x7fcdbc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:46.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.240+0000 7fcdc27fc700 1 -- 172.21.15.114:0/739136703 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdbc003a40 con 0x7fcdcc106ea0 2024-02-19T18:52:46.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.240+0000 7fcdc27fc700 1 -- 172.21.15.114:0/739136703 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcdbc003ba0 con 0x7fcdcc106ea0 2024-02-19T18:52:46.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.240+0000 7fcdd36cf700 1 -- 172.21.15.114:0/739136703 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdcc19a8b0 con 0x7fcdcc106ea0 2024-02-19T18:52:46.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.240+0000 7fcdc27fc700 1 -- 172.21.15.114:0/739136703 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdbc024820 con 0x7fcdcc106ea0 2024-02-19T18:52:46.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.241+0000 7fcdd36cf700 1 -- 172.21.15.114:0/739136703 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdcc19ad50 con 0x7fcdcc106ea0 2024-02-19T18:52:46.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.241+0000 7fcdc27fc700 1 -- 172.21.15.114:0/739136703 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fcdbc01ac70 con 0x7fcdcc106ea0 2024-02-19T18:52:46.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.241+0000 7fcdd36cf700 1 -- 172.21.15.114:0/739136703 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdcc10a5c0 con 0x7fcdcc106ea0 2024-02-19T18:52:46.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.242+0000 7fcdc27fc700 1 --2- 172.21.15.114:0/739136703 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fcdb8038420 0x7fcdb803a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:46.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.242+0000 7fcdc27fc700 1 -- 172.21.15.114:0/739136703 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcdbc04c4b0 con 0x7fcdcc106ea0 2024-02-19T18:52:46.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.242+0000 7fcdd0c6a700 1 --2- 172.21.15.114:0/739136703 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fcdb8038420 0x7fcdb803a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:46.058 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.242+0000 7fcdd0c6a700 1 --2- 172.21.15.114:0/739136703 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fcdb8038420 0x7fcdb803a8e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8006fd0 tx=0x7fcdc8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:46.059 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.246+0000 7fcdc27fc700 1 -- 172.21.15.114:0/739136703 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fcdbc018760 con 0x7fcdcc106ea0 2024-02-19T18:52:46.059 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.434+0000 7fcdd36cf700 1 -- 172.21.15.114:0/739136703 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fcdcc19db30 con 0x7fcdcc106ea0 2024-02-19T18:52:46.059 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.435+0000 7fcdc27fc700 1 -- 172.21.15.114:0/739136703 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fcdbc0172b0 con 0x7fcdcc106ea0 2024-02-19T18:52:46.059 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.437+0000 7fcdd36cf700 1 -- 172.21.15.114:0/739136703 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fcdb8038420 msgr2=0x7fcdb803a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:46.059 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.437+0000 7fcdd36cf700 1 --2- 172.21.15.114:0/739136703 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fcdb8038420 0x7fcdb803a8e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8006fd0 tx=0x7fcdc8006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:46.059 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.437+0000 7fcdd36cf700 1 -- 172.21.15.114:0/739136703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdcc106ea0 msgr2=0x7fcdcc19a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:46.059 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.437+0000 7fcdd36cf700 1 --2- 172.21.15.114:0/739136703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdcc106ea0 0x7fcdcc19a170 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc011670 tx=0x7fcdbc011aa0 comp rx=0 tx=0).stop 2024-02-19T18:52:46.059 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.437+0000 7fcdd36cf700 1 -- 172.21.15.114:0/739136703 shutdown_connections 2024-02-19T18:52:46.059 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.437+0000 7fcdd36cf700 1 --2- 172.21.15.114:0/739136703 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fcdb8038420 0x7fcdb803a8e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:46.059 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.437+0000 7fcdd36cf700 1 --2- 172.21.15.114:0/739136703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdcc106ea0 0x7fcdcc19a170 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:46.059 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.437+0000 7fcdd36cf700 1 -- 172.21.15.114:0/739136703 >> 172.21.15.114:0/739136703 conn(0x7fcdcc1008b0 msgr2=0x7fcdcc101f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:46.059 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.437+0000 7fcdd36cf700 1 -- 172.21.15.114:0/739136703 shutdown_connections 2024-02-19T18:52:46.059 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:45.437+0000 7fcdd36cf700 1 -- 172.21.15.114:0/739136703 wait complete. 2024-02-19T18:52:46.284 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:46 smithi114 ceph-mon[29441]: mgrmap e13: smithi114.pcvtrk(active, since 7s) 2024-02-19T18:52:47.362 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.469+0000 7f9579c1d700 1 Processor -- start 2024-02-19T18:52:47.362 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.469+0000 7f9579c1d700 1 -- start start 2024-02-19T18:52:47.362 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.470+0000 7f9579c1d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9574106f10 0x7f9574109300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:47.362 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.470+0000 7f9579c1d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9574074620 con 0x7f9574106f10 2024-02-19T18:52:47.362 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.470+0000 7f95737fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9574106f10 0x7f9574109300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:47.362 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.470+0000 7f95737fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9574106f10 0x7f9574109300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44146/0 (socket says 172.21.15.114:44146) 2024-02-19T18:52:47.362 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.470+0000 7f95737fe700 1 -- 172.21.15.114:0/4146301324 learned_addr learned my addr 172.21.15.114:0/4146301324 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:47.363 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.470+0000 7f95737fe700 1 -- 172.21.15.114:0/4146301324 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9574109840 con 0x7f9574106f10 2024-02-19T18:52:47.363 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.470+0000 7f95737fe700 1 --2- 172.21.15.114:0/4146301324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9574106f10 0x7f9574109300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f95640098a0 tx=0x7f9564009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c203fd082264bc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:47.363 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.470+0000 7f9572ffd700 1 -- 172.21.15.114:0/4146301324 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9564004180 con 0x7f9574106f10 2024-02-19T18:52:47.363 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.470+0000 7f9572ffd700 1 -- 172.21.15.114:0/4146301324 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f95640042e0 con 0x7f9574106f10 2024-02-19T18:52:47.363 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.470+0000 7f9572ffd700 1 -- 172.21.15.114:0/4146301324 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9564011af0 con 0x7f9574106f10 2024-02-19T18:52:47.363 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.471+0000 7f9579c1d700 1 -- 172.21.15.114:0/4146301324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9574106f10 msgr2=0x7f9574109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:47.363 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.471+0000 7f9579c1d700 1 --2- 172.21.15.114:0/4146301324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9574106f10 0x7f9574109300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f95640098a0 tx=0x7f9564009bb0 comp rx=0 tx=0).stop 2024-02-19T18:52:47.363 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.471+0000 7f9579c1d700 1 -- 172.21.15.114:0/4146301324 shutdown_connections 2024-02-19T18:52:47.363 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.471+0000 7f9579c1d700 1 --2- 172.21.15.114:0/4146301324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9574106f10 0x7f9574109300 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:47.363 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.471+0000 7f9579c1d700 1 -- 172.21.15.114:0/4146301324 >> 172.21.15.114:0/4146301324 conn(0x7f95741008c0 msgr2=0x7f9574102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:47.363 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.471+0000 7f9579c1d700 1 -- 172.21.15.114:0/4146301324 shutdown_connections 2024-02-19T18:52:47.364 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.471+0000 7f9579c1d700 1 -- 172.21.15.114:0/4146301324 wait complete. 2024-02-19T18:52:47.364 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.471+0000 7f9579c1d700 1 Processor -- start 2024-02-19T18:52:47.364 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.471+0000 7f9579c1d700 1 -- start start 2024-02-19T18:52:47.364 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.472+0000 7f9579c1d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9574106f10 0x7f957419a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:47.364 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.472+0000 7f9579c1d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957419a6c0 con 0x7f9574106f10 2024-02-19T18:52:47.364 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.472+0000 7f95737fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9574106f10 0x7f957419a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:47.364 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.472+0000 7f95737fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9574106f10 0x7f957419a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44152/0 (socket says 172.21.15.114:44152) 2024-02-19T18:52:47.364 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.472+0000 7f95737fe700 1 -- 172.21.15.114:0/1179989877 learned_addr learned my addr 172.21.15.114:0/1179989877 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:47.364 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.472+0000 7f95737fe700 1 -- 172.21.15.114:0/1179989877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9564009550 con 0x7f9574106f10 2024-02-19T18:52:47.364 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.472+0000 7f95737fe700 1 --2- 172.21.15.114:0/1179989877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9574106f10 0x7f957419a180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9564011670 tx=0x7f9564011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:47.364 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.472+0000 7f95717fa700 1 -- 172.21.15.114:0/1179989877 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9564003a40 con 0x7f9574106f10 2024-02-19T18:52:47.364 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.472+0000 7f95717fa700 1 -- 172.21.15.114:0/1179989877 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9564003ba0 con 0x7f9574106f10 2024-02-19T18:52:47.365 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.472+0000 7f95717fa700 1 -- 172.21.15.114:0/1179989877 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9564024820 con 0x7f9574106f10 2024-02-19T18:52:47.365 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.472+0000 7f9579c1d700 1 -- 172.21.15.114:0/1179989877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f957419a8c0 con 0x7f9574106f10 2024-02-19T18:52:47.365 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.472+0000 7f9579c1d700 1 -- 172.21.15.114:0/1179989877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f957419ad60 con 0x7f9574106f10 2024-02-19T18:52:47.365 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.473+0000 7f95717fa700 1 -- 172.21.15.114:0/1179989877 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f9564024ae0 con 0x7f9574106f10 2024-02-19T18:52:47.365 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.473+0000 7f9579c1d700 1 -- 172.21.15.114:0/1179989877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f957410a5d0 con 0x7f9574106f10 2024-02-19T18:52:47.365 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.473+0000 7f95717fa700 1 --2- 172.21.15.114:0/1179989877 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f95600383d0 0x7f956003a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:47.365 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.473+0000 7f95717fa700 1 -- 172.21.15.114:0/1179989877 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f956404c560 con 0x7f9574106f10 2024-02-19T18:52:47.365 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.474+0000 7f956affd700 1 --2- 172.21.15.114:0/1179989877 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f95600383d0 0x7f956003a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:47.365 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.474+0000 7f956affd700 1 --2- 172.21.15.114:0/1179989877 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f95600383d0 0x7f956003a890 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f955c006fd0 tx=0x7f955c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:47.365 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.478+0000 7f95717fa700 1 -- 172.21.15.114:0/1179989877 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9564022310 con 0x7f9574106f10 2024-02-19T18:52:47.365 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.772+0000 7f9579c1d700 1 -- 172.21.15.114:0/1179989877 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f957410a7f0 con 0x7f9574106f10 2024-02-19T18:52:47.365 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.775+0000 7f95717fa700 1 -- 172.21.15.114:0/1179989877 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7f9564018b40 con 0x7f9574106f10 2024-02-19T18:52:47.365 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.777+0000 7f9579c1d700 1 -- 172.21.15.114:0/1179989877 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f95600383d0 msgr2=0x7f956003a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:47.366 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.777+0000 7f9579c1d700 1 --2- 172.21.15.114:0/1179989877 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f95600383d0 0x7f956003a890 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f955c006fd0 tx=0x7f955c006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:47.366 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.777+0000 7f9579c1d700 1 -- 172.21.15.114:0/1179989877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9574106f10 msgr2=0x7f957419a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:47.366 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.777+0000 7f9579c1d700 1 --2- 172.21.15.114:0/1179989877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9574106f10 0x7f957419a180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9564011670 tx=0x7f9564011aa0 comp rx=0 tx=0).stop 2024-02-19T18:52:47.366 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.777+0000 7f9579c1d700 1 -- 172.21.15.114:0/1179989877 shutdown_connections 2024-02-19T18:52:47.366 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.777+0000 7f9579c1d700 1 --2- 172.21.15.114:0/1179989877 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f95600383d0 0x7f956003a890 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:47.366 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.777+0000 7f9579c1d700 1 --2- 172.21.15.114:0/1179989877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9574106f10 0x7f957419a180 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:47.366 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.777+0000 7f9579c1d700 1 -- 172.21.15.114:0/1179989877 >> 172.21.15.114:0/1179989877 conn(0x7f95741008c0 msgr2=0x7f9574101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:47.366 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.777+0000 7f9579c1d700 1 -- 172.21.15.114:0/1179989877 shutdown_connections 2024-02-19T18:52:47.366 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-02-19T18:52:46.777+0000 7f9579c1d700 1 -- 172.21.15.114:0/1179989877 wait complete. 2024-02-19T18:52:47.366 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-19T18:52:47.366 INFO:teuthology.orchestra.run.smithi114.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-19T18:52:47.366 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:52:47.366 INFO:teuthology.orchestra.run.smithi114.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-19T18:52:47.366 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:52:47.366 INFO:teuthology.orchestra.run.smithi114.stdout:Or, if you are only running a single cluster on this host: 2024-02-19T18:52:47.367 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:52:47.367 INFO:teuthology.orchestra.run.smithi114.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-19T18:52:47.367 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:52:47.367 INFO:teuthology.orchestra.run.smithi114.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-19T18:52:47.367 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:52:47.367 INFO:teuthology.orchestra.run.smithi114.stdout: ceph telemetry on 2024-02-19T18:52:47.367 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:52:47.367 INFO:teuthology.orchestra.run.smithi114.stdout:For more information see: 2024-02-19T18:52:47.367 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:52:47.367 INFO:teuthology.orchestra.run.smithi114.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-02-19T18:52:47.367 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:52:47.367 INFO:teuthology.orchestra.run.smithi114.stdout:Bootstrap complete. 2024-02-19T18:52:47.392 INFO:tasks.cephadm:Fetching config... 2024-02-19T18:52:47.392 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T18:52:47.392 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-19T18:52:47.447 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-19T18:52:47.447 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T18:52:47.447 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-19T18:52:47.503 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-19T18:52:47.503 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T18:52:47.503 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/keyring of=/dev/stdout 2024-02-19T18:52:47.570 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-19T18:52:47.570 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T18:52:47.570 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-19T18:52:47.624 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-19T18:52:47.625 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDeQI3jKEuC8R3rX7JE6I4AV3TXRjUyihASRChAo/bW1RWDnRWtdfxwxc72Frdm/ks8LyTIfxCDeCsXB/sAPx0mwVGNvTLvAzO/dpJDJ9XOJad7F4Z24xHk6DGBST+u5j9mS7LBY1TVzFX9BLKKBJ4Ewx/qEcZ/WZj2BXKTw1juu3fXuLreJPJ42TeZRMV3JGqZImBoTunDybwjMDcH8iCzSUfd7pVdG30Whz3XjuTD8/LB5d2klUQ37rW0bGwqqClnNsY8x2xDiyOJJrbFmLixtBO4XTB1gofAnwJ9JyzXKPrT9FN6ov0YsloKQR3rOa4QPhpDVigZUERMInhH/xNSwN8tTvfqD7WOmO3seZHF+CIuWqmk6/XDozblVaBAWSfBfd04yszEPCf2t5RL54mdRaQIsos+2+Twe+GpMTgey4rkCPf4QsMeGqiUHV9EQv9tJZFeY0h0h5ua/U+72GVRfyd2he2atyQoUCzpK0ryIPdBdNBSCgdZGAN8O8DBKOs= ceph-be02a9a6-cf57-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T18:52:47.703 INFO:teuthology.orchestra.run.smithi114.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDeQI3jKEuC8R3rX7JE6I4AV3TXRjUyihASRChAo/bW1RWDnRWtdfxwxc72Frdm/ks8LyTIfxCDeCsXB/sAPx0mwVGNvTLvAzO/dpJDJ9XOJad7F4Z24xHk6DGBST+u5j9mS7LBY1TVzFX9BLKKBJ4Ewx/qEcZ/WZj2BXKTw1juu3fXuLreJPJ42TeZRMV3JGqZImBoTunDybwjMDcH8iCzSUfd7pVdG30Whz3XjuTD8/LB5d2klUQ37rW0bGwqqClnNsY8x2xDiyOJJrbFmLixtBO4XTB1gofAnwJ9JyzXKPrT9FN6ov0YsloKQR3rOa4QPhpDVigZUERMInhH/xNSwN8tTvfqD7WOmO3seZHF+CIuWqmk6/XDozblVaBAWSfBfd04yszEPCf2t5RL54mdRaQIsos+2+Twe+GpMTgey4rkCPf4QsMeGqiUHV9EQv9tJZFeY0h0h5ua/U+72GVRfyd2he2atyQoUCzpK0ryIPdBdNBSCgdZGAN8O8DBKOs= ceph-be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T18:52:47.717 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDeQI3jKEuC8R3rX7JE6I4AV3TXRjUyihASRChAo/bW1RWDnRWtdfxwxc72Frdm/ks8LyTIfxCDeCsXB/sAPx0mwVGNvTLvAzO/dpJDJ9XOJad7F4Z24xHk6DGBST+u5j9mS7LBY1TVzFX9BLKKBJ4Ewx/qEcZ/WZj2BXKTw1juu3fXuLreJPJ42TeZRMV3JGqZImBoTunDybwjMDcH8iCzSUfd7pVdG30Whz3XjuTD8/LB5d2klUQ37rW0bGwqqClnNsY8x2xDiyOJJrbFmLixtBO4XTB1gofAnwJ9JyzXKPrT9FN6ov0YsloKQR3rOa4QPhpDVigZUERMInhH/xNSwN8tTvfqD7WOmO3seZHF+CIuWqmk6/XDozblVaBAWSfBfd04yszEPCf2t5RL54mdRaQIsos+2+Twe+GpMTgey4rkCPf4QsMeGqiUHV9EQv9tJZFeY0h0h5ua/U+72GVRfyd2he2atyQoUCzpK0ryIPdBdNBSCgdZGAN8O8DBKOs= ceph-be02a9a6-cf57-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T18:52:47.765 INFO:teuthology.orchestra.run.smithi174.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDeQI3jKEuC8R3rX7JE6I4AV3TXRjUyihASRChAo/bW1RWDnRWtdfxwxc72Frdm/ks8LyTIfxCDeCsXB/sAPx0mwVGNvTLvAzO/dpJDJ9XOJad7F4Z24xHk6DGBST+u5j9mS7LBY1TVzFX9BLKKBJ4Ewx/qEcZ/WZj2BXKTw1juu3fXuLreJPJ42TeZRMV3JGqZImBoTunDybwjMDcH8iCzSUfd7pVdG30Whz3XjuTD8/LB5d2klUQ37rW0bGwqqClnNsY8x2xDiyOJJrbFmLixtBO4XTB1gofAnwJ9JyzXKPrT9FN6ov0YsloKQR3rOa4QPhpDVigZUERMInhH/xNSwN8tTvfqD7WOmO3seZHF+CIuWqmk6/XDozblVaBAWSfBfd04yszEPCf2t5RL54mdRaQIsos+2+Twe+GpMTgey4rkCPf4QsMeGqiUHV9EQv9tJZFeY0h0h5ua/U+72GVRfyd2he2atyQoUCzpK0ryIPdBdNBSCgdZGAN8O8DBKOs= ceph-be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T18:52:47.780 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-19T18:52:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:52:48.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:47 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1179989877' entity='client.admin' 2024-02-19T18:52:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.829+0000 7f0e3aba3700 1 -- 172.21.15.114:0/2982053546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e340ff7a0 msgr2=0x7f0e340ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.829+0000 7f0e3aba3700 1 --2- 172.21.15.114:0/2982053546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e340ff7a0 0x7f0e340ffbc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009910 tx=0x7f0e28009c20 comp rx=0 tx=0).stop 2024-02-19T18:52:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.830+0000 7f0e3aba3700 1 -- 172.21.15.114:0/2982053546 shutdown_connections 2024-02-19T18:52:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.830+0000 7f0e3aba3700 1 --2- 172.21.15.114:0/2982053546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e340ff7a0 0x7f0e340ffbc0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.830+0000 7f0e3aba3700 1 -- 172.21.15.114:0/2982053546 >> 172.21.15.114:0/2982053546 conn(0x7f0e340fad20 msgr2=0x7f0e340fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.830+0000 7f0e3aba3700 1 -- 172.21.15.114:0/2982053546 shutdown_connections 2024-02-19T18:52:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.830+0000 7f0e3aba3700 1 -- 172.21.15.114:0/2982053546 wait complete. 2024-02-19T18:52:49.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.830+0000 7f0e3aba3700 1 Processor -- start 2024-02-19T18:52:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.830+0000 7f0e3aba3700 1 -- start start 2024-02-19T18:52:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.830+0000 7f0e3aba3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e340ff7a0 0x7f0e340fcce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.830+0000 7f0e3aba3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e340fd220 con 0x7f0e340ff7a0 2024-02-19T18:52:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.830+0000 7f0e3893f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e340ff7a0 0x7f0e340fcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.830+0000 7f0e3893f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e340ff7a0 0x7f0e340fcce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44162/0 (socket says 172.21.15.114:44162) 2024-02-19T18:52:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.830+0000 7f0e3893f700 1 -- 172.21.15.114:0/202059972 learned_addr learned my addr 172.21.15.114:0/202059972 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.831+0000 7f0e3893f700 1 -- 172.21.15.114:0/202059972 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e280095f0 con 0x7f0e340ff7a0 2024-02-19T18:52:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.831+0000 7f0e3893f700 1 --2- 172.21.15.114:0/202059972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e340ff7a0 0x7f0e340fcce0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0e28005b40 tx=0x7f0e28004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.831+0000 7f0e31ffb700 1 -- 172.21.15.114:0/202059972 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e2801d070 con 0x7f0e340ff7a0 2024-02-19T18:52:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.831+0000 7f0e31ffb700 1 -- 172.21.15.114:0/202059972 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0e2800be10 con 0x7f0e340ff7a0 2024-02-19T18:52:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.831+0000 7f0e3aba3700 1 -- 172.21.15.114:0/202059972 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e340fb330 con 0x7f0e340ff7a0 2024-02-19T18:52:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.831+0000 7f0e31ffb700 1 -- 172.21.15.114:0/202059972 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e2800f460 con 0x7f0e340ff7a0 2024-02-19T18:52:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.831+0000 7f0e3aba3700 1 -- 172.21.15.114:0/202059972 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e340fb7d0 con 0x7f0e340ff7a0 2024-02-19T18:52:49.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.832+0000 7f0e31ffb700 1 -- 172.21.15.114:0/202059972 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f0e2800f6d0 con 0x7f0e340ff7a0 2024-02-19T18:52:49.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.832+0000 7f0e31ffb700 1 --2- 172.21.15.114:0/202059972 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0e240383c0 0x7f0e2403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:49.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.832+0000 7f0e3aba3700 1 -- 172.21.15.114:0/202059972 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e340623c0 con 0x7f0e340ff7a0 2024-02-19T18:52:49.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.832+0000 7f0e31ffb700 1 -- 172.21.15.114:0/202059972 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0e2804c470 con 0x7f0e340ff7a0 2024-02-19T18:52:49.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.832+0000 7f0e33fff700 1 --2- 172.21.15.114:0/202059972 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0e240383c0 0x7f0e2403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:49.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.833+0000 7f0e33fff700 1 --2- 172.21.15.114:0/202059972 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0e240383c0 0x7f0e2403a880 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0e20006fd0 tx=0x7f0e20006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:49.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:49.837+0000 7f0e31ffb700 1 -- 172.21.15.114:0/202059972 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0e28027030 con 0x7f0e340ff7a0 2024-02-19T18:52:50.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:50.028+0000 7f0e3aba3700 1 -- 172.21.15.114:0/202059972 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f0e3418f200 con 0x7f0e340ff7a0 2024-02-19T18:52:50.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:50.031+0000 7f0e31ffb700 1 -- 172.21.15.114:0/202059972 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0e2802c440 con 0x7f0e340ff7a0 2024-02-19T18:52:50.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:50.033+0000 7f0e3aba3700 1 -- 172.21.15.114:0/202059972 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0e240383c0 msgr2=0x7f0e2403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:50.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:50.033+0000 7f0e3aba3700 1 --2- 172.21.15.114:0/202059972 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0e240383c0 0x7f0e2403a880 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0e20006fd0 tx=0x7f0e20006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:50.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:50.033+0000 7f0e3aba3700 1 -- 172.21.15.114:0/202059972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e340ff7a0 msgr2=0x7f0e340fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:50.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:50.033+0000 7f0e3aba3700 1 --2- 172.21.15.114:0/202059972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e340ff7a0 0x7f0e340fcce0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0e28005b40 tx=0x7f0e28004740 comp rx=0 tx=0).stop 2024-02-19T18:52:50.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:50.034+0000 7f0e3aba3700 1 -- 172.21.15.114:0/202059972 shutdown_connections 2024-02-19T18:52:50.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:50.034+0000 7f0e3aba3700 1 --2- 172.21.15.114:0/202059972 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0e240383c0 0x7f0e2403a880 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:50.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:50.034+0000 7f0e3aba3700 1 --2- 172.21.15.114:0/202059972 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e340ff7a0 0x7f0e340fcce0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:50.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:50.034+0000 7f0e3aba3700 1 -- 172.21.15.114:0/202059972 >> 172.21.15.114:0/202059972 conn(0x7f0e340fad20 msgr2=0x7f0e34069100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:50.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:50.034+0000 7f0e3aba3700 1 -- 172.21.15.114:0/202059972 shutdown_connections 2024-02-19T18:52:50.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:50.034+0000 7f0e3aba3700 1 -- 172.21.15.114:0/202059972 wait complete. 2024-02-19T18:52:50.779 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-19T18:52:50.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-19T18:52:51.137 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:51 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/202059972' entity='client.admin' 2024-02-19T18:52:51.256 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:52:53.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:53 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:53.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:53 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:53.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:53 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:52:53.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:53 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:53.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:53 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi114", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T18:52:53.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:53 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi114", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-02-19T18:52:53.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:53 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:52:53.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.854+0000 7f76c26f0700 1 -- 172.21.15.114:0/943629892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76bc101ea0 msgr2=0x7f76bc1022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:53.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.854+0000 7f76c26f0700 1 --2- 172.21.15.114:0/943629892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76bc101ea0 0x7f76bc1022c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f76ac009910 tx=0x7f76ac009c20 comp rx=0 tx=0).stop 2024-02-19T18:52:53.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.854+0000 7f76c26f0700 1 -- 172.21.15.114:0/943629892 shutdown_connections 2024-02-19T18:52:53.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.854+0000 7f76c26f0700 1 --2- 172.21.15.114:0/943629892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76bc101ea0 0x7f76bc1022c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:53.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.854+0000 7f76c26f0700 1 -- 172.21.15.114:0/943629892 >> 172.21.15.114:0/943629892 conn(0x7f76bc0fa4a0 msgr2=0x7f76bc0fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:53.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.854+0000 7f76c26f0700 1 -- 172.21.15.114:0/943629892 shutdown_connections 2024-02-19T18:52:53.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.854+0000 7f76c26f0700 1 -- 172.21.15.114:0/943629892 wait complete. 2024-02-19T18:52:53.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.854+0000 7f76c26f0700 1 Processor -- start 2024-02-19T18:52:53.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.855+0000 7f76c26f0700 1 -- start start 2024-02-19T18:52:53.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.855+0000 7f76c26f0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76bc101ea0 0x7f76bc1015a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:53.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.855+0000 7f76c26f0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc101ae0 con 0x7f76bc101ea0 2024-02-19T18:52:53.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.855+0000 7f76bbfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76bc101ea0 0x7f76bc1015a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:53.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.855+0000 7f76bbfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76bc101ea0 0x7f76bc1015a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44182/0 (socket says 172.21.15.114:44182) 2024-02-19T18:52:53.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.855+0000 7f76bbfff700 1 -- 172.21.15.114:0/667241308 learned_addr learned my addr 172.21.15.114:0/667241308 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:53.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.855+0000 7f76bbfff700 1 -- 172.21.15.114:0/667241308 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ac0095f0 con 0x7f76bc101ea0 2024-02-19T18:52:53.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.855+0000 7f76bbfff700 1 --2- 172.21.15.114:0/667241308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76bc101ea0 0x7f76bc1015a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f76ac005b40 tx=0x7f76ac004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:53.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.856+0000 7f76b97fa700 1 -- 172.21.15.114:0/667241308 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76ac01d070 con 0x7f76bc101ea0 2024-02-19T18:52:53.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.856+0000 7f76c26f0700 1 -- 172.21.15.114:0/667241308 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76bc0ffbf0 con 0x7f76bc101ea0 2024-02-19T18:52:53.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.856+0000 7f76b97fa700 1 -- 172.21.15.114:0/667241308 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f76ac00be10 con 0x7f76bc101ea0 2024-02-19T18:52:53.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.856+0000 7f76b97fa700 1 -- 172.21.15.114:0/667241308 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76ac00f460 con 0x7f76bc101ea0 2024-02-19T18:52:53.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.856+0000 7f76c26f0700 1 -- 172.21.15.114:0/667241308 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76bc100090 con 0x7f76bc101ea0 2024-02-19T18:52:53.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.856+0000 7f76b97fa700 1 -- 172.21.15.114:0/667241308 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f76ac00f6d0 con 0x7f76bc101ea0 2024-02-19T18:52:53.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.857+0000 7f76c26f0700 1 -- 172.21.15.114:0/667241308 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76bc0623c0 con 0x7f76bc101ea0 2024-02-19T18:52:53.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.857+0000 7f76b97fa700 1 --2- 172.21.15.114:0/667241308 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f76a40383c0 0x7f76a403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:53.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.857+0000 7f76b97fa700 1 -- 172.21.15.114:0/667241308 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f76ac04c510 con 0x7f76bc101ea0 2024-02-19T18:52:53.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.857+0000 7f76bb7fe700 1 --2- 172.21.15.114:0/667241308 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f76a40383c0 0x7f76a403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:53.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.857+0000 7f76bb7fe700 1 --2- 172.21.15.114:0/667241308 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f76a40383c0 0x7f76a403a880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f76b0006fd0 tx=0x7f76b0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:53.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:53.861+0000 7f76b97fa700 1 -- 172.21.15.114:0/667241308 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f76ac02a360 con 0x7f76bc101ea0 2024-02-19T18:52:54.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:54.057+0000 7f76c26f0700 1 -- 172.21.15.114:0/667241308 --> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f76bc100390 con 0x7f76a40383c0 2024-02-19T18:52:54.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:54.061+0000 7f76b97fa700 1 -- 172.21.15.114:0/667241308 <== mgr.14162 v2:172.21.15.114:6800/1719431365 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f76bc100390 con 0x7f76a40383c0 2024-02-19T18:52:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:54.063+0000 7f76c26f0700 1 -- 172.21.15.114:0/667241308 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f76a40383c0 msgr2=0x7f76a403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:54.063+0000 7f76c26f0700 1 --2- 172.21.15.114:0/667241308 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f76a40383c0 0x7f76a403a880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f76b0006fd0 tx=0x7f76b0006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:54.063+0000 7f76c26f0700 1 -- 172.21.15.114:0/667241308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76bc101ea0 msgr2=0x7f76bc1015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:54.063+0000 7f76c26f0700 1 --2- 172.21.15.114:0/667241308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76bc101ea0 0x7f76bc1015a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f76ac005b40 tx=0x7f76ac004740 comp rx=0 tx=0).stop 2024-02-19T18:52:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:54.063+0000 7f76c26f0700 1 -- 172.21.15.114:0/667241308 shutdown_connections 2024-02-19T18:52:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:54.063+0000 7f76c26f0700 1 --2- 172.21.15.114:0/667241308 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f76a40383c0 0x7f76a403a880 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:54.063+0000 7f76c26f0700 1 --2- 172.21.15.114:0/667241308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76bc101ea0 0x7f76bc1015a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:54.063+0000 7f76c26f0700 1 -- 172.21.15.114:0/667241308 >> 172.21.15.114:0/667241308 conn(0x7f76bc0fa4a0 msgr2=0x7f76bc0fb180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:54.064+0000 7f76c26f0700 1 -- 172.21.15.114:0/667241308 shutdown_connections 2024-02-19T18:52:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:54.064+0000 7f76c26f0700 1 -- 172.21.15.114:0/667241308 wait complete. 2024-02-19T18:52:54.724 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:54 smithi114 ceph-mon[29441]: Deploying daemon ceph-exporter.smithi114 on smithi114 2024-02-19T18:52:54.725 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:54 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:54.951 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi174 2024-02-19T18:52:54.951 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-02-19T18:52:54.952 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.conf 2024-02-19T18:52:54.970 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-02-19T18:52:54.970 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T18:52:55.027 INFO:tasks.cephadm:Adding host smithi174 to orchestrator... 2024-02-19T18:52:55.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph orch host add smithi174 2024-02-19T18:52:55.538 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:52:55.660 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:55 smithi114 ceph-mon[29441]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:52:56.573 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:56 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:56.574 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:56 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:56.574 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:56 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:56.574 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:56 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:56.574 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:56 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi114", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:52:56.574 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:56 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi114", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-19T18:52:56.574 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:56 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:52:56.574 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:56 smithi114 ceph-mon[29441]: Deploying daemon crash.smithi114 on smithi114 2024-02-19T18:52:57.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.965+0000 7fb8349c0700 1 -- 172.21.15.114:0/4137372178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8300ff7a0 msgr2=0x7fb8300ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:57.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.965+0000 7fb8349c0700 1 --2- 172.21.15.114:0/4137372178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8300ff7a0 0x7fb8300ffbc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb820009910 tx=0x7fb820009c20 comp rx=0 tx=0).stop 2024-02-19T18:52:57.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.966+0000 7fb8349c0700 1 -- 172.21.15.114:0/4137372178 shutdown_connections 2024-02-19T18:52:57.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.966+0000 7fb8349c0700 1 --2- 172.21.15.114:0/4137372178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8300ff7a0 0x7fb8300ffbc0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:57.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.966+0000 7fb8349c0700 1 -- 172.21.15.114:0/4137372178 >> 172.21.15.114:0/4137372178 conn(0x7fb8300fad20 msgr2=0x7fb8300fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:57.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.966+0000 7fb8349c0700 1 -- 172.21.15.114:0/4137372178 shutdown_connections 2024-02-19T18:52:57.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.966+0000 7fb8349c0700 1 -- 172.21.15.114:0/4137372178 wait complete. 2024-02-19T18:52:57.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.966+0000 7fb8349c0700 1 Processor -- start 2024-02-19T18:52:57.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.966+0000 7fb8349c0700 1 -- start start 2024-02-19T18:52:57.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.967+0000 7fb8349c0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8300ff7a0 0x7fb8301948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:57.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.967+0000 7fb8349c0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb830194de0 con 0x7fb8300ff7a0 2024-02-19T18:52:57.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.967+0000 7fb82e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8300ff7a0 0x7fb8301948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:57.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.967+0000 7fb82e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8300ff7a0 0x7fb8301948a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36556/0 (socket says 172.21.15.114:36556) 2024-02-19T18:52:57.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.967+0000 7fb82e59c700 1 -- 172.21.15.114:0/1092577865 learned_addr learned my addr 172.21.15.114:0/1092577865 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:52:57.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.967+0000 7fb82e59c700 1 -- 172.21.15.114:0/1092577865 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8200095f0 con 0x7fb8300ff7a0 2024-02-19T18:52:57.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.967+0000 7fb82e59c700 1 --2- 172.21.15.114:0/1092577865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8300ff7a0 0x7fb8301948a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb820005b40 tx=0x7fb820004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:57.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.967+0000 7fb81f7fe700 1 -- 172.21.15.114:0/1092577865 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb82001f070 con 0x7fb8300ff7a0 2024-02-19T18:52:57.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.967+0000 7fb8349c0700 1 -- 172.21.15.114:0/1092577865 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb830194fe0 con 0x7fb8300ff7a0 2024-02-19T18:52:57.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.967+0000 7fb81f7fe700 1 -- 172.21.15.114:0/1092577865 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb82000be10 con 0x7fb8300ff7a0 2024-02-19T18:52:57.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.967+0000 7fb81f7fe700 1 -- 172.21.15.114:0/1092577865 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb820024620 con 0x7fb8300ff7a0 2024-02-19T18:52:57.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.967+0000 7fb8349c0700 1 -- 172.21.15.114:0/1092577865 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb830195480 con 0x7fb8300ff7a0 2024-02-19T18:52:57.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.968+0000 7fb81f7fe700 1 -- 172.21.15.114:0/1092577865 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fb82000f460 con 0x7fb8300ff7a0 2024-02-19T18:52:57.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.968+0000 7fb81f7fe700 1 --2- 172.21.15.114:0/1092577865 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb818038410 0x7fb81803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:57.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.968+0000 7fb81f7fe700 1 -- 172.21.15.114:0/1092577865 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb82004e5c0 con 0x7fb8300ff7a0 2024-02-19T18:52:57.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.969+0000 7fb82dd9b700 1 --2- 172.21.15.114:0/1092577865 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb818038410 0x7fb81803a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:57.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.969+0000 7fb8349c0700 1 -- 172.21.15.114:0/1092577865 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8300623c0 con 0x7fb8300ff7a0 2024-02-19T18:52:57.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.969+0000 7fb82dd9b700 1 --2- 172.21.15.114:0/1092577865 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb818038410 0x7fb81803a8d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb824006fd0 tx=0x7fb824006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:57.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:57.974+0000 7fb81f7fe700 1 -- 172.21.15.114:0/1092577865 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb820029070 con 0x7fb8300ff7a0 2024-02-19T18:52:58.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:52:58.179+0000 7fb8349c0700 1 -- 172.21.15.114:0/1092577865 --> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}) v1 -- 0x7fb830068000 con 0x7fb818038410 2024-02-19T18:52:59.433 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:59 smithi114 ceph-mon[29441]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:59.433 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:59 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:59.433 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:59 smithi114 ceph-mon[29441]: from='client.14189 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:52:59.434 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:59 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:59.434 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:59 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:59.434 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:59 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:52:59.434 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:52:59 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:00.432 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:00 smithi114 ceph-mon[29441]: Deploying daemon node-exporter.smithi114 on smithi114 2024-02-19T18:53:00.433 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:00 smithi114 ceph-mon[29441]: Deploying cephadm binary to smithi174 2024-02-19T18:53:00.905 INFO:teuthology.orchestra.run.smithi114.stdout:Added host 'smithi174' with addr '172.21.15.174' 2024-02-19T18:53:00.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:00.887+0000 7fb81f7fe700 1 -- 172.21.15.114:0/1092577865 <== mgr.14162 v2:172.21.15.114:6800/1719431365 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fb830068000 con 0x7fb818038410 2024-02-19T18:53:00.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:00.889+0000 7fb8349c0700 1 -- 172.21.15.114:0/1092577865 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb818038410 msgr2=0x7fb81803a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:00.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:00.889+0000 7fb8349c0700 1 --2- 172.21.15.114:0/1092577865 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb818038410 0x7fb81803a8d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb824006fd0 tx=0x7fb824006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:00.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:00.889+0000 7fb8349c0700 1 -- 172.21.15.114:0/1092577865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8300ff7a0 msgr2=0x7fb8301948a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:00.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:00.889+0000 7fb8349c0700 1 --2- 172.21.15.114:0/1092577865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8300ff7a0 0x7fb8301948a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb820005b40 tx=0x7fb820004740 comp rx=0 tx=0).stop 2024-02-19T18:53:00.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:00.889+0000 7fb8349c0700 1 -- 172.21.15.114:0/1092577865 shutdown_connections 2024-02-19T18:53:00.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:00.889+0000 7fb8349c0700 1 --2- 172.21.15.114:0/1092577865 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb818038410 0x7fb81803a8d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:00.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:00.890+0000 7fb8349c0700 1 --2- 172.21.15.114:0/1092577865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8300ff7a0 0x7fb8301948a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:00.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:00.890+0000 7fb8349c0700 1 -- 172.21.15.114:0/1092577865 >> 172.21.15.114:0/1092577865 conn(0x7fb8300fad20 msgr2=0x7fb8300fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:00.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:00.890+0000 7fb8349c0700 1 -- 172.21.15.114:0/1092577865 shutdown_connections 2024-02-19T18:53:00.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:00.890+0000 7fb8349c0700 1 -- 172.21.15.114:0/1092577865 wait complete. 2024-02-19T18:53:01.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:01 smithi114 ceph-mon[29441]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:01.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:01 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:01.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph orch host ls --format=json 2024-02-19T18:53:02.124 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:53:02.490 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:02 smithi114 ceph-mon[29441]: Added host smithi174 2024-02-19T18:53:03.442 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:03 smithi114 ceph-mon[29441]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:03.443 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:03 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:03.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.833+0000 7fd75c48f700 1 -- 172.21.15.114:0/3676960308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd754100d70 msgr2=0x7fd754101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:03.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.833+0000 7fd75c48f700 1 --2- 172.21.15.114:0/3676960308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd754100d70 0x7fd754101190 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd750009910 tx=0x7fd750009c20 comp rx=0 tx=0).stop 2024-02-19T18:53:03.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.833+0000 7fd75c48f700 1 -- 172.21.15.114:0/3676960308 shutdown_connections 2024-02-19T18:53:03.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.833+0000 7fd75c48f700 1 --2- 172.21.15.114:0/3676960308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd754100d70 0x7fd754101190 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:03.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.833+0000 7fd75c48f700 1 -- 172.21.15.114:0/3676960308 >> 172.21.15.114:0/3676960308 conn(0x7fd7540fa4a0 msgr2=0x7fd7540fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:03.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.834+0000 7fd75c48f700 1 -- 172.21.15.114:0/3676960308 shutdown_connections 2024-02-19T18:53:03.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.834+0000 7fd75c48f700 1 -- 172.21.15.114:0/3676960308 wait complete. 2024-02-19T18:53:03.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.834+0000 7fd75c48f700 1 Processor -- start 2024-02-19T18:53:03.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.834+0000 7fd75c48f700 1 -- start start 2024-02-19T18:53:03.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.834+0000 7fd75c48f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd754100d70 0x7fd754197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:03.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.834+0000 7fd75c48f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd754198100 con 0x7fd754100d70 2024-02-19T18:53:03.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.834+0000 7fd75a22b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd754100d70 0x7fd754197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:03.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.834+0000 7fd75a22b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd754100d70 0x7fd754197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36574/0 (socket says 172.21.15.114:36574) 2024-02-19T18:53:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.834+0000 7fd75a22b700 1 -- 172.21.15.114:0/1912567794 learned_addr learned my addr 172.21.15.114:0/1912567794 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:53:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.835+0000 7fd75a22b700 1 -- 172.21.15.114:0/1912567794 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7500095f0 con 0x7fd754100d70 2024-02-19T18:53:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.835+0000 7fd75a22b700 1 --2- 172.21.15.114:0/1912567794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd754100d70 0x7fd754197bc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd750005b40 tx=0x7fd750004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.835+0000 7fd74b7fe700 1 -- 172.21.15.114:0/1912567794 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd75001d070 con 0x7fd754100d70 2024-02-19T18:53:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.835+0000 7fd75c48f700 1 -- 172.21.15.114:0/1912567794 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd754198300 con 0x7fd754100d70 2024-02-19T18:53:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.835+0000 7fd74b7fe700 1 -- 172.21.15.114:0/1912567794 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd75000be10 con 0x7fd754100d70 2024-02-19T18:53:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.835+0000 7fd74b7fe700 1 -- 172.21.15.114:0/1912567794 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd75000f460 con 0x7fd754100d70 2024-02-19T18:53:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.835+0000 7fd75c48f700 1 -- 172.21.15.114:0/1912567794 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7541987a0 con 0x7fd754100d70 2024-02-19T18:53:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.836+0000 7fd74b7fe700 1 -- 172.21.15.114:0/1912567794 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fd75000f6d0 con 0x7fd754100d70 2024-02-19T18:53:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.836+0000 7fd75c48f700 1 -- 172.21.15.114:0/1912567794 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7541916c0 con 0x7fd754100d70 2024-02-19T18:53:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.836+0000 7fd74b7fe700 1 --2- 172.21.15.114:0/1912567794 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fd7400383c0 0x7fd74003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.836+0000 7fd74b7fe700 1 -- 172.21.15.114:0/1912567794 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd75004c4e0 con 0x7fd754100d70 2024-02-19T18:53:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.836+0000 7fd759a2a700 1 --2- 172.21.15.114:0/1912567794 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fd7400383c0 0x7fd74003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:03.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.836+0000 7fd759a2a700 1 --2- 172.21.15.114:0/1912567794 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fd7400383c0 0x7fd74003a880 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd744006fd0 tx=0x7fd744006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:03.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:03.841+0000 7fd74b7fe700 1 -- 172.21.15.114:0/1912567794 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd7500209c0 con 0x7fd754100d70 2024-02-19T18:53:04.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:04.047+0000 7fd75c48f700 1 -- 172.21.15.114:0/1912567794 --> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd7540611d0 con 0x7fd7400383c0 2024-02-19T18:53:04.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:04.048+0000 7fd74b7fe700 1 -- 172.21.15.114:0/1912567794 <== mgr.14162 v2:172.21.15.114:6800/1719431365 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7fd7540611d0 con 0x7fd7400383c0 2024-02-19T18:53:04.064 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:53:04.064 INFO:teuthology.orchestra.run.smithi114.stdout:[{"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}, {"addr": "172.21.15.174", "hostname": "smithi174", "labels": [], "status": ""}] 2024-02-19T18:53:04.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:04.050+0000 7fd75c48f700 1 -- 172.21.15.114:0/1912567794 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fd7400383c0 msgr2=0x7fd74003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:04.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:04.050+0000 7fd75c48f700 1 --2- 172.21.15.114:0/1912567794 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fd7400383c0 0x7fd74003a880 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd744006fd0 tx=0x7fd744006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:04.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:04.050+0000 7fd75c48f700 1 -- 172.21.15.114:0/1912567794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd754100d70 msgr2=0x7fd754197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:04.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:04.050+0000 7fd75c48f700 1 --2- 172.21.15.114:0/1912567794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd754100d70 0x7fd754197bc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd750005b40 tx=0x7fd750004740 comp rx=0 tx=0).stop 2024-02-19T18:53:04.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:04.050+0000 7fd75c48f700 1 -- 172.21.15.114:0/1912567794 shutdown_connections 2024-02-19T18:53:04.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:04.050+0000 7fd75c48f700 1 --2- 172.21.15.114:0/1912567794 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fd7400383c0 0x7fd74003a880 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:04.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:04.050+0000 7fd75c48f700 1 --2- 172.21.15.114:0/1912567794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd754100d70 0x7fd754197bc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:04.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:04.050+0000 7fd75c48f700 1 -- 172.21.15.114:0/1912567794 >> 172.21.15.114:0/1912567794 conn(0x7fd7540fa4a0 msgr2=0x7fd7540fb180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:04.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:04.050+0000 7fd75c48f700 1 -- 172.21.15.114:0/1912567794 shutdown_connections 2024-02-19T18:53:04.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:04.050+0000 7fd75c48f700 1 -- 172.21.15.114:0/1912567794 wait complete. 2024-02-19T18:53:04.933 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-19T18:53:04.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph osd crush tunables default 2024-02-19T18:53:05.253 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:05 smithi114 ceph-mon[29441]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T18:53:05.253 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:05 smithi114 ceph-mon[29441]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:05.670 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:53:06.884 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:06 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:06.884 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:06 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:06.884 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:06 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:06.884 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:06 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:06.884 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:06 smithi114 ceph-mon[29441]: Deploying daemon alertmanager.smithi114 on smithi114 2024-02-19T18:53:06.884 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:06 smithi114 ceph-mon[29441]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:07.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.068+0000 7f5165f8f700 1 -- 172.21.15.114:0/315128395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5160101330 msgr2=0x7f5160103720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:07.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.068+0000 7f5165f8f700 1 --2- 172.21.15.114:0/315128395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5160101330 0x7f5160103720 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5150009910 tx=0x7f5150009c20 comp rx=0 tx=0).stop 2024-02-19T18:53:07.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.069+0000 7f5165f8f700 1 -- 172.21.15.114:0/315128395 shutdown_connections 2024-02-19T18:53:07.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.069+0000 7f5165f8f700 1 --2- 172.21.15.114:0/315128395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5160101330 0x7f5160103720 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:07.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.069+0000 7f5165f8f700 1 -- 172.21.15.114:0/315128395 >> 172.21.15.114:0/315128395 conn(0x7f51600fad20 msgr2=0x7f51600fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:07.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.069+0000 7f5165f8f700 1 -- 172.21.15.114:0/315128395 shutdown_connections 2024-02-19T18:53:07.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.069+0000 7f5165f8f700 1 -- 172.21.15.114:0/315128395 wait complete. 2024-02-19T18:53:07.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.069+0000 7f5165f8f700 1 Processor -- start 2024-02-19T18:53:07.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.069+0000 7f5165f8f700 1 -- start start 2024-02-19T18:53:07.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.069+0000 7f5165f8f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5160101330 0x7f5160195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:07.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.069+0000 7f5165f8f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5160195eb0 con 0x7f5160101330 2024-02-19T18:53:07.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.070+0000 7f515f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5160101330 0x7f5160195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:07.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.070+0000 7f515f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5160101330 0x7f5160195970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59750/0 (socket says 172.21.15.114:59750) 2024-02-19T18:53:07.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.070+0000 7f515f7fe700 1 -- 172.21.15.114:0/1091580454 learned_addr learned my addr 172.21.15.114:0/1091580454 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:53:07.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.070+0000 7f515f7fe700 1 -- 172.21.15.114:0/1091580454 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51500095f0 con 0x7f5160101330 2024-02-19T18:53:07.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.070+0000 7f515f7fe700 1 --2- 172.21.15.114:0/1091580454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5160101330 0x7f5160195970 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5150005b40 tx=0x7f5150004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:07.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.070+0000 7f515cff9700 1 -- 172.21.15.114:0/1091580454 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f515001d070 con 0x7f5160101330 2024-02-19T18:53:07.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.070+0000 7f515cff9700 1 -- 172.21.15.114:0/1091580454 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f515000be10 con 0x7f5160101330 2024-02-19T18:53:07.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.070+0000 7f5165f8f700 1 -- 172.21.15.114:0/1091580454 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51601960b0 con 0x7f5160101330 2024-02-19T18:53:07.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.070+0000 7f515cff9700 1 -- 172.21.15.114:0/1091580454 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f515000f460 con 0x7f5160101330 2024-02-19T18:53:07.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.070+0000 7f5165f8f700 1 -- 172.21.15.114:0/1091580454 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5160196550 con 0x7f5160101330 2024-02-19T18:53:07.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.071+0000 7f515cff9700 1 -- 172.21.15.114:0/1091580454 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f515000f6d0 con 0x7f5160101330 2024-02-19T18:53:07.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.071+0000 7f5165f8f700 1 -- 172.21.15.114:0/1091580454 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f516018f4c0 con 0x7f5160101330 2024-02-19T18:53:07.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.071+0000 7f515cff9700 1 --2- 172.21.15.114:0/1091580454 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f51480383c0 0x7f514803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:07.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.071+0000 7f515cff9700 1 -- 172.21.15.114:0/1091580454 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f515004c3f0 con 0x7f5160101330 2024-02-19T18:53:07.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.071+0000 7f515effd700 1 --2- 172.21.15.114:0/1091580454 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f51480383c0 0x7f514803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:07.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.072+0000 7f515effd700 1 --2- 172.21.15.114:0/1091580454 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f51480383c0 0x7f514803a880 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5154006fd0 tx=0x7f5154006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:07.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.076+0000 7f515cff9700 1 -- 172.21.15.114:0/1091580454 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f515002a360 con 0x7f5160101330 2024-02-19T18:53:07.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.270+0000 7f5165f8f700 1 -- 172.21.15.114:0/1091580454 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f516002cc70 con 0x7f5160101330 2024-02-19T18:53:07.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.632+0000 7f515cff9700 1 -- 172.21.15.114:0/1091580454 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5150027020 con 0x7f5160101330 2024-02-19T18:53:07.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.634+0000 7f5165f8f700 1 -- 172.21.15.114:0/1091580454 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f51480383c0 msgr2=0x7f514803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:07.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.634+0000 7f5165f8f700 1 --2- 172.21.15.114:0/1091580454 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f51480383c0 0x7f514803a880 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5154006fd0 tx=0x7f5154006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:07.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.634+0000 7f5165f8f700 1 -- 172.21.15.114:0/1091580454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5160101330 msgr2=0x7f5160195970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:07.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.634+0000 7f5165f8f700 1 --2- 172.21.15.114:0/1091580454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5160101330 0x7f5160195970 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5150005b40 tx=0x7f5150004740 comp rx=0 tx=0).stop 2024-02-19T18:53:07.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.634+0000 7f5165f8f700 1 -- 172.21.15.114:0/1091580454 shutdown_connections 2024-02-19T18:53:07.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.634+0000 7f5165f8f700 1 --2- 172.21.15.114:0/1091580454 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f51480383c0 0x7f514803a880 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:07.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.634+0000 7f5165f8f700 1 --2- 172.21.15.114:0/1091580454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5160101330 0x7f5160195970 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:07.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.634+0000 7f5165f8f700 1 -- 172.21.15.114:0/1091580454 >> 172.21.15.114:0/1091580454 conn(0x7f51600fad20 msgr2=0x7f51600fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:07.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.635+0000 7f5165f8f700 1 -- 172.21.15.114:0/1091580454 shutdown_connections 2024-02-19T18:53:07.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:53:07.635+0000 7f5165f8f700 1 -- 172.21.15.114:0/1091580454 wait complete. 2024-02-19T18:53:07.652 INFO:teuthology.orchestra.run.smithi114.stderr:adjusted tunables profile to default 2024-02-19T18:53:07.944 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:07 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1091580454' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-19T18:53:08.325 INFO:tasks.cephadm:Adding mon.smithi114 on smithi114 2024-02-19T18:53:08.325 INFO:tasks.cephadm:Adding mon.smithi174 on smithi174 2024-02-19T18:53:08.325 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph orch apply mon '2;smithi114:172.21.15.114=smithi114;smithi174:172.21.15.174=smithi174' 2024-02-19T18:53:08.528 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:08.574 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:09.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:08 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1091580454' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-19T18:53:09.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:08 smithi114 ceph-mon[29441]: osdmap e4: 0 total, 0 up, 0 in 2024-02-19T18:53:09.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:08 smithi114 ceph-mon[29441]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:09.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:08 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:10.033 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.015+0000 7f5de21a3700 1 -- 172.21.15.174:0/681378532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ddc074d60 msgr2=0x7f5ddc0731c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:10.033 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.015+0000 7f5de21a3700 1 --2- 172.21.15.174:0/681378532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ddc074d60 0x7f5ddc0731c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc009910 tx=0x7f5dcc009c20 comp rx=0 tx=0).stop 2024-02-19T18:53:10.033 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.016+0000 7f5de21a3700 1 -- 172.21.15.174:0/681378532 shutdown_connections 2024-02-19T18:53:10.033 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.016+0000 7f5de21a3700 1 --2- 172.21.15.174:0/681378532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ddc074d60 0x7f5ddc0731c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:10.034 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.016+0000 7f5de21a3700 1 -- 172.21.15.174:0/681378532 >> 172.21.15.174:0/681378532 conn(0x7f5ddc0fbe30 msgr2=0x7f5ddc0fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:10.034 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.016+0000 7f5de21a3700 1 -- 172.21.15.174:0/681378532 shutdown_connections 2024-02-19T18:53:10.034 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.016+0000 7f5de21a3700 1 -- 172.21.15.174:0/681378532 wait complete. 2024-02-19T18:53:10.034 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.016+0000 7f5de21a3700 1 Processor -- start 2024-02-19T18:53:10.034 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.016+0000 7f5de21a3700 1 -- start start 2024-02-19T18:53:10.034 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.016+0000 7f5de21a3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ddc074d60 0x7f5ddc197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:10.034 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.016+0000 7f5de21a3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc1980d0 con 0x7f5ddc074d60 2024-02-19T18:53:10.034 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.017+0000 7f5ddb7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ddc074d60 0x7f5ddc197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:10.034 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.017+0000 7f5ddb7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ddc074d60 0x7f5ddc197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:53728/0 (socket says 172.21.15.174:53728) 2024-02-19T18:53:10.034 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.017+0000 7f5ddb7fe700 1 -- 172.21.15.174:0/2689642217 learned_addr learned my addr 172.21.15.174:0/2689642217 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:53:10.034 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.017+0000 7f5ddb7fe700 1 -- 172.21.15.174:0/2689642217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc0095f0 con 0x7f5ddc074d60 2024-02-19T18:53:10.035 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.017+0000 7f5ddb7fe700 1 --2- 172.21.15.174:0/2689642217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ddc074d60 0x7f5ddc197b90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc005b40 tx=0x7f5dcc004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:10.036 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.017+0000 7f5dd97fa700 1 -- 172.21.15.174:0/2689642217 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5dcc01d070 con 0x7f5ddc074d60 2024-02-19T18:53:10.036 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.017+0000 7f5dd97fa700 1 -- 172.21.15.174:0/2689642217 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5dcc00be10 con 0x7f5ddc074d60 2024-02-19T18:53:10.036 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.017+0000 7f5de21a3700 1 -- 172.21.15.174:0/2689642217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ddc1982d0 con 0x7f5ddc074d60 2024-02-19T18:53:10.036 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.017+0000 7f5dd97fa700 1 -- 172.21.15.174:0/2689642217 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5dcc00f460 con 0x7f5ddc074d60 2024-02-19T18:53:10.036 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.017+0000 7f5de21a3700 1 -- 172.21.15.174:0/2689642217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ddc198770 con 0x7f5ddc074d60 2024-02-19T18:53:10.036 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.018+0000 7f5de21a3700 1 -- 172.21.15.174:0/2689642217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ddc1916d0 con 0x7f5ddc074d60 2024-02-19T18:53:10.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.018+0000 7f5dd97fa700 1 -- 172.21.15.174:0/2689642217 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f5dcc0227e0 con 0x7f5ddc074d60 2024-02-19T18:53:10.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.019+0000 7f5dd97fa700 1 --2- 172.21.15.174:0/2689642217 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f5dc80383c0 0x7f5dc803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:10.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.019+0000 7f5dd97fa700 1 -- 172.21.15.174:0/2689642217 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5dcc04c580 con 0x7f5ddc074d60 2024-02-19T18:53:10.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.019+0000 7f5dd2ffd700 1 --2- 172.21.15.174:0/2689642217 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f5dc80383c0 0x7f5dc803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:10.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.019+0000 7f5dd2ffd700 1 --2- 172.21.15.174:0/2689642217 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f5dc80383c0 0x7f5dc803a880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4006fd0 tx=0x7f5dc4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:10.040 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.024+0000 7f5dd97fa700 1 -- 172.21.15.174:0/2689642217 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5dcc02bba0 con 0x7f5ddc074d60 2024-02-19T18:53:10.238 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.221+0000 7f5de21a3700 1 -- 172.21.15.174:0/2689642217 --> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi114:172.21.15.114=smithi114;smithi174:172.21.15.174=smithi174", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ddc061340 con 0x7f5dc80383c0 2024-02-19T18:53:10.245 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.229+0000 7f5dd97fa700 1 -- 172.21.15.174:0/2689642217 <== mgr.14162 v2:172.21.15.114:6800/1719431365 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5ddc061340 con 0x7f5dc80383c0 2024-02-19T18:53:10.245 INFO:teuthology.orchestra.run.smithi174.stdout:Scheduled mon update... 2024-02-19T18:53:10.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.231+0000 7f5de21a3700 1 -- 172.21.15.174:0/2689642217 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f5dc80383c0 msgr2=0x7f5dc803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:10.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.231+0000 7f5de21a3700 1 --2- 172.21.15.174:0/2689642217 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f5dc80383c0 0x7f5dc803a880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4006fd0 tx=0x7f5dc4006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:10.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.231+0000 7f5de21a3700 1 -- 172.21.15.174:0/2689642217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ddc074d60 msgr2=0x7f5ddc197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:10.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.231+0000 7f5de21a3700 1 --2- 172.21.15.174:0/2689642217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ddc074d60 0x7f5ddc197b90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc005b40 tx=0x7f5dcc004740 comp rx=0 tx=0).stop 2024-02-19T18:53:10.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.231+0000 7f5de21a3700 1 -- 172.21.15.174:0/2689642217 shutdown_connections 2024-02-19T18:53:10.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.231+0000 7f5de21a3700 1 --2- 172.21.15.174:0/2689642217 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f5dc80383c0 0x7f5dc803a880 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:10.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.231+0000 7f5de21a3700 1 --2- 172.21.15.174:0/2689642217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ddc074d60 0x7f5ddc197b90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:10.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.231+0000 7f5de21a3700 1 -- 172.21.15.174:0/2689642217 >> 172.21.15.174:0/2689642217 conn(0x7f5ddc0fbe30 msgr2=0x7f5ddc0fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:10.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.231+0000 7f5de21a3700 1 -- 172.21.15.174:0/2689642217 shutdown_connections 2024-02-19T18:53:10.248 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:10.231+0000 7f5de21a3700 1 -- 172.21.15.174:0/2689642217 wait complete. 2024-02-19T18:53:10.812 DEBUG:teuthology.orchestra.run.smithi174:mon.smithi174> sudo journalctl -f -n 0 -u ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@mon.smithi174.service 2024-02-19T18:53:10.815 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:10.815 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:11.196 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:11.242 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:11.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:11 smithi114 ceph-mon[29441]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:11.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:11 smithi114 ceph-mon[29441]: from='client.14195 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi114:172.21.15.114=smithi114;smithi174:172.21.15.174=smithi174", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:53:11.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:11 smithi114 ceph-mon[29441]: Saving service mon spec with placement smithi114:172.21.15.114=smithi114;smithi174:172.21.15.174=smithi174;count:2 2024-02-19T18:53:11.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:11 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:12.503 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.486+0000 7f91cd606700 1 -- 172.21.15.174:0/3426149885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91c81008b0 msgr2=0x7f91c8100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:12.503 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.486+0000 7f91cd606700 1 --2- 172.21.15.174:0/3426149885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91c81008b0 0x7f91c8100cd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f91b8009910 tx=0x7f91b8009c20 comp rx=0 tx=0).stop 2024-02-19T18:53:12.503 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.486+0000 7f91cd606700 1 -- 172.21.15.174:0/3426149885 shutdown_connections 2024-02-19T18:53:12.503 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.486+0000 7f91cd606700 1 --2- 172.21.15.174:0/3426149885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91c81008b0 0x7f91c8100cd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:12.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.486+0000 7f91cd606700 1 -- 172.21.15.174:0/3426149885 >> 172.21.15.174:0/3426149885 conn(0x7f91c80fbe30 msgr2=0x7f91c80fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:12.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.486+0000 7f91cd606700 1 -- 172.21.15.174:0/3426149885 shutdown_connections 2024-02-19T18:53:12.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.486+0000 7f91cd606700 1 -- 172.21.15.174:0/3426149885 wait complete. 2024-02-19T18:53:12.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.487+0000 7f91cd606700 1 Processor -- start 2024-02-19T18:53:12.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.487+0000 7f91cd606700 1 -- start start 2024-02-19T18:53:12.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.487+0000 7f91cd606700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91c81008b0 0x7f91c8074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:12.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.487+0000 7f91cd606700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c8075070 con 0x7f91c81008b0 2024-02-19T18:53:12.506 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.487+0000 7f91c6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91c81008b0 0x7f91c8074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:12.506 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.487+0000 7f91c6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91c81008b0 0x7f91c8074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:53738/0 (socket says 172.21.15.174:53738) 2024-02-19T18:53:12.506 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.487+0000 7f91c6ffd700 1 -- 172.21.15.174:0/355250765 learned_addr learned my addr 172.21.15.174:0/355250765 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:53:12.506 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.488+0000 7f91c6ffd700 1 -- 172.21.15.174:0/355250765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b80095f0 con 0x7f91c81008b0 2024-02-19T18:53:12.506 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.488+0000 7f91c6ffd700 1 --2- 172.21.15.174:0/355250765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91c81008b0 0x7f91c8074b30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f91b8005b40 tx=0x7f91b8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:12.506 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.488+0000 7f91bffff700 1 -- 172.21.15.174:0/355250765 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91b801d070 con 0x7f91c81008b0 2024-02-19T18:53:12.506 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.488+0000 7f91cd606700 1 -- 172.21.15.174:0/355250765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c8073180 con 0x7f91c81008b0 2024-02-19T18:53:12.506 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.488+0000 7f91bffff700 1 -- 172.21.15.174:0/355250765 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f91b800be10 con 0x7f91c81008b0 2024-02-19T18:53:12.506 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.488+0000 7f91bffff700 1 -- 172.21.15.174:0/355250765 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91b800f460 con 0x7f91c81008b0 2024-02-19T18:53:12.506 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.488+0000 7f91cd606700 1 -- 172.21.15.174:0/355250765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c8073620 con 0x7f91c81008b0 2024-02-19T18:53:12.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.489+0000 7f91bffff700 1 -- 172.21.15.174:0/355250765 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f91b80227e0 con 0x7f91c81008b0 2024-02-19T18:53:12.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.489+0000 7f91cd606700 1 -- 172.21.15.174:0/355250765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c80623c0 con 0x7f91c81008b0 2024-02-19T18:53:12.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.489+0000 7f91bffff700 1 --2- 172.21.15.174:0/355250765 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f91b40383c0 0x7f91b403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:12.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.489+0000 7f91bffff700 1 -- 172.21.15.174:0/355250765 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f91b804c4e0 con 0x7f91c81008b0 2024-02-19T18:53:12.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.489+0000 7f91c67fc700 1 --2- 172.21.15.174:0/355250765 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f91b40383c0 0x7f91b403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:12.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.490+0000 7f91c67fc700 1 --2- 172.21.15.174:0/355250765 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f91b40383c0 0x7f91b403a880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f91b0006fd0 tx=0x7f91b0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:12.510 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.494+0000 7f91bffff700 1 -- 172.21.15.174:0/355250765 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f91b802bba0 con 0x7f91c81008b0 2024-02-19T18:53:12.776 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.760+0000 7f91cd606700 1 -- 172.21.15.174:0/355250765 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91c818eeb0 con 0x7f91c81008b0 2024-02-19T18:53:12.777 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.761+0000 7f91bffff700 1 -- 172.21.15.174:0/355250765 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f91b8027020 con 0x7f91c81008b0 2024-02-19T18:53:12.777 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:53:12.778 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:12.780 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.763+0000 7f91cd606700 1 -- 172.21.15.174:0/355250765 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f91b40383c0 msgr2=0x7f91b403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:12.780 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.763+0000 7f91cd606700 1 --2- 172.21.15.174:0/355250765 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f91b40383c0 0x7f91b403a880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f91b0006fd0 tx=0x7f91b0006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:12.780 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.763+0000 7f91cd606700 1 -- 172.21.15.174:0/355250765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91c81008b0 msgr2=0x7f91c8074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:12.780 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.763+0000 7f91cd606700 1 --2- 172.21.15.174:0/355250765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91c81008b0 0x7f91c8074b30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f91b8005b40 tx=0x7f91b8004740 comp rx=0 tx=0).stop 2024-02-19T18:53:12.780 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.763+0000 7f91cd606700 1 -- 172.21.15.174:0/355250765 shutdown_connections 2024-02-19T18:53:12.780 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.763+0000 7f91cd606700 1 --2- 172.21.15.174:0/355250765 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f91b40383c0 0x7f91b403a880 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:12.780 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.763+0000 7f91cd606700 1 --2- 172.21.15.174:0/355250765 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91c81008b0 0x7f91c8074b30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:12.780 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.763+0000 7f91cd606700 1 -- 172.21.15.174:0/355250765 >> 172.21.15.174:0/355250765 conn(0x7f91c80fbe30 msgr2=0x7f91c80fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:12.781 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.763+0000 7f91cd606700 1 -- 172.21.15.174:0/355250765 shutdown_connections 2024-02-19T18:53:12.781 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:12.763+0000 7f91cd606700 1 -- 172.21.15.174:0/355250765 wait complete. 2024-02-19T18:53:12.781 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:53:13.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:13 smithi114 ceph-mon[29441]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:13.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:13 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/355250765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:14.288 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:14.288 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:14.488 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:14.535 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:15.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:15 smithi114 ceph-mon[29441]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:15.939 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.921+0000 7f9d734f8700 1 -- 172.21.15.174:0/1463464700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d6c0ff790 msgr2=0x7f9d6c0ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:15.939 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.921+0000 7f9d734f8700 1 --2- 172.21.15.174:0/1463464700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d6c0ff790 0x7f9d6c0ffbb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9d6400b0a0 tx=0x7f9d6400b3b0 comp rx=0 tx=0).stop 2024-02-19T18:53:15.939 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.922+0000 7f9d734f8700 1 -- 172.21.15.174:0/1463464700 shutdown_connections 2024-02-19T18:53:15.939 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.922+0000 7f9d734f8700 1 --2- 172.21.15.174:0/1463464700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d6c0ff790 0x7f9d6c0ffbb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:15.939 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.922+0000 7f9d734f8700 1 -- 172.21.15.174:0/1463464700 >> 172.21.15.174:0/1463464700 conn(0x7f9d6c0fad10 msgr2=0x7f9d6c0fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:15.939 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.922+0000 7f9d734f8700 1 -- 172.21.15.174:0/1463464700 shutdown_connections 2024-02-19T18:53:15.939 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.922+0000 7f9d734f8700 1 -- 172.21.15.174:0/1463464700 wait complete. 2024-02-19T18:53:15.939 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.922+0000 7f9d734f8700 1 Processor -- start 2024-02-19T18:53:15.939 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.922+0000 7f9d734f8700 1 -- start start 2024-02-19T18:53:15.939 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.922+0000 7f9d734f8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d6c0ff790 0x7f9d6c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:15.939 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.922+0000 7f9d734f8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c1964e0 con 0x7f9d6c0ff790 2024-02-19T18:53:15.940 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.923+0000 7f9d71294700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d6c0ff790 0x7f9d6c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:15.940 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.923+0000 7f9d71294700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d6c0ff790 0x7f9d6c069410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:55010/0 (socket says 172.21.15.174:55010) 2024-02-19T18:53:15.940 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.923+0000 7f9d71294700 1 -- 172.21.15.174:0/3454236952 learned_addr learned my addr 172.21.15.174:0/3454236952 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:53:15.940 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.923+0000 7f9d71294700 1 -- 172.21.15.174:0/3454236952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d64009d00 con 0x7f9d6c0ff790 2024-02-19T18:53:15.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.923+0000 7f9d71294700 1 --2- 172.21.15.174:0/3454236952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d6c0ff790 0x7f9d6c069410 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9d6400b830 tx=0x7f9d6400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:15.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.923+0000 7f9d627fc700 1 -- 172.21.15.174:0/3454236952 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d6400e040 con 0x7f9d6c0ff790 2024-02-19T18:53:15.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.923+0000 7f9d627fc700 1 -- 172.21.15.174:0/3454236952 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9d64012af0 con 0x7f9d6c0ff790 2024-02-19T18:53:15.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.923+0000 7f9d734f8700 1 -- 172.21.15.174:0/3454236952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d6c19aaf0 con 0x7f9d6c0ff790 2024-02-19T18:53:15.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.923+0000 7f9d627fc700 1 -- 172.21.15.174:0/3454236952 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d6401d9f0 con 0x7f9d6c0ff790 2024-02-19T18:53:15.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.923+0000 7f9d734f8700 1 -- 172.21.15.174:0/3454236952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6c19ae50 con 0x7f9d6c0ff790 2024-02-19T18:53:15.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.924+0000 7f9d627fc700 1 -- 172.21.15.174:0/3454236952 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f9d6401dcb0 con 0x7f9d6c0ff790 2024-02-19T18:53:15.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.924+0000 7f9d734f8700 1 -- 172.21.15.174:0/3454236952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6c18e3b0 con 0x7f9d6c0ff790 2024-02-19T18:53:15.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.924+0000 7f9d627fc700 1 --2- 172.21.15.174:0/3454236952 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f9d580383c0 0x7f9d5803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:15.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.924+0000 7f9d627fc700 1 -- 172.21.15.174:0/3454236952 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9d6404c9b0 con 0x7f9d6c0ff790 2024-02-19T18:53:15.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.924+0000 7f9d70a93700 1 --2- 172.21.15.174:0/3454236952 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f9d580383c0 0x7f9d5803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:15.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.925+0000 7f9d70a93700 1 --2- 172.21.15.174:0/3454236952 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f9d580383c0 0x7f9d5803a880 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c006fd0 tx=0x7f9d5c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:15.945 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:15.929+0000 7f9d627fc700 1 -- 172.21.15.174:0/3454236952 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9d6401abf0 con 0x7f9d6c0ff790 2024-02-19T18:53:16.209 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:16.192+0000 7f9d734f8700 1 -- 172.21.15.174:0/3454236952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d6c02cc70 con 0x7f9d6c0ff790 2024-02-19T18:53:16.209 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:16.193+0000 7f9d627fc700 1 -- 172.21.15.174:0/3454236952 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f9d6401abf0 con 0x7f9d6c0ff790 2024-02-19T18:53:16.209 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:53:16.209 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:16.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:16.195+0000 7f9d734f8700 1 -- 172.21.15.174:0/3454236952 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f9d580383c0 msgr2=0x7f9d5803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:16.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:16.195+0000 7f9d734f8700 1 --2- 172.21.15.174:0/3454236952 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f9d580383c0 0x7f9d5803a880 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c006fd0 tx=0x7f9d5c006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:16.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:16.195+0000 7f9d734f8700 1 -- 172.21.15.174:0/3454236952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d6c0ff790 msgr2=0x7f9d6c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:16.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:16.195+0000 7f9d734f8700 1 --2- 172.21.15.174:0/3454236952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d6c0ff790 0x7f9d6c069410 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9d6400b830 tx=0x7f9d6400bd40 comp rx=0 tx=0).stop 2024-02-19T18:53:16.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:16.195+0000 7f9d734f8700 1 -- 172.21.15.174:0/3454236952 shutdown_connections 2024-02-19T18:53:16.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:16.195+0000 7f9d734f8700 1 --2- 172.21.15.174:0/3454236952 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f9d580383c0 0x7f9d5803a880 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:16.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:16.195+0000 7f9d734f8700 1 --2- 172.21.15.174:0/3454236952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d6c0ff790 0x7f9d6c069410 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:16.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:16.195+0000 7f9d734f8700 1 -- 172.21.15.174:0/3454236952 >> 172.21.15.174:0/3454236952 conn(0x7f9d6c0fad10 msgr2=0x7f9d6c0fba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:16.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:16.195+0000 7f9d734f8700 1 -- 172.21.15.174:0/3454236952 shutdown_connections 2024-02-19T18:53:16.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:16.195+0000 7f9d734f8700 1 -- 172.21.15.174:0/3454236952 wait complete. 2024-02-19T18:53:16.212 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:53:16.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:16 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/3454236952' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:17.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:17 smithi114 ceph-mon[29441]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:17.791 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:17.792 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:17.991 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:18.037 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:19.201 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:18 smithi114 ceph-mon[29441]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:19.535 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.518+0000 7fa392398700 1 -- 172.21.15.174:0/1699854815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa38c0ff7a0 msgr2=0x7fa38c0ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:19.535 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.518+0000 7fa392398700 1 --2- 172.21.15.174:0/1699854815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa38c0ff7a0 0x7fa38c0ffbc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa37c009910 tx=0x7fa37c009c20 comp rx=0 tx=0).stop 2024-02-19T18:53:19.535 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.518+0000 7fa392398700 1 -- 172.21.15.174:0/1699854815 shutdown_connections 2024-02-19T18:53:19.535 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.518+0000 7fa392398700 1 --2- 172.21.15.174:0/1699854815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa38c0ff7a0 0x7fa38c0ffbc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:19.535 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.518+0000 7fa392398700 1 -- 172.21.15.174:0/1699854815 >> 172.21.15.174:0/1699854815 conn(0x7fa38c0fad20 msgr2=0x7fa38c0fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:19.536 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.518+0000 7fa392398700 1 -- 172.21.15.174:0/1699854815 shutdown_connections 2024-02-19T18:53:19.536 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.518+0000 7fa392398700 1 -- 172.21.15.174:0/1699854815 wait complete. 2024-02-19T18:53:19.536 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.518+0000 7fa392398700 1 Processor -- start 2024-02-19T18:53:19.536 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.518+0000 7fa392398700 1 -- start start 2024-02-19T18:53:19.536 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.519+0000 7fa392398700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa38c0ff7a0 0x7fa38c0fcce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:19.536 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.519+0000 7fa392398700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c0fd220 con 0x7fa38c0ff7a0 2024-02-19T18:53:19.536 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.519+0000 7fa38bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa38c0ff7a0 0x7fa38c0fcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:19.536 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.519+0000 7fa38bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa38c0ff7a0 0x7fa38c0fcce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:55024/0 (socket says 172.21.15.174:55024) 2024-02-19T18:53:19.536 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.519+0000 7fa38bfff700 1 -- 172.21.15.174:0/3423903034 learned_addr learned my addr 172.21.15.174:0/3423903034 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:53:19.538 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.519+0000 7fa38bfff700 1 -- 172.21.15.174:0/3423903034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c0095f0 con 0x7fa38c0ff7a0 2024-02-19T18:53:19.538 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.519+0000 7fa38bfff700 1 --2- 172.21.15.174:0/3423903034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa38c0ff7a0 0x7fa38c0fcce0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa37c005b40 tx=0x7fa37c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:19.538 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.520+0000 7fa3897fa700 1 -- 172.21.15.174:0/3423903034 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa37c01d070 con 0x7fa38c0ff7a0 2024-02-19T18:53:19.538 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.520+0000 7fa392398700 1 -- 172.21.15.174:0/3423903034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa38c0fb330 con 0x7fa38c0ff7a0 2024-02-19T18:53:19.538 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.520+0000 7fa3897fa700 1 -- 172.21.15.174:0/3423903034 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa37c00be10 con 0x7fa38c0ff7a0 2024-02-19T18:53:19.538 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.520+0000 7fa3897fa700 1 -- 172.21.15.174:0/3423903034 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa37c00f460 con 0x7fa38c0ff7a0 2024-02-19T18:53:19.538 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.520+0000 7fa392398700 1 -- 172.21.15.174:0/3423903034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa38c0fb7d0 con 0x7fa38c0ff7a0 2024-02-19T18:53:19.538 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.521+0000 7fa3897fa700 1 -- 172.21.15.174:0/3423903034 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fa37c00b960 con 0x7fa38c0ff7a0 2024-02-19T18:53:19.538 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.521+0000 7fa392398700 1 -- 172.21.15.174:0/3423903034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38c0623c0 con 0x7fa38c0ff7a0 2024-02-19T18:53:19.538 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.521+0000 7fa3897fa700 1 --2- 172.21.15.174:0/3423903034 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fa3740383c0 0x7fa37403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:19.538 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.521+0000 7fa3897fa700 1 -- 172.21.15.174:0/3423903034 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa37c04c480 con 0x7fa38c0ff7a0 2024-02-19T18:53:19.539 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.521+0000 7fa38b7fe700 1 --2- 172.21.15.174:0/3423903034 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fa3740383c0 0x7fa37403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:19.539 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.521+0000 7fa38b7fe700 1 --2- 172.21.15.174:0/3423903034 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fa3740383c0 0x7fa37403a880 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa380006fd0 tx=0x7fa380006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:19.542 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.526+0000 7fa3897fa700 1 -- 172.21.15.174:0/3423903034 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa37c02a360 con 0x7fa38c0ff7a0 2024-02-19T18:53:19.806 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.789+0000 7fa392398700 1 -- 172.21.15.174:0/3423903034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa38c18edb0 con 0x7fa38c0ff7a0 2024-02-19T18:53:19.806 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.790+0000 7fa3897fa700 1 -- 172.21.15.174:0/3423903034 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fa37c027030 con 0x7fa38c0ff7a0 2024-02-19T18:53:19.807 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:53:19.807 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:19.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.792+0000 7fa392398700 1 -- 172.21.15.174:0/3423903034 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fa3740383c0 msgr2=0x7fa37403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:19.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.792+0000 7fa392398700 1 --2- 172.21.15.174:0/3423903034 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fa3740383c0 0x7fa37403a880 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa380006fd0 tx=0x7fa380006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:19.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.792+0000 7fa392398700 1 -- 172.21.15.174:0/3423903034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa38c0ff7a0 msgr2=0x7fa38c0fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:19.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.792+0000 7fa392398700 1 --2- 172.21.15.174:0/3423903034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa38c0ff7a0 0x7fa38c0fcce0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa37c005b40 tx=0x7fa37c004740 comp rx=0 tx=0).stop 2024-02-19T18:53:19.809 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.792+0000 7fa392398700 1 -- 172.21.15.174:0/3423903034 shutdown_connections 2024-02-19T18:53:19.809 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.792+0000 7fa392398700 1 --2- 172.21.15.174:0/3423903034 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fa3740383c0 0x7fa37403a880 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:19.809 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.792+0000 7fa392398700 1 --2- 172.21.15.174:0/3423903034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa38c0ff7a0 0x7fa38c0fcce0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:19.809 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.792+0000 7fa392398700 1 -- 172.21.15.174:0/3423903034 >> 172.21.15.174:0/3423903034 conn(0x7fa38c0fad20 msgr2=0x7fa38c069100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:19.809 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.792+0000 7fa392398700 1 -- 172.21.15.174:0/3423903034 shutdown_connections 2024-02-19T18:53:19.809 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:19.792+0000 7fa392398700 1 -- 172.21.15.174:0/3423903034 wait complete. 2024-02-19T18:53:19.809 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:53:20.200 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:19 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/3423903034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:21.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:20 smithi114 ceph-mon[29441]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:21.360 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:21.360 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:21.564 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:21.609 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:22.576 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:22 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:22.577 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:22 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:22.577 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:22 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:22.577 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:22 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:22.577 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:22 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:22.577 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:22 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:22.578 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:22 smithi114 ceph-mon[29441]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-19T18:53:22.578 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:22 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:22.578 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:22 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:22.578 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:22 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T18:53:22.578 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:22 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:23.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.026+0000 7f396d573700 1 -- 172.21.15.174:0/994486250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39680ff7a0 msgr2=0x7f39680ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:23.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.026+0000 7f396d573700 1 --2- 172.21.15.174:0/994486250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39680ff7a0 0x7f39680ffbc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3958009910 tx=0x7f3958009c20 comp rx=0 tx=0).stop 2024-02-19T18:53:23.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.026+0000 7f396d573700 1 -- 172.21.15.174:0/994486250 shutdown_connections 2024-02-19T18:53:23.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.026+0000 7f396d573700 1 --2- 172.21.15.174:0/994486250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39680ff7a0 0x7f39680ffbc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:23.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.026+0000 7f396d573700 1 -- 172.21.15.174:0/994486250 >> 172.21.15.174:0/994486250 conn(0x7f39680fad20 msgr2=0x7f39680fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:23.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.026+0000 7f396d573700 1 -- 172.21.15.174:0/994486250 shutdown_connections 2024-02-19T18:53:23.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.026+0000 7f396d573700 1 -- 172.21.15.174:0/994486250 wait complete. 2024-02-19T18:53:23.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.026+0000 7f396d573700 1 Processor -- start 2024-02-19T18:53:23.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.027+0000 7f396d573700 1 -- start start 2024-02-19T18:53:23.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.027+0000 7f396d573700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39680ff7a0 0x7f39680fcce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:23.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.027+0000 7f396d573700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39680fd220 con 0x7f39680ff7a0 2024-02-19T18:53:23.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.027+0000 7f3966ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39680ff7a0 0x7f39680fcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:23.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.027+0000 7f3966ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39680ff7a0 0x7f39680fcce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:55044/0 (socket says 172.21.15.174:55044) 2024-02-19T18:53:23.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.027+0000 7f3966ffd700 1 -- 172.21.15.174:0/3423464834 learned_addr learned my addr 172.21.15.174:0/3423464834 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:53:23.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.027+0000 7f3966ffd700 1 -- 172.21.15.174:0/3423464834 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39580095f0 con 0x7f39680ff7a0 2024-02-19T18:53:23.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.027+0000 7f3966ffd700 1 --2- 172.21.15.174:0/3423464834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39680ff7a0 0x7f39680fcce0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3958005b40 tx=0x7f3958004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:23.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.027+0000 7f3964ff9700 1 -- 172.21.15.174:0/3423464834 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f395801d070 con 0x7f39680ff7a0 2024-02-19T18:53:23.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.027+0000 7f3964ff9700 1 -- 172.21.15.174:0/3423464834 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f395800be10 con 0x7f39680ff7a0 2024-02-19T18:53:23.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.027+0000 7f396d573700 1 -- 172.21.15.174:0/3423464834 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39680fb330 con 0x7f39680ff7a0 2024-02-19T18:53:23.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.028+0000 7f3964ff9700 1 -- 172.21.15.174:0/3423464834 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f395800f460 con 0x7f39680ff7a0 2024-02-19T18:53:23.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.028+0000 7f396d573700 1 -- 172.21.15.174:0/3423464834 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39680fb7d0 con 0x7f39680ff7a0 2024-02-19T18:53:23.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.028+0000 7f3964ff9700 1 -- 172.21.15.174:0/3423464834 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f395800b960 con 0x7f39680ff7a0 2024-02-19T18:53:23.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.028+0000 7f396d573700 1 -- 172.21.15.174:0/3423464834 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39680623c0 con 0x7f39680ff7a0 2024-02-19T18:53:23.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.029+0000 7f3964ff9700 1 --2- 172.21.15.174:0/3423464834 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f39540383c0 0x7f395403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:23.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.029+0000 7f3964ff9700 1 -- 172.21.15.174:0/3423464834 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f395804c3c0 con 0x7f39680ff7a0 2024-02-19T18:53:23.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.029+0000 7f395e7fc700 1 --2- 172.21.15.174:0/3423464834 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f39540383c0 0x7f395403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:23.048 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.029+0000 7f395e7fc700 1 --2- 172.21.15.174:0/3423464834 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f39540383c0 0x7f395403a880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3950006fd0 tx=0x7f3950006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:23.050 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.033+0000 7f3964ff9700 1 -- 172.21.15.174:0/3423464834 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f395802a360 con 0x7f39680ff7a0 2024-02-19T18:53:23.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.298+0000 7f396d573700 1 -- 172.21.15.174:0/3423464834 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f396818edb0 con 0x7f39680ff7a0 2024-02-19T18:53:23.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.299+0000 7f3964ff9700 1 -- 172.21.15.174:0/3423464834 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3958027030 con 0x7f39680ff7a0 2024-02-19T18:53:23.315 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:53:23.316 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:23.318 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.301+0000 7f396d573700 1 -- 172.21.15.174:0/3423464834 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f39540383c0 msgr2=0x7f395403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:23.318 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.301+0000 7f396d573700 1 --2- 172.21.15.174:0/3423464834 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f39540383c0 0x7f395403a880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3950006fd0 tx=0x7f3950006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:23.318 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.301+0000 7f396d573700 1 -- 172.21.15.174:0/3423464834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39680ff7a0 msgr2=0x7f39680fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:23.318 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.301+0000 7f396d573700 1 --2- 172.21.15.174:0/3423464834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39680ff7a0 0x7f39680fcce0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3958005b40 tx=0x7f3958004740 comp rx=0 tx=0).stop 2024-02-19T18:53:23.318 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.301+0000 7f396d573700 1 -- 172.21.15.174:0/3423464834 shutdown_connections 2024-02-19T18:53:23.318 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.301+0000 7f396d573700 1 --2- 172.21.15.174:0/3423464834 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f39540383c0 0x7f395403a880 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:23.318 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.301+0000 7f396d573700 1 --2- 172.21.15.174:0/3423464834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f39680ff7a0 0x7f39680fcce0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:23.318 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.301+0000 7f396d573700 1 -- 172.21.15.174:0/3423464834 >> 172.21.15.174:0/3423464834 conn(0x7f39680fad20 msgr2=0x7f3968068810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:23.318 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.301+0000 7f396d573700 1 -- 172.21.15.174:0/3423464834 shutdown_connections 2024-02-19T18:53:23.319 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:23.301+0000 7f396d573700 1 -- 172.21.15.174:0/3423464834 wait complete. 2024-02-19T18:53:23.319 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:53:23.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:23 smithi114 ceph-mon[29441]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T18:53:23.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:23 smithi114 ceph-mon[29441]: Deploying daemon grafana.smithi114 on smithi114 2024-02-19T18:53:23.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:23 smithi114 ceph-mon[29441]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:23.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:23 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:53:23.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:23 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/3423464834' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:24.906 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:24.907 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:25.109 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:25.154 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:25.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:25 smithi114 ceph-mon[29441]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:26.442 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.425+0000 7f73d6660700 1 -- 172.21.15.174:0/1065164487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73d01008b0 msgr2=0x7f73d0100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:26.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.425+0000 7f73d6660700 1 --2- 172.21.15.174:0/1065164487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73d01008b0 0x7f73d0100cd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f73c0009910 tx=0x7f73c0009c20 comp rx=0 tx=0).stop 2024-02-19T18:53:26.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.425+0000 7f73d6660700 1 -- 172.21.15.174:0/1065164487 shutdown_connections 2024-02-19T18:53:26.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.426+0000 7f73d6660700 1 --2- 172.21.15.174:0/1065164487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73d01008b0 0x7f73d0100cd0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:26.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.426+0000 7f73d6660700 1 -- 172.21.15.174:0/1065164487 >> 172.21.15.174:0/1065164487 conn(0x7f73d00fbe30 msgr2=0x7f73d00fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:26.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.426+0000 7f73d6660700 1 -- 172.21.15.174:0/1065164487 shutdown_connections 2024-02-19T18:53:26.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.426+0000 7f73d6660700 1 -- 172.21.15.174:0/1065164487 wait complete. 2024-02-19T18:53:26.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.426+0000 7f73d6660700 1 Processor -- start 2024-02-19T18:53:26.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.426+0000 7f73d6660700 1 -- start start 2024-02-19T18:53:26.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.426+0000 7f73d6660700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73d01008b0 0x7f73d01959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:26.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.426+0000 7f73d6660700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d0195f20 con 0x7f73d01008b0 2024-02-19T18:53:26.445 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.427+0000 7f73cffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73d01008b0 0x7f73d01959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:26.445 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.427+0000 7f73cffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73d01008b0 0x7f73d01959e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:59204/0 (socket says 172.21.15.174:59204) 2024-02-19T18:53:26.445 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.427+0000 7f73cffff700 1 -- 172.21.15.174:0/4148803150 learned_addr learned my addr 172.21.15.174:0/4148803150 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:53:26.445 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.427+0000 7f73cffff700 1 -- 172.21.15.174:0/4148803150 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c00095f0 con 0x7f73d01008b0 2024-02-19T18:53:26.445 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.427+0000 7f73cffff700 1 --2- 172.21.15.174:0/4148803150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73d01008b0 0x7f73d01959e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f73c0005b40 tx=0x7f73c0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:26.445 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.427+0000 7f73cd7fa700 1 -- 172.21.15.174:0/4148803150 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73c001d070 con 0x7f73d01008b0 2024-02-19T18:53:26.446 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.427+0000 7f73d6660700 1 -- 172.21.15.174:0/4148803150 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d0196120 con 0x7f73d01008b0 2024-02-19T18:53:26.446 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.427+0000 7f73cd7fa700 1 -- 172.21.15.174:0/4148803150 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f73c000be10 con 0x7f73d01008b0 2024-02-19T18:53:26.446 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.427+0000 7f73cd7fa700 1 -- 172.21.15.174:0/4148803150 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73c000f460 con 0x7f73d01008b0 2024-02-19T18:53:26.446 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.428+0000 7f73d6660700 1 -- 172.21.15.174:0/4148803150 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d01965c0 con 0x7f73d01008b0 2024-02-19T18:53:26.446 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.428+0000 7f73cd7fa700 1 -- 172.21.15.174:0/4148803150 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f73c000b960 con 0x7f73d01008b0 2024-02-19T18:53:26.446 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.428+0000 7f73d6660700 1 -- 172.21.15.174:0/4148803150 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d018f4d0 con 0x7f73d01008b0 2024-02-19T18:53:26.446 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.429+0000 7f73cd7fa700 1 --2- 172.21.15.174:0/4148803150 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f73b80383c0 0x7f73b803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:26.446 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.429+0000 7f73cd7fa700 1 -- 172.21.15.174:0/4148803150 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f73c004c480 con 0x7f73d01008b0 2024-02-19T18:53:26.446 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.429+0000 7f73cf7fe700 1 --2- 172.21.15.174:0/4148803150 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f73b80383c0 0x7f73b803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:26.446 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.429+0000 7f73cf7fe700 1 --2- 172.21.15.174:0/4148803150 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f73b80383c0 0x7f73b803a880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f73c4006fd0 tx=0x7f73c4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:26.450 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.434+0000 7f73cd7fa700 1 -- 172.21.15.174:0/4148803150 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f73c002a360 con 0x7f73d01008b0 2024-02-19T18:53:26.712 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.696+0000 7f73d6660700 1 -- 172.21.15.174:0/4148803150 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f73d002cc70 con 0x7f73d01008b0 2024-02-19T18:53:26.713 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.697+0000 7f73cd7fa700 1 -- 172.21.15.174:0/4148803150 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f73c0027030 con 0x7f73d01008b0 2024-02-19T18:53:26.714 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:53:26.714 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:26.715 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.699+0000 7f73d6660700 1 -- 172.21.15.174:0/4148803150 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f73b80383c0 msgr2=0x7f73b803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:26.715 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.699+0000 7f73d6660700 1 --2- 172.21.15.174:0/4148803150 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f73b80383c0 0x7f73b803a880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f73c4006fd0 tx=0x7f73c4006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:26.716 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.699+0000 7f73d6660700 1 -- 172.21.15.174:0/4148803150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73d01008b0 msgr2=0x7f73d01959e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:26.716 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.699+0000 7f73d6660700 1 --2- 172.21.15.174:0/4148803150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73d01008b0 0x7f73d01959e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f73c0005b40 tx=0x7f73c0004740 comp rx=0 tx=0).stop 2024-02-19T18:53:26.716 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.699+0000 7f73d6660700 1 -- 172.21.15.174:0/4148803150 shutdown_connections 2024-02-19T18:53:26.716 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.699+0000 7f73d6660700 1 --2- 172.21.15.174:0/4148803150 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f73b80383c0 0x7f73b803a880 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:26.716 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.699+0000 7f73d6660700 1 --2- 172.21.15.174:0/4148803150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73d01008b0 0x7f73d01959e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:26.716 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.699+0000 7f73d6660700 1 -- 172.21.15.174:0/4148803150 >> 172.21.15.174:0/4148803150 conn(0x7f73d00fbe30 msgr2=0x7f73d00fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:26.716 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.699+0000 7f73d6660700 1 -- 172.21.15.174:0/4148803150 shutdown_connections 2024-02-19T18:53:26.716 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:26.699+0000 7f73d6660700 1 -- 172.21.15.174:0/4148803150 wait complete. 2024-02-19T18:53:26.717 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:53:27.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:27 smithi114 ceph-mon[29441]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:27.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:27 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/4148803150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:28.282 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:28.282 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:28.487 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:28.532 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:29.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:28 smithi114 ceph-mon[29441]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:29.918 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.901+0000 7f103af63700 1 -- 172.21.15.174:0/3006865077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10341008b0 msgr2=0x7f1034100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:29.918 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.901+0000 7f103af63700 1 --2- 172.21.15.174:0/3006865077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10341008b0 0x7f1034100cd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1028009910 tx=0x7f1028009c20 comp rx=0 tx=0).stop 2024-02-19T18:53:29.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.901+0000 7f103af63700 1 -- 172.21.15.174:0/3006865077 shutdown_connections 2024-02-19T18:53:29.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.901+0000 7f103af63700 1 --2- 172.21.15.174:0/3006865077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10341008b0 0x7f1034100cd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:29.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.901+0000 7f103af63700 1 -- 172.21.15.174:0/3006865077 >> 172.21.15.174:0/3006865077 conn(0x7f10340fbe30 msgr2=0x7f10340fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:29.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.901+0000 7f103af63700 1 -- 172.21.15.174:0/3006865077 shutdown_connections 2024-02-19T18:53:29.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.901+0000 7f103af63700 1 -- 172.21.15.174:0/3006865077 wait complete. 2024-02-19T18:53:29.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.901+0000 7f103af63700 1 Processor -- start 2024-02-19T18:53:29.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.901+0000 7f103af63700 1 -- start start 2024-02-19T18:53:29.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.901+0000 7f103af63700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10341008b0 0x7f10341959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:29.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.901+0000 7f103af63700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1034195f00 con 0x7f10341008b0 2024-02-19T18:53:29.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.902+0000 7f1038cff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10341008b0 0x7f10341959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:29.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.902+0000 7f1038cff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10341008b0 0x7f10341959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:59226/0 (socket says 172.21.15.174:59226) 2024-02-19T18:53:29.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.902+0000 7f1038cff700 1 -- 172.21.15.174:0/1602275725 learned_addr learned my addr 172.21.15.174:0/1602275725 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:53:29.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.902+0000 7f1038cff700 1 -- 172.21.15.174:0/1602275725 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10280095f0 con 0x7f10341008b0 2024-02-19T18:53:29.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.902+0000 7f1038cff700 1 --2- 172.21.15.174:0/1602275725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10341008b0 0x7f10341959c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1028005b40 tx=0x7f1028004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:29.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.902+0000 7f1031ffb700 1 -- 172.21.15.174:0/1602275725 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f102801d070 con 0x7f10341008b0 2024-02-19T18:53:29.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.902+0000 7f1031ffb700 1 -- 172.21.15.174:0/1602275725 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f102800be10 con 0x7f10341008b0 2024-02-19T18:53:29.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.902+0000 7f103af63700 1 -- 172.21.15.174:0/1602275725 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1034196100 con 0x7f10341008b0 2024-02-19T18:53:29.920 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.902+0000 7f1031ffb700 1 -- 172.21.15.174:0/1602275725 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f102800f460 con 0x7f10341008b0 2024-02-19T18:53:29.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.903+0000 7f103af63700 1 -- 172.21.15.174:0/1602275725 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10341965a0 con 0x7f10341008b0 2024-02-19T18:53:29.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.903+0000 7f103af63700 1 -- 172.21.15.174:0/1602275725 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f103418f4d0 con 0x7f10341008b0 2024-02-19T18:53:29.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.904+0000 7f1031ffb700 1 -- 172.21.15.174:0/1602275725 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f10280227e0 con 0x7f10341008b0 2024-02-19T18:53:29.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.904+0000 7f1031ffb700 1 --2- 172.21.15.174:0/1602275725 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f10240383c0 0x7f102403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:29.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.904+0000 7f1031ffb700 1 -- 172.21.15.174:0/1602275725 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f102804c470 con 0x7f10341008b0 2024-02-19T18:53:29.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.904+0000 7f1033fff700 1 --2- 172.21.15.174:0/1602275725 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f10240383c0 0x7f102403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:29.921 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.904+0000 7f1033fff700 1 --2- 172.21.15.174:0/1602275725 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f10240383c0 0x7f102403a880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1020006fd0 tx=0x7f1020006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:29.925 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:29.909+0000 7f1031ffb700 1 -- 172.21.15.174:0/1602275725 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f102802bba0 con 0x7f10341008b0 2024-02-19T18:53:30.192 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:30.176+0000 7f103af63700 1 -- 172.21.15.174:0/1602275725 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f103402cc70 con 0x7f10341008b0 2024-02-19T18:53:30.193 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:30.176+0000 7f1031ffb700 1 -- 172.21.15.174:0/1602275725 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f1028027020 con 0x7f10341008b0 2024-02-19T18:53:30.193 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:53:30.193 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:30.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:30.178+0000 7f103af63700 1 -- 172.21.15.174:0/1602275725 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f10240383c0 msgr2=0x7f102403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:30.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:30.178+0000 7f103af63700 1 --2- 172.21.15.174:0/1602275725 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f10240383c0 0x7f102403a880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1020006fd0 tx=0x7f1020006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:30.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:30.178+0000 7f103af63700 1 -- 172.21.15.174:0/1602275725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10341008b0 msgr2=0x7f10341959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:30.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:30.178+0000 7f103af63700 1 --2- 172.21.15.174:0/1602275725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10341008b0 0x7f10341959c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1028005b40 tx=0x7f1028004740 comp rx=0 tx=0).stop 2024-02-19T18:53:30.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:30.178+0000 7f103af63700 1 -- 172.21.15.174:0/1602275725 shutdown_connections 2024-02-19T18:53:30.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:30.178+0000 7f103af63700 1 --2- 172.21.15.174:0/1602275725 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f10240383c0 0x7f102403a880 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:30.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:30.178+0000 7f103af63700 1 --2- 172.21.15.174:0/1602275725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10341008b0 0x7f10341959c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:30.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:30.178+0000 7f103af63700 1 -- 172.21.15.174:0/1602275725 >> 172.21.15.174:0/1602275725 conn(0x7f10340fbe30 msgr2=0x7f10340fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:30.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:30.179+0000 7f103af63700 1 -- 172.21.15.174:0/1602275725 shutdown_connections 2024-02-19T18:53:30.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:30.179+0000 7f103af63700 1 -- 172.21.15.174:0/1602275725 wait complete. 2024-02-19T18:53:30.196 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:53:31.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:31 smithi114 ceph-mon[29441]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:31.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:31 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/1602275725' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:31.694 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:31.694 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:31.896 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:31.943 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:33.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.227+0000 7fb84a51f700 1 -- 172.21.15.174:0/2848124206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8441008b0 msgr2=0x7fb844100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:33.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.227+0000 7fb84a51f700 1 --2- 172.21.15.174:0/2848124206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8441008b0 0x7fb844100cd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb834009910 tx=0x7fb834009c20 comp rx=0 tx=0).stop 2024-02-19T18:53:33.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.227+0000 7fb84a51f700 1 -- 172.21.15.174:0/2848124206 shutdown_connections 2024-02-19T18:53:33.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.227+0000 7fb84a51f700 1 --2- 172.21.15.174:0/2848124206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8441008b0 0x7fb844100cd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:33.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.227+0000 7fb84a51f700 1 -- 172.21.15.174:0/2848124206 >> 172.21.15.174:0/2848124206 conn(0x7fb8440fbe30 msgr2=0x7fb8440fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:33.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.227+0000 7fb84a51f700 1 -- 172.21.15.174:0/2848124206 shutdown_connections 2024-02-19T18:53:33.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.227+0000 7fb84a51f700 1 -- 172.21.15.174:0/2848124206 wait complete. 2024-02-19T18:53:33.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.228+0000 7fb84a51f700 1 Processor -- start 2024-02-19T18:53:33.245 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.228+0000 7fb84a51f700 1 -- start start 2024-02-19T18:53:33.246 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.228+0000 7fb84a51f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8441008b0 0x7fb8441959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:33.246 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.228+0000 7fb84a51f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb844195f00 con 0x7fb8441008b0 2024-02-19T18:53:33.246 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.228+0000 7fb843fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8441008b0 0x7fb8441959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:33.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.228+0000 7fb843fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8441008b0 0x7fb8441959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:59236/0 (socket says 172.21.15.174:59236) 2024-02-19T18:53:33.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.228+0000 7fb843fff700 1 -- 172.21.15.174:0/2703336860 learned_addr learned my addr 172.21.15.174:0/2703336860 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:53:33.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.229+0000 7fb843fff700 1 -- 172.21.15.174:0/2703336860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8340095f0 con 0x7fb8441008b0 2024-02-19T18:53:33.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.229+0000 7fb843fff700 1 --2- 172.21.15.174:0/2703336860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8441008b0 0x7fb8441959c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb834005b40 tx=0x7fb834004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:33.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.229+0000 7fb8417fa700 1 -- 172.21.15.174:0/2703336860 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb83401d070 con 0x7fb8441008b0 2024-02-19T18:53:33.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.229+0000 7fb84a51f700 1 -- 172.21.15.174:0/2703336860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb844196100 con 0x7fb8441008b0 2024-02-19T18:53:33.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.229+0000 7fb8417fa700 1 -- 172.21.15.174:0/2703336860 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb83400be10 con 0x7fb8441008b0 2024-02-19T18:53:33.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.229+0000 7fb8417fa700 1 -- 172.21.15.174:0/2703336860 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb83400f460 con 0x7fb8441008b0 2024-02-19T18:53:33.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.229+0000 7fb84a51f700 1 -- 172.21.15.174:0/2703336860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8441965a0 con 0x7fb8441008b0 2024-02-19T18:53:33.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.230+0000 7fb8417fa700 1 -- 172.21.15.174:0/2703336860 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fb83400b960 con 0x7fb8441008b0 2024-02-19T18:53:33.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.230+0000 7fb84a51f700 1 -- 172.21.15.174:0/2703336860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb84418f4d0 con 0x7fb8441008b0 2024-02-19T18:53:33.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.230+0000 7fb8417fa700 1 --2- 172.21.15.174:0/2703336860 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb82c0383c0 0x7fb82c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:33.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.230+0000 7fb8417fa700 1 -- 172.21.15.174:0/2703336860 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb83404c3c0 con 0x7fb8441008b0 2024-02-19T18:53:33.248 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.230+0000 7fb8437fe700 1 --2- 172.21.15.174:0/2703336860 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb82c0383c0 0x7fb82c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:33.248 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.231+0000 7fb8437fe700 1 --2- 172.21.15.174:0/2703336860 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb82c0383c0 0x7fb82c03a880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb838006fd0 tx=0x7fb838006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:33.251 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.235+0000 7fb8417fa700 1 -- 172.21.15.174:0/2703336860 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb83402a360 con 0x7fb8441008b0 2024-02-19T18:53:33.515 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.499+0000 7fb84a51f700 1 -- 172.21.15.174:0/2703336860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb84402cc70 con 0x7fb8441008b0 2024-02-19T18:53:33.516 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.500+0000 7fb8417fa700 1 -- 172.21.15.174:0/2703336860 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fb834027030 con 0x7fb8441008b0 2024-02-19T18:53:33.516 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:53:33.516 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:33.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.502+0000 7fb84a51f700 1 -- 172.21.15.174:0/2703336860 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb82c0383c0 msgr2=0x7fb82c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:33.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.502+0000 7fb84a51f700 1 --2- 172.21.15.174:0/2703336860 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb82c0383c0 0x7fb82c03a880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb838006fd0 tx=0x7fb838006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:33.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.502+0000 7fb84a51f700 1 -- 172.21.15.174:0/2703336860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8441008b0 msgr2=0x7fb8441959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:33.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.502+0000 7fb84a51f700 1 --2- 172.21.15.174:0/2703336860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8441008b0 0x7fb8441959c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb834005b40 tx=0x7fb834004740 comp rx=0 tx=0).stop 2024-02-19T18:53:33.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.502+0000 7fb84a51f700 1 -- 172.21.15.174:0/2703336860 shutdown_connections 2024-02-19T18:53:33.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.502+0000 7fb84a51f700 1 --2- 172.21.15.174:0/2703336860 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb82c0383c0 0x7fb82c03a880 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:33.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.502+0000 7fb84a51f700 1 --2- 172.21.15.174:0/2703336860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb8441008b0 0x7fb8441959c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:33.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.502+0000 7fb84a51f700 1 -- 172.21.15.174:0/2703336860 >> 172.21.15.174:0/2703336860 conn(0x7fb8440fbe30 msgr2=0x7fb8440fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:33.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.502+0000 7fb84a51f700 1 -- 172.21.15.174:0/2703336860 shutdown_connections 2024-02-19T18:53:33.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:33.502+0000 7fb84a51f700 1 -- 172.21.15.174:0/2703336860 wait complete. 2024-02-19T18:53:33.520 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:53:33.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:33 smithi114 ceph-mon[29441]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:34.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:34 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/2703336860' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:35.096 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:35.097 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:35.298 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:35.343 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:35.483 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:35 smithi114 ceph-mon[29441]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:36.811 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.794+0000 7f1f5565e700 1 -- 172.21.15.174:0/4080410227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f50100840 msgr2=0x7f1f50100c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:36.811 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.794+0000 7f1f5565e700 1 --2- 172.21.15.174:0/4080410227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f50100840 0x7f1f50100c60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009910 tx=0x7f1f40009c20 comp rx=0 tx=0).stop 2024-02-19T18:53:36.811 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.794+0000 7f1f5565e700 1 -- 172.21.15.174:0/4080410227 shutdown_connections 2024-02-19T18:53:36.812 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.794+0000 7f1f5565e700 1 --2- 172.21.15.174:0/4080410227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f50100840 0x7f1f50100c60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:36.812 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.794+0000 7f1f5565e700 1 -- 172.21.15.174:0/4080410227 >> 172.21.15.174:0/4080410227 conn(0x7f1f500fbde0 msgr2=0x7f1f500fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:36.812 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.794+0000 7f1f5565e700 1 -- 172.21.15.174:0/4080410227 shutdown_connections 2024-02-19T18:53:36.812 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.794+0000 7f1f5565e700 1 -- 172.21.15.174:0/4080410227 wait complete. 2024-02-19T18:53:36.812 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.795+0000 7f1f5565e700 1 Processor -- start 2024-02-19T18:53:36.812 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.795+0000 7f1f5565e700 1 -- start start 2024-02-19T18:53:36.812 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.795+0000 7f1f5565e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f50100840 0x7f1f50074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:36.812 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.795+0000 7f1f5565e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f50075070 con 0x7f1f50100840 2024-02-19T18:53:36.812 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.795+0000 7f1f4effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f50100840 0x7f1f50074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:36.812 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.795+0000 7f1f4effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f50100840 0x7f1f50074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:55196/0 (socket says 172.21.15.174:55196) 2024-02-19T18:53:36.812 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.795+0000 7f1f4effd700 1 -- 172.21.15.174:0/414420980 learned_addr learned my addr 172.21.15.174:0/414420980 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:53:36.814 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.795+0000 7f1f4effd700 1 -- 172.21.15.174:0/414420980 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f400095f0 con 0x7f1f50100840 2024-02-19T18:53:36.814 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.796+0000 7f1f4effd700 1 --2- 172.21.15.174:0/414420980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f50100840 0x7f1f50074b30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1f40005b40 tx=0x7f1f40004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:36.814 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.796+0000 7f1f4cff9700 1 -- 172.21.15.174:0/414420980 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f4001d070 con 0x7f1f50100840 2024-02-19T18:53:36.814 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.796+0000 7f1f4cff9700 1 -- 172.21.15.174:0/414420980 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1f4000be10 con 0x7f1f50100840 2024-02-19T18:53:36.814 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.796+0000 7f1f5565e700 1 -- 172.21.15.174:0/414420980 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f50073180 con 0x7f1f50100840 2024-02-19T18:53:36.814 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.796+0000 7f1f4cff9700 1 -- 172.21.15.174:0/414420980 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f4000f460 con 0x7f1f50100840 2024-02-19T18:53:36.814 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.796+0000 7f1f5565e700 1 -- 172.21.15.174:0/414420980 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f50073620 con 0x7f1f50100840 2024-02-19T18:53:36.814 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.797+0000 7f1f4cff9700 1 -- 172.21.15.174:0/414420980 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f1f4000b960 con 0x7f1f50100840 2024-02-19T18:53:36.815 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.797+0000 7f1f5565e700 1 -- 172.21.15.174:0/414420980 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f500623c0 con 0x7f1f50100840 2024-02-19T18:53:36.815 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.797+0000 7f1f4cff9700 1 --2- 172.21.15.174:0/414420980 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f1f3c0383d0 0x7f1f3c03a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:36.815 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.797+0000 7f1f4cff9700 1 -- 172.21.15.174:0/414420980 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1f4004c3c0 con 0x7f1f50100840 2024-02-19T18:53:36.815 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.797+0000 7f1f467fc700 1 --2- 172.21.15.174:0/414420980 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f1f3c0383d0 0x7f1f3c03a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:36.815 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.798+0000 7f1f467fc700 1 --2- 172.21.15.174:0/414420980 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f1f3c0383d0 0x7f1f3c03a890 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1f38006fd0 tx=0x7f1f38006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:36.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:36.802+0000 7f1f4cff9700 1 -- 172.21.15.174:0/414420980 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1f4002a360 con 0x7f1f50100840 2024-02-19T18:53:37.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:37.067+0000 7f1f5565e700 1 -- 172.21.15.174:0/414420980 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f5018ee40 con 0x7f1f50100840 2024-02-19T18:53:37.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:37.068+0000 7f1f4cff9700 1 -- 172.21.15.174:0/414420980 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f1f40027030 con 0x7f1f50100840 2024-02-19T18:53:37.084 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:53:37.084 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:37.070+0000 7f1f5565e700 1 -- 172.21.15.174:0/414420980 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f1f3c0383d0 msgr2=0x7f1f3c03a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:37.070+0000 7f1f5565e700 1 --2- 172.21.15.174:0/414420980 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f1f3c0383d0 0x7f1f3c03a890 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1f38006fd0 tx=0x7f1f38006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:37.070+0000 7f1f5565e700 1 -- 172.21.15.174:0/414420980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f50100840 msgr2=0x7f1f50074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:37.070+0000 7f1f5565e700 1 --2- 172.21.15.174:0/414420980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f50100840 0x7f1f50074b30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1f40005b40 tx=0x7f1f40004740 comp rx=0 tx=0).stop 2024-02-19T18:53:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:37.070+0000 7f1f5565e700 1 -- 172.21.15.174:0/414420980 shutdown_connections 2024-02-19T18:53:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:37.070+0000 7f1f5565e700 1 --2- 172.21.15.174:0/414420980 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f1f3c0383d0 0x7f1f3c03a890 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:37.070+0000 7f1f5565e700 1 --2- 172.21.15.174:0/414420980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f50100840 0x7f1f50074b30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:37.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:37.070+0000 7f1f5565e700 1 -- 172.21.15.174:0/414420980 >> 172.21.15.174:0/414420980 conn(0x7f1f500fbde0 msgr2=0x7f1f500fcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:37.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:37.070+0000 7f1f5565e700 1 -- 172.21.15.174:0/414420980 shutdown_connections 2024-02-19T18:53:37.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:37.070+0000 7f1f5565e700 1 -- 172.21.15.174:0/414420980 wait complete. 2024-02-19T18:53:37.087 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:53:37.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:37 smithi114 ceph-mon[29441]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:37.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:37 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/414420980' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:38.624 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:38.624 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:38.824 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:38.871 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:39.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:38 smithi114 ceph-mon[29441]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:40.335 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.319+0000 7f4e389ca700 1 -- 172.21.15.174:0/563929099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e340ff7a0 msgr2=0x7f4e340ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:40.335 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.319+0000 7f4e389ca700 1 --2- 172.21.15.174:0/563929099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e340ff7a0 0x7f4e340ffbc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009910 tx=0x7f4e24009c20 comp rx=0 tx=0).stop 2024-02-19T18:53:40.336 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.319+0000 7f4e389ca700 1 -- 172.21.15.174:0/563929099 shutdown_connections 2024-02-19T18:53:40.336 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.319+0000 7f4e389ca700 1 --2- 172.21.15.174:0/563929099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e340ff7a0 0x7f4e340ffbc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:40.336 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.319+0000 7f4e389ca700 1 -- 172.21.15.174:0/563929099 >> 172.21.15.174:0/563929099 conn(0x7f4e340fad20 msgr2=0x7f4e340fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:40.336 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.319+0000 7f4e389ca700 1 -- 172.21.15.174:0/563929099 shutdown_connections 2024-02-19T18:53:40.336 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.319+0000 7f4e389ca700 1 -- 172.21.15.174:0/563929099 wait complete. 2024-02-19T18:53:40.337 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.320+0000 7f4e389ca700 1 Processor -- start 2024-02-19T18:53:40.337 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.320+0000 7f4e389ca700 1 -- start start 2024-02-19T18:53:40.338 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.320+0000 7f4e389ca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e340ff7a0 0x7f4e341959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:40.338 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.320+0000 7f4e389ca700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e34195f00 con 0x7f4e340ff7a0 2024-02-19T18:53:40.338 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.320+0000 7f4e3259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e340ff7a0 0x7f4e341959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:40.338 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.320+0000 7f4e3259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e340ff7a0 0x7f4e341959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:55220/0 (socket says 172.21.15.174:55220) 2024-02-19T18:53:40.338 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.320+0000 7f4e3259c700 1 -- 172.21.15.174:0/2075704612 learned_addr learned my addr 172.21.15.174:0/2075704612 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:53:40.338 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.320+0000 7f4e3259c700 1 -- 172.21.15.174:0/2075704612 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e240095f0 con 0x7f4e340ff7a0 2024-02-19T18:53:40.338 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.320+0000 7f4e3259c700 1 --2- 172.21.15.174:0/2075704612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e340ff7a0 0x7f4e341959c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4e24005b40 tx=0x7f4e24004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:40.338 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.321+0000 7f4e237fe700 1 -- 172.21.15.174:0/2075704612 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e2401d070 con 0x7f4e340ff7a0 2024-02-19T18:53:40.338 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.321+0000 7f4e237fe700 1 -- 172.21.15.174:0/2075704612 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4e2400be10 con 0x7f4e340ff7a0 2024-02-19T18:53:40.338 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.321+0000 7f4e389ca700 1 -- 172.21.15.174:0/2075704612 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e34196100 con 0x7f4e340ff7a0 2024-02-19T18:53:40.338 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.321+0000 7f4e237fe700 1 -- 172.21.15.174:0/2075704612 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e2400f460 con 0x7f4e340ff7a0 2024-02-19T18:53:40.338 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.321+0000 7f4e389ca700 1 -- 172.21.15.174:0/2075704612 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e341965a0 con 0x7f4e340ff7a0 2024-02-19T18:53:40.339 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.321+0000 7f4e389ca700 1 -- 172.21.15.174:0/2075704612 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e3418f3d0 con 0x7f4e340ff7a0 2024-02-19T18:53:40.339 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.322+0000 7f4e237fe700 1 -- 172.21.15.174:0/2075704612 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f4e240227e0 con 0x7f4e340ff7a0 2024-02-19T18:53:40.339 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.322+0000 7f4e237fe700 1 --2- 172.21.15.174:0/2075704612 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f4e1c0383c0 0x7f4e1c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:40.340 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.322+0000 7f4e237fe700 1 -- 172.21.15.174:0/2075704612 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4e2404c560 con 0x7f4e340ff7a0 2024-02-19T18:53:40.340 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.322+0000 7f4e31d9b700 1 --2- 172.21.15.174:0/2075704612 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f4e1c0383c0 0x7f4e1c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:40.340 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.322+0000 7f4e31d9b700 1 --2- 172.21.15.174:0/2075704612 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f4e1c0383c0 0x7f4e1c03a880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4e28006fd0 tx=0x7f4e28006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:40.343 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.327+0000 7f4e237fe700 1 -- 172.21.15.174:0/2075704612 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4e2402bba0 con 0x7f4e340ff7a0 2024-02-19T18:53:40.605 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.588+0000 7f4e389ca700 1 -- 172.21.15.174:0/2075704612 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e3402cc70 con 0x7f4e340ff7a0 2024-02-19T18:53:40.605 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.589+0000 7f4e237fe700 1 -- 172.21.15.174:0/2075704612 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f4e24027020 con 0x7f4e340ff7a0 2024-02-19T18:53:40.605 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:53:40.605 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:40.607 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.591+0000 7f4e389ca700 1 -- 172.21.15.174:0/2075704612 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f4e1c0383c0 msgr2=0x7f4e1c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:40.607 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.591+0000 7f4e389ca700 1 --2- 172.21.15.174:0/2075704612 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f4e1c0383c0 0x7f4e1c03a880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4e28006fd0 tx=0x7f4e28006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:40.607 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.591+0000 7f4e389ca700 1 -- 172.21.15.174:0/2075704612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e340ff7a0 msgr2=0x7f4e341959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:40.608 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.591+0000 7f4e389ca700 1 --2- 172.21.15.174:0/2075704612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e340ff7a0 0x7f4e341959c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4e24005b40 tx=0x7f4e24004740 comp rx=0 tx=0).stop 2024-02-19T18:53:40.608 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.591+0000 7f4e389ca700 1 -- 172.21.15.174:0/2075704612 shutdown_connections 2024-02-19T18:53:40.608 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.591+0000 7f4e389ca700 1 --2- 172.21.15.174:0/2075704612 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f4e1c0383c0 0x7f4e1c03a880 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:40.608 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.591+0000 7f4e389ca700 1 --2- 172.21.15.174:0/2075704612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e340ff7a0 0x7f4e341959c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:40.608 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.591+0000 7f4e389ca700 1 -- 172.21.15.174:0/2075704612 >> 172.21.15.174:0/2075704612 conn(0x7f4e340fad20 msgr2=0x7f4e34069100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:40.608 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.591+0000 7f4e389ca700 1 -- 172.21.15.174:0/2075704612 shutdown_connections 2024-02-19T18:53:40.608 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:40.591+0000 7f4e389ca700 1 -- 172.21.15.174:0/2075704612 wait complete. 2024-02-19T18:53:40.608 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:53:41.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:41 smithi114 ceph-mon[29441]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:41.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:41 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/2075704612' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:42.177 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:42.177 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:42.379 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:42.424 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:43.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:43 smithi114 ceph-mon[29441]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:43.929 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.910+0000 7f8d18da4700 1 -- 172.21.15.174:0/636975430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d140ff7a0 msgr2=0x7f8d140ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:43.929 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.910+0000 7f8d18da4700 1 --2- 172.21.15.174:0/636975430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d140ff7a0 0x7f8d140ffbc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009910 tx=0x7f8d08009c20 comp rx=0 tx=0).stop 2024-02-19T18:53:43.929 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.910+0000 7f8d18da4700 1 -- 172.21.15.174:0/636975430 shutdown_connections 2024-02-19T18:53:43.929 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.910+0000 7f8d18da4700 1 --2- 172.21.15.174:0/636975430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d140ff7a0 0x7f8d140ffbc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:43.929 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.910+0000 7f8d18da4700 1 -- 172.21.15.174:0/636975430 >> 172.21.15.174:0/636975430 conn(0x7f8d140fad20 msgr2=0x7f8d140fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:43.929 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.910+0000 7f8d18da4700 1 -- 172.21.15.174:0/636975430 shutdown_connections 2024-02-19T18:53:43.929 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.910+0000 7f8d18da4700 1 -- 172.21.15.174:0/636975430 wait complete. 2024-02-19T18:53:43.929 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.911+0000 7f8d18da4700 1 Processor -- start 2024-02-19T18:53:43.929 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.911+0000 7f8d18da4700 1 -- start start 2024-02-19T18:53:43.929 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.911+0000 7f8d18da4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d140ff7a0 0x7f8d140fcce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:43.930 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.911+0000 7f8d18da4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d140fd220 con 0x7f8d140ff7a0 2024-02-19T18:53:43.930 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.912+0000 7f8d1259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d140ff7a0 0x7f8d140fcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:43.930 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.912+0000 7f8d1259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d140ff7a0 0x7f8d140fcce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:55230/0 (socket says 172.21.15.174:55230) 2024-02-19T18:53:43.930 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.912+0000 7f8d1259c700 1 -- 172.21.15.174:0/3134073323 learned_addr learned my addr 172.21.15.174:0/3134073323 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:53:43.930 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.912+0000 7f8d1259c700 1 -- 172.21.15.174:0/3134073323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d080095f0 con 0x7f8d140ff7a0 2024-02-19T18:53:43.930 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.912+0000 7f8d1259c700 1 --2- 172.21.15.174:0/3134073323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d140ff7a0 0x7f8d140fcce0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8d08005b40 tx=0x7f8d08004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:43.930 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.912+0000 7f8d037fe700 1 -- 172.21.15.174:0/3134073323 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d0801d070 con 0x7f8d140ff7a0 2024-02-19T18:53:43.930 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.912+0000 7f8d18da4700 1 -- 172.21.15.174:0/3134073323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d140fb330 con 0x7f8d140ff7a0 2024-02-19T18:53:43.930 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.912+0000 7f8d037fe700 1 -- 172.21.15.174:0/3134073323 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8d0800be10 con 0x7f8d140ff7a0 2024-02-19T18:53:43.930 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.912+0000 7f8d037fe700 1 -- 172.21.15.174:0/3134073323 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d0800f460 con 0x7f8d140ff7a0 2024-02-19T18:53:43.930 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.912+0000 7f8d18da4700 1 -- 172.21.15.174:0/3134073323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d140fb7d0 con 0x7f8d140ff7a0 2024-02-19T18:53:43.931 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.913+0000 7f8d18da4700 1 -- 172.21.15.174:0/3134073323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d140623c0 con 0x7f8d140ff7a0 2024-02-19T18:53:43.931 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.913+0000 7f8d037fe700 1 -- 172.21.15.174:0/3134073323 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f8d080227e0 con 0x7f8d140ff7a0 2024-02-19T18:53:43.931 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.913+0000 7f8d037fe700 1 --2- 172.21.15.174:0/3134073323 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f8cfc0383c0 0x7f8cfc03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:43.931 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.913+0000 7f8d037fe700 1 -- 172.21.15.174:0/3134073323 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8d0804c560 con 0x7f8d140ff7a0 2024-02-19T18:53:43.931 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.913+0000 7f8d11d9b700 1 --2- 172.21.15.174:0/3134073323 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f8cfc0383c0 0x7f8cfc03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:43.932 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.914+0000 7f8d11d9b700 1 --2- 172.21.15.174:0/3134073323 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f8cfc0383c0 0x7f8cfc03a880 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8d04006fd0 tx=0x7f8d04006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:43.934 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:43.918+0000 7f8d037fe700 1 -- 172.21.15.174:0/3134073323 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8d0802bba0 con 0x7f8d140ff7a0 2024-02-19T18:53:44.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:44.179+0000 7f8d18da4700 1 -- 172.21.15.174:0/3134073323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d1418edb0 con 0x7f8d140ff7a0 2024-02-19T18:53:44.196 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:44.179+0000 7f8d037fe700 1 -- 172.21.15.174:0/3134073323 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f8d08027020 con 0x7f8d140ff7a0 2024-02-19T18:53:44.196 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:53:44.196 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:44.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:44.181+0000 7f8d18da4700 1 -- 172.21.15.174:0/3134073323 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f8cfc0383c0 msgr2=0x7f8cfc03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:44.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:44.181+0000 7f8d18da4700 1 --2- 172.21.15.174:0/3134073323 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f8cfc0383c0 0x7f8cfc03a880 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8d04006fd0 tx=0x7f8d04006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:44.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:44.181+0000 7f8d18da4700 1 -- 172.21.15.174:0/3134073323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d140ff7a0 msgr2=0x7f8d140fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:44.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:44.181+0000 7f8d18da4700 1 --2- 172.21.15.174:0/3134073323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d140ff7a0 0x7f8d140fcce0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8d08005b40 tx=0x7f8d08004740 comp rx=0 tx=0).stop 2024-02-19T18:53:44.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:44.182+0000 7f8d18da4700 1 -- 172.21.15.174:0/3134073323 shutdown_connections 2024-02-19T18:53:44.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:44.182+0000 7f8d18da4700 1 --2- 172.21.15.174:0/3134073323 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f8cfc0383c0 0x7f8cfc03a880 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:44.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:44.182+0000 7f8d18da4700 1 --2- 172.21.15.174:0/3134073323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d140ff7a0 0x7f8d140fcce0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:44.199 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:44.182+0000 7f8d18da4700 1 -- 172.21.15.174:0/3134073323 >> 172.21.15.174:0/3134073323 conn(0x7f8d140fad20 msgr2=0x7f8d14068810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:44.199 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:44.182+0000 7f8d18da4700 1 -- 172.21.15.174:0/3134073323 shutdown_connections 2024-02-19T18:53:44.199 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:44.182+0000 7f8d18da4700 1 -- 172.21.15.174:0/3134073323 wait complete. 2024-02-19T18:53:44.199 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:53:45.497 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:45 smithi114 ceph-mon[29441]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:45.497 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:45 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/3134073323' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:45.627 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:45.628 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:45.829 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:45.876 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:47.290 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.273+0000 7f271d64d700 1 -- 172.21.15.174:0/2406764835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27181008b0 msgr2=0x7f2718100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:47.290 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.273+0000 7f271d64d700 1 --2- 172.21.15.174:0/2406764835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27181008b0 0x7f2718100cd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2700009910 tx=0x7f2700009c20 comp rx=0 tx=0).stop 2024-02-19T18:53:47.290 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.273+0000 7f271d64d700 1 -- 172.21.15.174:0/2406764835 shutdown_connections 2024-02-19T18:53:47.290 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.273+0000 7f271d64d700 1 --2- 172.21.15.174:0/2406764835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27181008b0 0x7f2718100cd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:47.290 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.273+0000 7f271d64d700 1 -- 172.21.15.174:0/2406764835 >> 172.21.15.174:0/2406764835 conn(0x7f27180fbe30 msgr2=0x7f27180fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:47.291 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.273+0000 7f271d64d700 1 -- 172.21.15.174:0/2406764835 shutdown_connections 2024-02-19T18:53:47.291 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.273+0000 7f271d64d700 1 -- 172.21.15.174:0/2406764835 wait complete. 2024-02-19T18:53:47.293 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.274+0000 7f271d64d700 1 Processor -- start 2024-02-19T18:53:47.293 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.274+0000 7f271d64d700 1 -- start start 2024-02-19T18:53:47.293 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.274+0000 7f271d64d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27181008b0 0x7f27181959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:47.293 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.274+0000 7f271d64d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2718195f10 con 0x7f27181008b0 2024-02-19T18:53:47.293 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.274+0000 7f2716ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27181008b0 0x7f27181959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:47.293 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.275+0000 7f2716ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27181008b0 0x7f27181959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:50516/0 (socket says 172.21.15.174:50516) 2024-02-19T18:53:47.293 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.275+0000 7f2716ffd700 1 -- 172.21.15.174:0/2970540407 learned_addr learned my addr 172.21.15.174:0/2970540407 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:53:47.293 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.275+0000 7f2716ffd700 1 -- 172.21.15.174:0/2970540407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27000095f0 con 0x7f27181008b0 2024-02-19T18:53:47.293 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.275+0000 7f2716ffd700 1 --2- 172.21.15.174:0/2970540407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27181008b0 0x7f27181959d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2700005b40 tx=0x7f2700004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:47.293 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.275+0000 7f270ffff700 1 -- 172.21.15.174:0/2970540407 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f270001d070 con 0x7f27181008b0 2024-02-19T18:53:47.293 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.275+0000 7f271d64d700 1 -- 172.21.15.174:0/2970540407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2718196110 con 0x7f27181008b0 2024-02-19T18:53:47.294 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.275+0000 7f270ffff700 1 -- 172.21.15.174:0/2970540407 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f270000be10 con 0x7f27181008b0 2024-02-19T18:53:47.294 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.275+0000 7f270ffff700 1 -- 172.21.15.174:0/2970540407 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f270000f460 con 0x7f27181008b0 2024-02-19T18:53:47.294 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.275+0000 7f271d64d700 1 -- 172.21.15.174:0/2970540407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27181965b0 con 0x7f27181008b0 2024-02-19T18:53:47.294 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.276+0000 7f270ffff700 1 -- 172.21.15.174:0/2970540407 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f270000f6d0 con 0x7f27181008b0 2024-02-19T18:53:47.294 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.276+0000 7f271d64d700 1 -- 172.21.15.174:0/2970540407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f271818f4d0 con 0x7f27181008b0 2024-02-19T18:53:47.294 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.276+0000 7f270ffff700 1 --2- 172.21.15.174:0/2970540407 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f27040383c0 0x7f270403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:47.294 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.276+0000 7f270ffff700 1 -- 172.21.15.174:0/2970540407 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f270004c470 con 0x7f27181008b0 2024-02-19T18:53:47.294 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.277+0000 7f27167fc700 1 --2- 172.21.15.174:0/2970540407 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f27040383c0 0x7f270403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:47.294 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.277+0000 7f27167fc700 1 --2- 172.21.15.174:0/2970540407 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f27040383c0 0x7f270403a880 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2708006fd0 tx=0x7f2708006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:47.298 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.282+0000 7f270ffff700 1 -- 172.21.15.174:0/2970540407 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f270002a360 con 0x7f27181008b0 2024-02-19T18:53:47.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:47 smithi114 ceph-mon[29441]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:47.562 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.545+0000 7f271d64d700 1 -- 172.21.15.174:0/2970540407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f271802cc70 con 0x7f27181008b0 2024-02-19T18:53:47.563 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.546+0000 7f270ffff700 1 -- 172.21.15.174:0/2970540407 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f2700027030 con 0x7f27181008b0 2024-02-19T18:53:47.564 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:53:47.564 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:47.565 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.548+0000 7f271d64d700 1 -- 172.21.15.174:0/2970540407 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f27040383c0 msgr2=0x7f270403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:47.565 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.548+0000 7f271d64d700 1 --2- 172.21.15.174:0/2970540407 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f27040383c0 0x7f270403a880 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2708006fd0 tx=0x7f2708006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:47.565 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.548+0000 7f271d64d700 1 -- 172.21.15.174:0/2970540407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27181008b0 msgr2=0x7f27181959d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:47.566 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.548+0000 7f271d64d700 1 --2- 172.21.15.174:0/2970540407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27181008b0 0x7f27181959d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2700005b40 tx=0x7f2700004740 comp rx=0 tx=0).stop 2024-02-19T18:53:47.566 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.548+0000 7f271d64d700 1 -- 172.21.15.174:0/2970540407 shutdown_connections 2024-02-19T18:53:47.566 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.548+0000 7f271d64d700 1 --2- 172.21.15.174:0/2970540407 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f27040383c0 0x7f270403a880 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:47.566 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.548+0000 7f271d64d700 1 --2- 172.21.15.174:0/2970540407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27181008b0 0x7f27181959d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:47.566 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.548+0000 7f271d64d700 1 -- 172.21.15.174:0/2970540407 >> 172.21.15.174:0/2970540407 conn(0x7f27180fbe30 msgr2=0x7f27180fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:47.566 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.548+0000 7f271d64d700 1 -- 172.21.15.174:0/2970540407 shutdown_connections 2024-02-19T18:53:47.567 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:47.548+0000 7f271d64d700 1 -- 172.21.15.174:0/2970540407 wait complete. 2024-02-19T18:53:47.567 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:53:48.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:48 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/2970540407' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:49.125 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:49.126 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:49.337 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:49.383 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:49.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:49 smithi114 ceph-mon[29441]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:50.901 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.884+0000 7f7bb9610700 1 -- 172.21.15.174:0/3022873195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bb41008b0 msgr2=0x7f7bb4100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:50.901 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.884+0000 7f7bb9610700 1 --2- 172.21.15.174:0/3022873195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bb41008b0 0x7f7bb4100cd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8009910 tx=0x7f7ba8009c20 comp rx=0 tx=0).stop 2024-02-19T18:53:50.902 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.884+0000 7f7bb9610700 1 -- 172.21.15.174:0/3022873195 shutdown_connections 2024-02-19T18:53:50.902 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.884+0000 7f7bb9610700 1 --2- 172.21.15.174:0/3022873195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bb41008b0 0x7f7bb4100cd0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:50.902 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.884+0000 7f7bb9610700 1 -- 172.21.15.174:0/3022873195 >> 172.21.15.174:0/3022873195 conn(0x7f7bb40fbe30 msgr2=0x7f7bb40fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:50.902 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.884+0000 7f7bb9610700 1 -- 172.21.15.174:0/3022873195 shutdown_connections 2024-02-19T18:53:50.902 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.884+0000 7f7bb9610700 1 -- 172.21.15.174:0/3022873195 wait complete. 2024-02-19T18:53:50.902 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.885+0000 7f7bb9610700 1 Processor -- start 2024-02-19T18:53:50.904 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.885+0000 7f7bb9610700 1 -- start start 2024-02-19T18:53:50.904 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.886+0000 7f7bb9610700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bb41008b0 0x7f7bb41959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:50.904 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.886+0000 7f7bb9610700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb4195f10 con 0x7f7bb41008b0 2024-02-19T18:53:50.904 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.886+0000 7f7bb2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bb41008b0 0x7f7bb41959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:50.904 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.886+0000 7f7bb2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bb41008b0 0x7f7bb41959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:50536/0 (socket says 172.21.15.174:50536) 2024-02-19T18:53:50.904 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.886+0000 7f7bb2ffd700 1 -- 172.21.15.174:0/897351665 learned_addr learned my addr 172.21.15.174:0/897351665 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:53:50.904 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.886+0000 7f7bb2ffd700 1 -- 172.21.15.174:0/897351665 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba80095f0 con 0x7f7bb41008b0 2024-02-19T18:53:50.904 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.886+0000 7f7bb2ffd700 1 --2- 172.21.15.174:0/897351665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bb41008b0 0x7f7bb41959d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8005b40 tx=0x7f7ba8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:50.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.886+0000 7f7b9bfff700 1 -- 172.21.15.174:0/897351665 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ba801d070 con 0x7f7bb41008b0 2024-02-19T18:53:50.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.886+0000 7f7bb9610700 1 -- 172.21.15.174:0/897351665 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb4196110 con 0x7f7bb41008b0 2024-02-19T18:53:50.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.886+0000 7f7b9bfff700 1 -- 172.21.15.174:0/897351665 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7ba800be10 con 0x7f7bb41008b0 2024-02-19T18:53:50.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.886+0000 7f7b9bfff700 1 -- 172.21.15.174:0/897351665 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ba800f460 con 0x7f7bb41008b0 2024-02-19T18:53:50.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.887+0000 7f7bb9610700 1 -- 172.21.15.174:0/897351665 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb41965b0 con 0x7f7bb41008b0 2024-02-19T18:53:50.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.887+0000 7f7b9bfff700 1 -- 172.21.15.174:0/897351665 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f7ba800f6d0 con 0x7f7bb41008b0 2024-02-19T18:53:50.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.888+0000 7f7bb9610700 1 -- 172.21.15.174:0/897351665 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb418f4d0 con 0x7f7bb41008b0 2024-02-19T18:53:50.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.888+0000 7f7b9bfff700 1 --2- 172.21.15.174:0/897351665 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7b9c0383c0 0x7f7b9c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:50.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.888+0000 7f7b9bfff700 1 -- 172.21.15.174:0/897351665 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7ba804c6e0 con 0x7f7bb41008b0 2024-02-19T18:53:50.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.888+0000 7f7bb27fc700 1 --2- 172.21.15.174:0/897351665 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7b9c0383c0 0x7f7b9c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:50.906 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.888+0000 7f7bb27fc700 1 --2- 172.21.15.174:0/897351665 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7b9c0383c0 0x7f7b9c03a880 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4006fd0 tx=0x7f7ba4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:50.909 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:50.893+0000 7f7b9bfff700 1 -- 172.21.15.174:0/897351665 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7ba802a360 con 0x7f7bb41008b0 2024-02-19T18:53:51.173 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:51.156+0000 7f7bb9610700 1 -- 172.21.15.174:0/897351665 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7bb402cc70 con 0x7f7bb41008b0 2024-02-19T18:53:51.173 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:51.157+0000 7f7b9bfff700 1 -- 172.21.15.174:0/897351665 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f7ba8027030 con 0x7f7bb41008b0 2024-02-19T18:53:51.174 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:53:51.174 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:51.175 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:51.159+0000 7f7bb9610700 1 -- 172.21.15.174:0/897351665 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7b9c0383c0 msgr2=0x7f7b9c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:51.176 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:51.159+0000 7f7bb9610700 1 --2- 172.21.15.174:0/897351665 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7b9c0383c0 0x7f7b9c03a880 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4006fd0 tx=0x7f7ba4006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:51.176 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:51.159+0000 7f7bb9610700 1 -- 172.21.15.174:0/897351665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bb41008b0 msgr2=0x7f7bb41959d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:51.176 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:51.159+0000 7f7bb9610700 1 --2- 172.21.15.174:0/897351665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bb41008b0 0x7f7bb41959d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8005b40 tx=0x7f7ba8004740 comp rx=0 tx=0).stop 2024-02-19T18:53:51.176 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:51.159+0000 7f7bb9610700 1 -- 172.21.15.174:0/897351665 shutdown_connections 2024-02-19T18:53:51.176 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:51.159+0000 7f7bb9610700 1 --2- 172.21.15.174:0/897351665 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7b9c0383c0 0x7f7b9c03a880 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:51.176 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:51.159+0000 7f7bb9610700 1 --2- 172.21.15.174:0/897351665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bb41008b0 0x7f7bb41959d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:51.176 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:51.159+0000 7f7bb9610700 1 -- 172.21.15.174:0/897351665 >> 172.21.15.174:0/897351665 conn(0x7f7bb40fbe30 msgr2=0x7f7bb40fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:51.176 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:51.159+0000 7f7bb9610700 1 -- 172.21.15.174:0/897351665 shutdown_connections 2024-02-19T18:53:51.176 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:51.159+0000 7f7bb9610700 1 -- 172.21.15.174:0/897351665 wait complete. 2024-02-19T18:53:51.177 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:53:51.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:51 smithi114 ceph-mon[29441]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:51.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:51 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/897351665' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:52.702 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:52.702 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:52.905 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:52.951 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:53.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:53 smithi114 ceph-mon[29441]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:54.362 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.344+0000 7fbfb8913700 1 -- 172.21.15.174:0/2687568395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfb41008b0 msgr2=0x7fbfb4100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:54.362 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.344+0000 7fbfb8913700 1 --2- 172.21.15.174:0/2687568395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfb41008b0 0x7fbfb4100cd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbfac009910 tx=0x7fbfac009c20 comp rx=0 tx=0).stop 2024-02-19T18:53:54.362 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.344+0000 7fbfb8913700 1 -- 172.21.15.174:0/2687568395 shutdown_connections 2024-02-19T18:53:54.362 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.344+0000 7fbfb8913700 1 --2- 172.21.15.174:0/2687568395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfb41008b0 0x7fbfb4100cd0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:54.362 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.344+0000 7fbfb8913700 1 -- 172.21.15.174:0/2687568395 >> 172.21.15.174:0/2687568395 conn(0x7fbfb40fbe30 msgr2=0x7fbfb40fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:54.362 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.344+0000 7fbfb8913700 1 -- 172.21.15.174:0/2687568395 shutdown_connections 2024-02-19T18:53:54.362 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.344+0000 7fbfb8913700 1 -- 172.21.15.174:0/2687568395 wait complete. 2024-02-19T18:53:54.363 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.345+0000 7fbfb8913700 1 Processor -- start 2024-02-19T18:53:54.363 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.345+0000 7fbfb8913700 1 -- start start 2024-02-19T18:53:54.363 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.345+0000 7fbfb8913700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfb41008b0 0x7fbfb4074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:54.363 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.345+0000 7fbfb8913700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb4075070 con 0x7fbfb41008b0 2024-02-19T18:53:54.363 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.346+0000 7fbfb259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfb41008b0 0x7fbfb4074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:54.363 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.346+0000 7fbfb259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfb41008b0 0x7fbfb4074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:50546/0 (socket says 172.21.15.174:50546) 2024-02-19T18:53:54.363 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.346+0000 7fbfb259c700 1 -- 172.21.15.174:0/3679723512 learned_addr learned my addr 172.21.15.174:0/3679723512 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:53:54.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.346+0000 7fbfb259c700 1 -- 172.21.15.174:0/3679723512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac0095f0 con 0x7fbfb41008b0 2024-02-19T18:53:54.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.346+0000 7fbfb259c700 1 --2- 172.21.15.174:0/3679723512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfb41008b0 0x7fbfb4074b30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbfac005950 tx=0x7fbfac009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:54.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.346+0000 7fbfa37fe700 1 -- 172.21.15.174:0/3679723512 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfac01d070 con 0x7fbfb41008b0 2024-02-19T18:53:54.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.346+0000 7fbfb8913700 1 -- 172.21.15.174:0/3679723512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb4073180 con 0x7fbfb41008b0 2024-02-19T18:53:54.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.346+0000 7fbfb8913700 1 -- 172.21.15.174:0/3679723512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb4073620 con 0x7fbfb41008b0 2024-02-19T18:53:54.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.346+0000 7fbfa37fe700 1 -- 172.21.15.174:0/3679723512 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbfac00be10 con 0x7fbfb41008b0 2024-02-19T18:53:54.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.346+0000 7fbfa37fe700 1 -- 172.21.15.174:0/3679723512 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfac00f460 con 0x7fbfb41008b0 2024-02-19T18:53:54.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.347+0000 7fbfb8913700 1 -- 172.21.15.174:0/3679723512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf94005320 con 0x7fbfb41008b0 2024-02-19T18:53:54.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.347+0000 7fbfa37fe700 1 -- 172.21.15.174:0/3679723512 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fbfac00b960 con 0x7fbfb41008b0 2024-02-19T18:53:54.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.347+0000 7fbfa37fe700 1 --2- 172.21.15.174:0/3679723512 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fbf9c038370 0x7fbf9c03a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:54.366 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.348+0000 7fbfa37fe700 1 -- 172.21.15.174:0/3679723512 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbfac04c270 con 0x7fbfb41008b0 2024-02-19T18:53:54.366 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.348+0000 7fbfb1d9b700 1 --2- 172.21.15.174:0/3679723512 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fbf9c038370 0x7fbf9c03a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:54.366 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.348+0000 7fbfb1d9b700 1 --2- 172.21.15.174:0/3679723512 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fbf9c038370 0x7fbf9c03a830 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800ad30 tx=0x7fbfa80093f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:54.369 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.352+0000 7fbfa37fe700 1 -- 172.21.15.174:0/3679723512 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbfac00f5c0 con 0x7fbfb41008b0 2024-02-19T18:53:54.629 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.613+0000 7fbfb8913700 1 -- 172.21.15.174:0/3679723512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf94005190 con 0x7fbfb41008b0 2024-02-19T18:53:54.630 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.613+0000 7fbfa37fe700 1 -- 172.21.15.174:0/3679723512 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fbfac027020 con 0x7fbfb41008b0 2024-02-19T18:53:54.630 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:53:54.630 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:54.632 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.615+0000 7fbfb8913700 1 -- 172.21.15.174:0/3679723512 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fbf9c038370 msgr2=0x7fbf9c03a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:54.632 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.615+0000 7fbfb8913700 1 --2- 172.21.15.174:0/3679723512 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fbf9c038370 0x7fbf9c03a830 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800ad30 tx=0x7fbfa80093f0 comp rx=0 tx=0).stop 2024-02-19T18:53:54.632 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.615+0000 7fbfb8913700 1 -- 172.21.15.174:0/3679723512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfb41008b0 msgr2=0x7fbfb4074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:54.632 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.615+0000 7fbfb8913700 1 --2- 172.21.15.174:0/3679723512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfb41008b0 0x7fbfb4074b30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbfac005950 tx=0x7fbfac009e60 comp rx=0 tx=0).stop 2024-02-19T18:53:54.632 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.616+0000 7fbfb8913700 1 -- 172.21.15.174:0/3679723512 shutdown_connections 2024-02-19T18:53:54.632 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.616+0000 7fbfb8913700 1 --2- 172.21.15.174:0/3679723512 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fbf9c038370 0x7fbf9c03a830 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:54.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.616+0000 7fbfb8913700 1 --2- 172.21.15.174:0/3679723512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbfb41008b0 0x7fbfb4074b30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:54.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.616+0000 7fbfb8913700 1 -- 172.21.15.174:0/3679723512 >> 172.21.15.174:0/3679723512 conn(0x7fbfb40fbe30 msgr2=0x7fbfb40fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:54.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.616+0000 7fbfb8913700 1 -- 172.21.15.174:0/3679723512 shutdown_connections 2024-02-19T18:53:54.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:54.616+0000 7fbfb8913700 1 -- 172.21.15.174:0/3679723512 wait complete. 2024-02-19T18:53:54.633 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:53:55.510 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:55 smithi114 ceph-mon[29441]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:55.510 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:55 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/3679723512' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:56.194 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:56.195 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:56.397 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:56.442 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:57.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:57 smithi114 ceph-mon[29441]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:57.890 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.871+0000 7fb223fc5700 1 -- 172.21.15.174:0/2696234344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb21c0ff790 msgr2=0x7fb21c0ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:57.890 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.871+0000 7fb223fc5700 1 --2- 172.21.15.174:0/2696234344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb21c0ff790 0x7fb21c0ffbb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb20c009910 tx=0x7fb20c009c20 comp rx=0 tx=0).stop 2024-02-19T18:53:57.890 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.871+0000 7fb223fc5700 1 -- 172.21.15.174:0/2696234344 shutdown_connections 2024-02-19T18:53:57.890 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.871+0000 7fb223fc5700 1 --2- 172.21.15.174:0/2696234344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb21c0ff790 0x7fb21c0ffbb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:57.891 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.871+0000 7fb223fc5700 1 -- 172.21.15.174:0/2696234344 >> 172.21.15.174:0/2696234344 conn(0x7fb21c0fad10 msgr2=0x7fb21c0fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:57.891 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.872+0000 7fb223fc5700 1 -- 172.21.15.174:0/2696234344 shutdown_connections 2024-02-19T18:53:57.891 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.872+0000 7fb223fc5700 1 -- 172.21.15.174:0/2696234344 wait complete. 2024-02-19T18:53:57.891 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.872+0000 7fb223fc5700 1 Processor -- start 2024-02-19T18:53:57.891 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.872+0000 7fb223fc5700 1 -- start start 2024-02-19T18:53:57.891 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.872+0000 7fb223fc5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb21c0ff790 0x7fb21c069430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:57.891 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.872+0000 7fb223fc5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c1986d0 con 0x7fb21c0ff790 2024-02-19T18:53:57.891 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.872+0000 7fb221d61700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb21c0ff790 0x7fb21c069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:57.891 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.872+0000 7fb221d61700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb21c0ff790 0x7fb21c069430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:45866/0 (socket says 172.21.15.174:45866) 2024-02-19T18:53:57.891 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.872+0000 7fb221d61700 1 -- 172.21.15.174:0/1503718323 learned_addr learned my addr 172.21.15.174:0/1503718323 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:53:57.891 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.873+0000 7fb221d61700 1 -- 172.21.15.174:0/1503718323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20c0095f0 con 0x7fb21c0ff790 2024-02-19T18:53:57.891 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.873+0000 7fb221d61700 1 --2- 172.21.15.174:0/1503718323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb21c0ff790 0x7fb21c069430 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb20c005b40 tx=0x7fb20c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:57.892 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.873+0000 7fb212ffd700 1 -- 172.21.15.174:0/1503718323 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb20c01d070 con 0x7fb21c0ff790 2024-02-19T18:53:57.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.873+0000 7fb223fc5700 1 -- 172.21.15.174:0/1503718323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21c198810 con 0x7fb21c0ff790 2024-02-19T18:53:57.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.873+0000 7fb223fc5700 1 -- 172.21.15.174:0/1503718323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21c198cb0 con 0x7fb21c0ff790 2024-02-19T18:53:57.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.873+0000 7fb212ffd700 1 -- 172.21.15.174:0/1503718323 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb20c00be10 con 0x7fb21c0ff790 2024-02-19T18:53:57.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.873+0000 7fb212ffd700 1 -- 172.21.15.174:0/1503718323 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb20c00f460 con 0x7fb21c0ff790 2024-02-19T18:53:57.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.874+0000 7fb223fc5700 1 -- 172.21.15.174:0/1503718323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21c18e3b0 con 0x7fb21c0ff790 2024-02-19T18:53:57.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.874+0000 7fb212ffd700 1 -- 172.21.15.174:0/1503718323 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fb20c0227c0 con 0x7fb21c0ff790 2024-02-19T18:53:57.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.874+0000 7fb212ffd700 1 --2- 172.21.15.174:0/1503718323 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb2080383c0 0x7fb20803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:57.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.875+0000 7fb212ffd700 1 -- 172.21.15.174:0/1503718323 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb20c04c330 con 0x7fb21c0ff790 2024-02-19T18:53:57.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.875+0000 7fb221560700 1 --2- 172.21.15.174:0/1503718323 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb2080383c0 0x7fb20803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:57.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.875+0000 7fb221560700 1 --2- 172.21.15.174:0/1503718323 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb2080383c0 0x7fb20803a880 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb218006fd0 tx=0x7fb218006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:57.897 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:57.880+0000 7fb212ffd700 1 -- 172.21.15.174:0/1503718323 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb20c00b960 con 0x7fb21c0ff790 2024-02-19T18:53:58.158 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:58.141+0000 7fb223fc5700 1 -- 172.21.15.174:0/1503718323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb21c02cc70 con 0x7fb21c0ff790 2024-02-19T18:53:58.159 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:58.141+0000 7fb212ffd700 1 -- 172.21.15.174:0/1503718323 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fb20c027020 con 0x7fb21c0ff790 2024-02-19T18:53:58.159 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:53:58.159 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:58.161 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:58.143+0000 7fb223fc5700 1 -- 172.21.15.174:0/1503718323 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb2080383c0 msgr2=0x7fb20803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:58.161 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:58.143+0000 7fb223fc5700 1 --2- 172.21.15.174:0/1503718323 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb2080383c0 0x7fb20803a880 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb218006fd0 tx=0x7fb218006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:58.161 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:58.143+0000 7fb223fc5700 1 -- 172.21.15.174:0/1503718323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb21c0ff790 msgr2=0x7fb21c069430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:58.161 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:58.143+0000 7fb223fc5700 1 --2- 172.21.15.174:0/1503718323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb21c0ff790 0x7fb21c069430 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb20c005b40 tx=0x7fb20c004740 comp rx=0 tx=0).stop 2024-02-19T18:53:58.161 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:58.143+0000 7fb223fc5700 1 -- 172.21.15.174:0/1503718323 shutdown_connections 2024-02-19T18:53:58.161 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:58.143+0000 7fb223fc5700 1 --2- 172.21.15.174:0/1503718323 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fb2080383c0 0x7fb20803a880 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:58.162 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:58.143+0000 7fb223fc5700 1 --2- 172.21.15.174:0/1503718323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb21c0ff790 0x7fb21c069430 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:58.162 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:58.143+0000 7fb223fc5700 1 -- 172.21.15.174:0/1503718323 >> 172.21.15.174:0/1503718323 conn(0x7fb21c0fad10 msgr2=0x7fb21c0fba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:58.162 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:58.144+0000 7fb223fc5700 1 -- 172.21.15.174:0/1503718323 shutdown_connections 2024-02-19T18:53:58.162 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:53:58.144+0000 7fb223fc5700 1 -- 172.21.15.174:0/1503718323 wait complete. 2024-02-19T18:53:58.162 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:53:59.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:58 smithi114 ceph-mon[29441]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:59.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:53:58 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/1503718323' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:59.732 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:59.732 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:59.936 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:59.982 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:01.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:01 smithi114 ceph-mon[29441]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:01.544 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.525+0000 7f096f4b3700 1 -- 172.21.15.174:0/1616380238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09680ff790 msgr2=0x7f09680ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:01.544 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.525+0000 7f096f4b3700 1 --2- 172.21.15.174:0/1616380238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09680ff790 0x7f09680ffbb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f096000b0a0 tx=0x7f096000b3b0 comp rx=0 tx=0).stop 2024-02-19T18:54:01.544 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.525+0000 7f096f4b3700 1 -- 172.21.15.174:0/1616380238 shutdown_connections 2024-02-19T18:54:01.544 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.525+0000 7f096f4b3700 1 --2- 172.21.15.174:0/1616380238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09680ff790 0x7f09680ffbb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:01.544 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.525+0000 7f096f4b3700 1 -- 172.21.15.174:0/1616380238 >> 172.21.15.174:0/1616380238 conn(0x7f09680fad30 msgr2=0x7f09680fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:01.544 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.525+0000 7f096f4b3700 1 -- 172.21.15.174:0/1616380238 shutdown_connections 2024-02-19T18:54:01.545 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.525+0000 7f096f4b3700 1 -- 172.21.15.174:0/1616380238 wait complete. 2024-02-19T18:54:01.545 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.526+0000 7f096f4b3700 1 Processor -- start 2024-02-19T18:54:01.545 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.526+0000 7f096f4b3700 1 -- start start 2024-02-19T18:54:01.545 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.526+0000 7f096f4b3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09680ff790 0x7f0968195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:01.545 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.526+0000 7f096f4b3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0968195ed0 con 0x7f09680ff790 2024-02-19T18:54:01.545 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.526+0000 7f096d24f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09680ff790 0x7f0968195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:01.545 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.526+0000 7f096d24f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09680ff790 0x7f0968195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:45894/0 (socket says 172.21.15.174:45894) 2024-02-19T18:54:01.545 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.526+0000 7f096d24f700 1 -- 172.21.15.174:0/2077950852 learned_addr learned my addr 172.21.15.174:0/2077950852 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:54:01.545 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.527+0000 7f096d24f700 1 -- 172.21.15.174:0/2077950852 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0960009d00 con 0x7f09680ff790 2024-02-19T18:54:01.545 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.527+0000 7f096d24f700 1 --2- 172.21.15.174:0/2077950852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09680ff790 0x7f0968195990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f096000d010 tx=0x7f096000bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:01.546 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.527+0000 7f095e7fc700 1 -- 172.21.15.174:0/2077950852 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f096000e050 con 0x7f09680ff790 2024-02-19T18:54:01.547 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.527+0000 7f096f4b3700 1 -- 172.21.15.174:0/2077950852 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09681960d0 con 0x7f09680ff790 2024-02-19T18:54:01.547 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.527+0000 7f095e7fc700 1 -- 172.21.15.174:0/2077950852 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f096001d920 con 0x7f09680ff790 2024-02-19T18:54:01.547 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.527+0000 7f095e7fc700 1 -- 172.21.15.174:0/2077950852 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09600129b0 con 0x7f09680ff790 2024-02-19T18:54:01.547 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.527+0000 7f096f4b3700 1 -- 172.21.15.174:0/2077950852 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09681964f0 con 0x7f09680ff790 2024-02-19T18:54:01.547 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.528+0000 7f096f4b3700 1 -- 172.21.15.174:0/2077950852 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096818f3c0 con 0x7f09680ff790 2024-02-19T18:54:01.547 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.528+0000 7f095e7fc700 1 -- 172.21.15.174:0/2077950852 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f0960019070 con 0x7f09680ff790 2024-02-19T18:54:01.547 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.528+0000 7f095e7fc700 1 --2- 172.21.15.174:0/2077950852 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0954038410 0x7f095403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:01.547 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.528+0000 7f095e7fc700 1 -- 172.21.15.174:0/2077950852 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f096004c6b0 con 0x7f09680ff790 2024-02-19T18:54:01.547 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.528+0000 7f096ca4e700 1 --2- 172.21.15.174:0/2077950852 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0954038410 0x7f095403a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:01.547 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.529+0000 7f096ca4e700 1 --2- 172.21.15.174:0/2077950852 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0954038410 0x7f095403a8d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0958006fd0 tx=0x7f0958006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:01.551 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.534+0000 7f095e7fc700 1 -- 172.21.15.174:0/2077950852 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f096001bdf0 con 0x7f09680ff790 2024-02-19T18:54:01.813 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.796+0000 7f096f4b3700 1 -- 172.21.15.174:0/2077950852 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09680623c0 con 0x7f09680ff790 2024-02-19T18:54:01.814 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.797+0000 7f095e7fc700 1 -- 172.21.15.174:0/2077950852 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f096001bc10 con 0x7f09680ff790 2024-02-19T18:54:01.814 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:54:01.814 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:54:01.816 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.799+0000 7f096f4b3700 1 -- 172.21.15.174:0/2077950852 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0954038410 msgr2=0x7f095403a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:01.816 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.799+0000 7f096f4b3700 1 --2- 172.21.15.174:0/2077950852 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0954038410 0x7f095403a8d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0958006fd0 tx=0x7f0958006e40 comp rx=0 tx=0).stop 2024-02-19T18:54:01.816 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.799+0000 7f096f4b3700 1 -- 172.21.15.174:0/2077950852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09680ff790 msgr2=0x7f0968195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:01.816 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.799+0000 7f096f4b3700 1 --2- 172.21.15.174:0/2077950852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09680ff790 0x7f0968195990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f096000d010 tx=0x7f096000bb10 comp rx=0 tx=0).stop 2024-02-19T18:54:01.817 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.799+0000 7f096f4b3700 1 -- 172.21.15.174:0/2077950852 shutdown_connections 2024-02-19T18:54:01.817 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.799+0000 7f096f4b3700 1 --2- 172.21.15.174:0/2077950852 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f0954038410 0x7f095403a8d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:01.817 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.799+0000 7f096f4b3700 1 --2- 172.21.15.174:0/2077950852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09680ff790 0x7f0968195990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:01.817 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.799+0000 7f096f4b3700 1 -- 172.21.15.174:0/2077950852 >> 172.21.15.174:0/2077950852 conn(0x7f09680fad30 msgr2=0x7f0968069120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:01.817 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.799+0000 7f096f4b3700 1 -- 172.21.15.174:0/2077950852 shutdown_connections 2024-02-19T18:54:01.817 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:01.799+0000 7f096f4b3700 1 -- 172.21.15.174:0/2077950852 wait complete. 2024-02-19T18:54:01.817 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:54:02.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:02 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/2077950852' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:54:03.435 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:54:03.435 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:54:03.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:03 smithi114 ceph-mon[29441]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:03.636 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:03.684 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:05.079 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.060+0000 7fc467e6b700 1 -- 172.21.15.174:0/3431026721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4600ff7a0 msgr2=0x7fc4600ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:05.079 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.060+0000 7fc467e6b700 1 --2- 172.21.15.174:0/3431026721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4600ff7a0 0x7fc4600ffbc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc450009910 tx=0x7fc450009c20 comp rx=0 tx=0).stop 2024-02-19T18:54:05.079 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.060+0000 7fc467e6b700 1 -- 172.21.15.174:0/3431026721 shutdown_connections 2024-02-19T18:54:05.079 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.060+0000 7fc467e6b700 1 --2- 172.21.15.174:0/3431026721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4600ff7a0 0x7fc4600ffbc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:05.079 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.060+0000 7fc467e6b700 1 -- 172.21.15.174:0/3431026721 >> 172.21.15.174:0/3431026721 conn(0x7fc4600fad20 msgr2=0x7fc4600fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:05.079 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.060+0000 7fc467e6b700 1 -- 172.21.15.174:0/3431026721 shutdown_connections 2024-02-19T18:54:05.079 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.060+0000 7fc467e6b700 1 -- 172.21.15.174:0/3431026721 wait complete. 2024-02-19T18:54:05.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.061+0000 7fc467e6b700 1 Processor -- start 2024-02-19T18:54:05.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.061+0000 7fc467e6b700 1 -- start start 2024-02-19T18:54:05.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.061+0000 7fc467e6b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4600ff7a0 0x7fc460068b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:05.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.061+0000 7fc467e6b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4601986f0 con 0x7fc4600ff7a0 2024-02-19T18:54:05.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.061+0000 7fc465c07700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4600ff7a0 0x7fc460068b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:05.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.061+0000 7fc465c07700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4600ff7a0 0x7fc460068b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:58176/0 (socket says 172.21.15.174:58176) 2024-02-19T18:54:05.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.061+0000 7fc465c07700 1 -- 172.21.15.174:0/940383232 learned_addr learned my addr 172.21.15.174:0/940383232 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:54:05.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.061+0000 7fc465c07700 1 -- 172.21.15.174:0/940383232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4500095f0 con 0x7fc4600ff7a0 2024-02-19T18:54:05.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.062+0000 7fc465c07700 1 --2- 172.21.15.174:0/940383232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4600ff7a0 0x7fc460068b00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc450005b40 tx=0x7fc450004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:05.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.062+0000 7fc456ffd700 1 -- 172.21.15.174:0/940383232 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc45001d070 con 0x7fc4600ff7a0 2024-02-19T18:54:05.082 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.062+0000 7fc467e6b700 1 -- 172.21.15.174:0/940383232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc460198830 con 0x7fc4600ff7a0 2024-02-19T18:54:05.082 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.062+0000 7fc467e6b700 1 -- 172.21.15.174:0/940383232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc460198cd0 con 0x7fc4600ff7a0 2024-02-19T18:54:05.082 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.062+0000 7fc456ffd700 1 -- 172.21.15.174:0/940383232 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc450005c00 con 0x7fc4600ff7a0 2024-02-19T18:54:05.082 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.062+0000 7fc456ffd700 1 -- 172.21.15.174:0/940383232 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc45000fc30 con 0x7fc4600ff7a0 2024-02-19T18:54:05.082 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.063+0000 7fc467e6b700 1 -- 172.21.15.174:0/940383232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc444005320 con 0x7fc4600ff7a0 2024-02-19T18:54:05.082 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.063+0000 7fc456ffd700 1 -- 172.21.15.174:0/940383232 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fc45000b960 con 0x7fc4600ff7a0 2024-02-19T18:54:05.082 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.063+0000 7fc456ffd700 1 --2- 172.21.15.174:0/940383232 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fc44c0383c0 0x7fc44c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:05.082 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.063+0000 7fc456ffd700 1 -- 172.21.15.174:0/940383232 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc45004c490 con 0x7fc4600ff7a0 2024-02-19T18:54:05.082 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.064+0000 7fc465406700 1 --2- 172.21.15.174:0/940383232 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fc44c0383c0 0x7fc44c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:05.082 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.064+0000 7fc465406700 1 --2- 172.21.15.174:0/940383232 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fc44c0383c0 0x7fc44c03a880 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc45c006fd0 tx=0x7fc45c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:05.086 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.069+0000 7fc456ffd700 1 -- 172.21.15.174:0/940383232 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc450017440 con 0x7fc4600ff7a0 2024-02-19T18:54:05.346 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.329+0000 7fc467e6b700 1 -- 172.21.15.174:0/940383232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc444005190 con 0x7fc4600ff7a0 2024-02-19T18:54:05.347 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.329+0000 7fc456ffd700 1 -- 172.21.15.174:0/940383232 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fc450027070 con 0x7fc4600ff7a0 2024-02-19T18:54:05.347 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:54:05.347 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:54:05.349 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.331+0000 7fc467e6b700 1 -- 172.21.15.174:0/940383232 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fc44c0383c0 msgr2=0x7fc44c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:05.349 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.331+0000 7fc467e6b700 1 --2- 172.21.15.174:0/940383232 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fc44c0383c0 0x7fc44c03a880 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc45c006fd0 tx=0x7fc45c006e40 comp rx=0 tx=0).stop 2024-02-19T18:54:05.349 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.331+0000 7fc467e6b700 1 -- 172.21.15.174:0/940383232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4600ff7a0 msgr2=0x7fc460068b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:05.349 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.331+0000 7fc467e6b700 1 --2- 172.21.15.174:0/940383232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4600ff7a0 0x7fc460068b00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc450005b40 tx=0x7fc450004740 comp rx=0 tx=0).stop 2024-02-19T18:54:05.349 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.331+0000 7fc467e6b700 1 -- 172.21.15.174:0/940383232 shutdown_connections 2024-02-19T18:54:05.349 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.331+0000 7fc467e6b700 1 --2- 172.21.15.174:0/940383232 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fc44c0383c0 0x7fc44c03a880 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:05.350 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.331+0000 7fc467e6b700 1 --2- 172.21.15.174:0/940383232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4600ff7a0 0x7fc460068b00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:05.350 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.331+0000 7fc467e6b700 1 -- 172.21.15.174:0/940383232 >> 172.21.15.174:0/940383232 conn(0x7fc4600fad20 msgr2=0x7fc4600fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:05.350 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.331+0000 7fc467e6b700 1 -- 172.21.15.174:0/940383232 shutdown_connections 2024-02-19T18:54:05.350 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:05.331+0000 7fc467e6b700 1 -- 172.21.15.174:0/940383232 wait complete. 2024-02-19T18:54:05.350 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:54:05.523 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:05 smithi114 ceph-mon[29441]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:06.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:06 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/940383232' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:54:06.906 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:54:06.906 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:54:07.107 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:07.153 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:07.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:07 smithi114 ceph-mon[29441]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:08.639 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.621+0000 7f7d9dfc2700 1 -- 172.21.15.174:0/4120000276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d980ff7a0 msgr2=0x7f7d980ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:08.639 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.621+0000 7f7d9dfc2700 1 --2- 172.21.15.174:0/4120000276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d980ff7a0 0x7f7d980ffbc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7d88009910 tx=0x7f7d88009c20 comp rx=0 tx=0).stop 2024-02-19T18:54:08.639 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.621+0000 7f7d9dfc2700 1 -- 172.21.15.174:0/4120000276 shutdown_connections 2024-02-19T18:54:08.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.621+0000 7f7d9dfc2700 1 --2- 172.21.15.174:0/4120000276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d980ff7a0 0x7f7d980ffbc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:08.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.621+0000 7f7d9dfc2700 1 -- 172.21.15.174:0/4120000276 >> 172.21.15.174:0/4120000276 conn(0x7f7d980fad20 msgr2=0x7f7d980fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:08.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.621+0000 7f7d9dfc2700 1 -- 172.21.15.174:0/4120000276 shutdown_connections 2024-02-19T18:54:08.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.621+0000 7f7d9dfc2700 1 -- 172.21.15.174:0/4120000276 wait complete. 2024-02-19T18:54:08.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.622+0000 7f7d9dfc2700 1 Processor -- start 2024-02-19T18:54:08.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.622+0000 7f7d9dfc2700 1 -- start start 2024-02-19T18:54:08.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.622+0000 7f7d9dfc2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d980ff7a0 0x7f7d981948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:08.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.622+0000 7f7d9dfc2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d98194de0 con 0x7f7d980ff7a0 2024-02-19T18:54:08.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.623+0000 7f7d977fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d980ff7a0 0x7f7d981948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:08.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.623+0000 7f7d977fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d980ff7a0 0x7f7d981948a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:58192/0 (socket says 172.21.15.174:58192) 2024-02-19T18:54:08.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.623+0000 7f7d977fe700 1 -- 172.21.15.174:0/3557362483 learned_addr learned my addr 172.21.15.174:0/3557362483 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:54:08.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.623+0000 7f7d977fe700 1 -- 172.21.15.174:0/3557362483 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d880095f0 con 0x7f7d980ff7a0 2024-02-19T18:54:08.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.623+0000 7f7d977fe700 1 --2- 172.21.15.174:0/3557362483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d980ff7a0 0x7f7d981948a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7d88005b40 tx=0x7f7d88004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:08.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.623+0000 7f7d957fa700 1 -- 172.21.15.174:0/3557362483 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d8801d070 con 0x7f7d980ff7a0 2024-02-19T18:54:08.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.623+0000 7f7d9dfc2700 1 -- 172.21.15.174:0/3557362483 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d98194fe0 con 0x7f7d980ff7a0 2024-02-19T18:54:08.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.623+0000 7f7d957fa700 1 -- 172.21.15.174:0/3557362483 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7d8800be10 con 0x7f7d980ff7a0 2024-02-19T18:54:08.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.623+0000 7f7d957fa700 1 -- 172.21.15.174:0/3557362483 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d8800f460 con 0x7f7d980ff7a0 2024-02-19T18:54:08.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.623+0000 7f7d9dfc2700 1 -- 172.21.15.174:0/3557362483 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d98195480 con 0x7f7d980ff7a0 2024-02-19T18:54:08.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.624+0000 7f7d9dfc2700 1 -- 172.21.15.174:0/3557362483 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d981055e0 con 0x7f7d980ff7a0 2024-02-19T18:54:08.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.624+0000 7f7d957fa700 1 -- 172.21.15.174:0/3557362483 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f7d8800b960 con 0x7f7d980ff7a0 2024-02-19T18:54:08.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.624+0000 7f7d957fa700 1 --2- 172.21.15.174:0/3557362483 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7d840383c0 0x7f7d8403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:08.642 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.624+0000 7f7d957fa700 1 -- 172.21.15.174:0/3557362483 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7d8804c440 con 0x7f7d980ff7a0 2024-02-19T18:54:08.643 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.625+0000 7f7d8effd700 1 --2- 172.21.15.174:0/3557362483 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7d840383c0 0x7f7d8403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:08.643 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.625+0000 7f7d8effd700 1 --2- 172.21.15.174:0/3557362483 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7d840383c0 0x7f7d8403a880 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7d80006fd0 tx=0x7f7d80006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:08.646 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.630+0000 7f7d957fa700 1 -- 172.21.15.174:0/3557362483 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7d8802a360 con 0x7f7d980ff7a0 2024-02-19T18:54:08.907 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.890+0000 7f7d9dfc2700 1 -- 172.21.15.174:0/3557362483 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d981057d0 con 0x7f7d980ff7a0 2024-02-19T18:54:08.907 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.891+0000 7f7d957fa700 1 -- 172.21.15.174:0/3557362483 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f7d88027030 con 0x7f7d980ff7a0 2024-02-19T18:54:08.908 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:54:08.908 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:54:08.909 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.893+0000 7f7d9dfc2700 1 -- 172.21.15.174:0/3557362483 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7d840383c0 msgr2=0x7f7d8403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:08.910 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.893+0000 7f7d9dfc2700 1 --2- 172.21.15.174:0/3557362483 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7d840383c0 0x7f7d8403a880 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7d80006fd0 tx=0x7f7d80006e40 comp rx=0 tx=0).stop 2024-02-19T18:54:08.910 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.893+0000 7f7d9dfc2700 1 -- 172.21.15.174:0/3557362483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d980ff7a0 msgr2=0x7f7d981948a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:08.910 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.893+0000 7f7d9dfc2700 1 --2- 172.21.15.174:0/3557362483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d980ff7a0 0x7f7d981948a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7d88005b40 tx=0x7f7d88004740 comp rx=0 tx=0).stop 2024-02-19T18:54:08.910 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.893+0000 7f7d9dfc2700 1 -- 172.21.15.174:0/3557362483 shutdown_connections 2024-02-19T18:54:08.910 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.893+0000 7f7d9dfc2700 1 --2- 172.21.15.174:0/3557362483 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7d840383c0 0x7f7d8403a880 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:08.910 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.893+0000 7f7d9dfc2700 1 --2- 172.21.15.174:0/3557362483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d980ff7a0 0x7f7d981948a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:08.910 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.893+0000 7f7d9dfc2700 1 -- 172.21.15.174:0/3557362483 >> 172.21.15.174:0/3557362483 conn(0x7f7d980fad20 msgr2=0x7f7d980fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:08.910 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.893+0000 7f7d9dfc2700 1 -- 172.21.15.174:0/3557362483 shutdown_connections 2024-02-19T18:54:08.910 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:08.893+0000 7f7d9dfc2700 1 -- 172.21.15.174:0/3557362483 wait complete. 2024-02-19T18:54:08.911 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:54:09.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:08 smithi114 ceph-mon[29441]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:10.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:09 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/3557362483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:54:10.631 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:54:10.632 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:54:10.833 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:10.879 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:11.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:10 smithi114 ceph-mon[29441]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:12.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.335+0000 7f01c6ba6700 1 -- 172.21.15.174:0/3358033873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01c00ff7a0 msgr2=0x7f01c00ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:12.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.335+0000 7f01c6ba6700 1 --2- 172.21.15.174:0/3358033873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01c00ff7a0 0x7f01c00ffbc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f01b800b0a0 tx=0x7f01b800b3b0 comp rx=0 tx=0).stop 2024-02-19T18:54:12.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.335+0000 7f01c6ba6700 1 -- 172.21.15.174:0/3358033873 shutdown_connections 2024-02-19T18:54:12.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.335+0000 7f01c6ba6700 1 --2- 172.21.15.174:0/3358033873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01c00ff7a0 0x7f01c00ffbc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:12.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.335+0000 7f01c6ba6700 1 -- 172.21.15.174:0/3358033873 >> 172.21.15.174:0/3358033873 conn(0x7f01c00fad20 msgr2=0x7f01c00fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:12.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.335+0000 7f01c6ba6700 1 -- 172.21.15.174:0/3358033873 shutdown_connections 2024-02-19T18:54:12.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.335+0000 7f01c6ba6700 1 -- 172.21.15.174:0/3358033873 wait complete. 2024-02-19T18:54:12.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.335+0000 7f01c6ba6700 1 Processor -- start 2024-02-19T18:54:12.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.336+0000 7f01c6ba6700 1 -- start start 2024-02-19T18:54:12.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.336+0000 7f01c6ba6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01c00ff7a0 0x7f01c00693f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:12.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.336+0000 7f01c6ba6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c01986e0 con 0x7f01c00ff7a0 2024-02-19T18:54:12.354 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.336+0000 7f01c4942700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01c00ff7a0 0x7f01c00693f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:12.354 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.336+0000 7f01c4942700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01c00ff7a0 0x7f01c00693f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:58220/0 (socket says 172.21.15.174:58220) 2024-02-19T18:54:12.354 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.336+0000 7f01c4942700 1 -- 172.21.15.174:0/2733829719 learned_addr learned my addr 172.21.15.174:0/2733829719 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:54:12.354 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.336+0000 7f01c4942700 1 -- 172.21.15.174:0/2733829719 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b8009d00 con 0x7f01c00ff7a0 2024-02-19T18:54:12.354 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.336+0000 7f01c4942700 1 --2- 172.21.15.174:0/2733829719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01c00ff7a0 0x7f01c00693f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f01b800b830 tx=0x7f01b800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:12.355 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.336+0000 7f01bdffb700 1 -- 172.21.15.174:0/2733829719 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01b800e040 con 0x7f01c00ff7a0 2024-02-19T18:54:12.355 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.337+0000 7f01c6ba6700 1 -- 172.21.15.174:0/2733829719 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c0198820 con 0x7f01c00ff7a0 2024-02-19T18:54:12.355 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.337+0000 7f01bdffb700 1 -- 172.21.15.174:0/2733829719 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f01b8012af0 con 0x7f01c00ff7a0 2024-02-19T18:54:12.355 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.337+0000 7f01c6ba6700 1 -- 172.21.15.174:0/2733829719 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c0198c40 con 0x7f01c00ff7a0 2024-02-19T18:54:12.355 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.337+0000 7f01bdffb700 1 -- 172.21.15.174:0/2733829719 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01b801d9e0 con 0x7f01c00ff7a0 2024-02-19T18:54:12.355 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.338+0000 7f01bdffb700 1 -- 172.21.15.174:0/2733829719 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f01b8012c60 con 0x7f01c00ff7a0 2024-02-19T18:54:12.356 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.338+0000 7f01c6ba6700 1 -- 172.21.15.174:0/2733829719 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c018e3c0 con 0x7f01c00ff7a0 2024-02-19T18:54:12.356 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.338+0000 7f01bdffb700 1 --2- 172.21.15.174:0/2733829719 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f01a80383c0 0x7f01a803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:12.356 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.338+0000 7f01bdffb700 1 -- 172.21.15.174:0/2733829719 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f01b804c940 con 0x7f01c00ff7a0 2024-02-19T18:54:12.356 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.338+0000 7f01bffff700 1 --2- 172.21.15.174:0/2733829719 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f01a80383c0 0x7f01a803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:12.356 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.338+0000 7f01bffff700 1 --2- 172.21.15.174:0/2733829719 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f01a80383c0 0x7f01a803a880 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01b4006fd0 tx=0x7f01b4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:12.359 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.342+0000 7f01bdffb700 1 -- 172.21.15.174:0/2733829719 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f01b801c5c0 con 0x7f01c00ff7a0 2024-02-19T18:54:12.625 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.609+0000 7f01c6ba6700 1 -- 172.21.15.174:0/2733829719 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01c002cc70 con 0x7f01c00ff7a0 2024-02-19T18:54:12.625 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.609+0000 7f01bdffb700 1 -- 172.21.15.174:0/2733829719 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f01b801c3e0 con 0x7f01c00ff7a0 2024-02-19T18:54:12.626 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:54:12.626 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:54:12.628 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.611+0000 7f01c6ba6700 1 -- 172.21.15.174:0/2733829719 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f01a80383c0 msgr2=0x7f01a803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:12.628 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.611+0000 7f01c6ba6700 1 --2- 172.21.15.174:0/2733829719 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f01a80383c0 0x7f01a803a880 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01b4006fd0 tx=0x7f01b4006e40 comp rx=0 tx=0).stop 2024-02-19T18:54:12.628 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.611+0000 7f01c6ba6700 1 -- 172.21.15.174:0/2733829719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01c00ff7a0 msgr2=0x7f01c00693f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:12.628 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.611+0000 7f01c6ba6700 1 --2- 172.21.15.174:0/2733829719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01c00ff7a0 0x7f01c00693f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f01b800b830 tx=0x7f01b800bd40 comp rx=0 tx=0).stop 2024-02-19T18:54:12.628 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.611+0000 7f01c6ba6700 1 -- 172.21.15.174:0/2733829719 shutdown_connections 2024-02-19T18:54:12.628 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.611+0000 7f01c6ba6700 1 --2- 172.21.15.174:0/2733829719 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f01a80383c0 0x7f01a803a880 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:12.628 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.611+0000 7f01c6ba6700 1 --2- 172.21.15.174:0/2733829719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01c00ff7a0 0x7f01c00693f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:12.629 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.611+0000 7f01c6ba6700 1 -- 172.21.15.174:0/2733829719 >> 172.21.15.174:0/2733829719 conn(0x7f01c00fad20 msgr2=0x7f01c00fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:12.629 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.611+0000 7f01c6ba6700 1 -- 172.21.15.174:0/2733829719 shutdown_connections 2024-02-19T18:54:12.629 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:12.611+0000 7f01c6ba6700 1 -- 172.21.15.174:0/2733829719 wait complete. 2024-02-19T18:54:12.629 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:54:13.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:13 smithi114 ceph-mon[29441]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:13.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:13 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/2733829719' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:54:14.168 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:54:14.168 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:54:14.370 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:14.416 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:15.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:15 smithi114 ceph-mon[29441]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:15.796 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.777+0000 7fd213897700 1 -- 172.21.15.174:0/3245132539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd20c1008b0 msgr2=0x7fd20c100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:15.797 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.777+0000 7fd213897700 1 --2- 172.21.15.174:0/3245132539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd20c1008b0 0x7fd20c100cd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd20400b0a0 tx=0x7fd20400b3b0 comp rx=0 tx=0).stop 2024-02-19T18:54:15.797 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.777+0000 7fd213897700 1 -- 172.21.15.174:0/3245132539 shutdown_connections 2024-02-19T18:54:15.797 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.777+0000 7fd213897700 1 --2- 172.21.15.174:0/3245132539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd20c1008b0 0x7fd20c100cd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:15.797 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.777+0000 7fd213897700 1 -- 172.21.15.174:0/3245132539 >> 172.21.15.174:0/3245132539 conn(0x7fd20c0fbe30 msgr2=0x7fd20c0fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:15.797 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.777+0000 7fd213897700 1 -- 172.21.15.174:0/3245132539 shutdown_connections 2024-02-19T18:54:15.797 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.777+0000 7fd213897700 1 -- 172.21.15.174:0/3245132539 wait complete. 2024-02-19T18:54:15.797 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.778+0000 7fd213897700 1 Processor -- start 2024-02-19T18:54:15.797 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.778+0000 7fd213897700 1 -- start start 2024-02-19T18:54:15.797 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.778+0000 7fd213897700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd20c1008b0 0x7fd20c074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:15.797 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.778+0000 7fd213897700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20c075070 con 0x7fd20c1008b0 2024-02-19T18:54:15.797 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.778+0000 7fd211633700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd20c1008b0 0x7fd20c074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:15.797 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.778+0000 7fd211633700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd20c1008b0 0x7fd20c074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:34280/0 (socket says 172.21.15.174:34280) 2024-02-19T18:54:15.797 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.778+0000 7fd211633700 1 -- 172.21.15.174:0/301996349 learned_addr learned my addr 172.21.15.174:0/301996349 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:54:15.797 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.779+0000 7fd211633700 1 -- 172.21.15.174:0/301996349 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd204009d00 con 0x7fd20c1008b0 2024-02-19T18:54:15.798 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.779+0000 7fd211633700 1 --2- 172.21.15.174:0/301996349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd20c1008b0 0x7fd20c074b30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd20400b830 tx=0x7fd20400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:15.798 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.779+0000 7fd2027fc700 1 -- 172.21.15.174:0/301996349 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd20400e040 con 0x7fd20c1008b0 2024-02-19T18:54:15.798 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.779+0000 7fd2027fc700 1 -- 172.21.15.174:0/301996349 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd204012af0 con 0x7fd20c1008b0 2024-02-19T18:54:15.798 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.779+0000 7fd2027fc700 1 -- 172.21.15.174:0/301996349 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd20401d9f0 con 0x7fd20c1008b0 2024-02-19T18:54:15.798 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.779+0000 7fd213897700 1 -- 172.21.15.174:0/301996349 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd20c073180 con 0x7fd20c1008b0 2024-02-19T18:54:15.798 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.779+0000 7fd213897700 1 -- 172.21.15.174:0/301996349 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd20c0735a0 con 0x7fd20c1008b0 2024-02-19T18:54:15.798 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.780+0000 7fd2027fc700 1 -- 172.21.15.174:0/301996349 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fd204012c60 con 0x7fd20c1008b0 2024-02-19T18:54:15.798 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.780+0000 7fd213897700 1 -- 172.21.15.174:0/301996349 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd20c073fb0 con 0x7fd20c1008b0 2024-02-19T18:54:15.798 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.780+0000 7fd2027fc700 1 --2- 172.21.15.174:0/301996349 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fd1f8038410 0x7fd1f803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:15.798 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.780+0000 7fd2027fc700 1 -- 172.21.15.174:0/301996349 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd20404c920 con 0x7fd20c1008b0 2024-02-19T18:54:15.798 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.780+0000 7fd210e32700 1 --2- 172.21.15.174:0/301996349 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fd1f8038410 0x7fd1f803a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:15.798 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.781+0000 7fd210e32700 1 --2- 172.21.15.174:0/301996349 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fd1f8038410 0x7fd1f803a8d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc006fd0 tx=0x7fd1fc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:15.801 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:15.785+0000 7fd2027fc700 1 -- 172.21.15.174:0/301996349 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd20401c5c0 con 0x7fd20c1008b0 2024-02-19T18:54:16.064 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:16.048+0000 7fd213897700 1 -- 172.21.15.174:0/301996349 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd20c18eeb0 con 0x7fd20c1008b0 2024-02-19T18:54:16.065 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:16.048+0000 7fd2027fc700 1 -- 172.21.15.174:0/301996349 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fd20401c3e0 con 0x7fd20c1008b0 2024-02-19T18:54:16.065 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:54:16.065 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:54:16.067 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:16.050+0000 7fd213897700 1 -- 172.21.15.174:0/301996349 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fd1f8038410 msgr2=0x7fd1f803a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:16.067 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:16.050+0000 7fd213897700 1 --2- 172.21.15.174:0/301996349 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fd1f8038410 0x7fd1f803a8d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc006fd0 tx=0x7fd1fc006e40 comp rx=0 tx=0).stop 2024-02-19T18:54:16.067 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:16.050+0000 7fd213897700 1 -- 172.21.15.174:0/301996349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd20c1008b0 msgr2=0x7fd20c074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:16.067 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:16.050+0000 7fd213897700 1 --2- 172.21.15.174:0/301996349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd20c1008b0 0x7fd20c074b30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd20400b830 tx=0x7fd20400bd40 comp rx=0 tx=0).stop 2024-02-19T18:54:16.067 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:16.050+0000 7fd213897700 1 -- 172.21.15.174:0/301996349 shutdown_connections 2024-02-19T18:54:16.067 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:16.050+0000 7fd213897700 1 --2- 172.21.15.174:0/301996349 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fd1f8038410 0x7fd1f803a8d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:16.068 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:16.050+0000 7fd213897700 1 --2- 172.21.15.174:0/301996349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd20c1008b0 0x7fd20c074b30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:16.068 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:16.050+0000 7fd213897700 1 -- 172.21.15.174:0/301996349 >> 172.21.15.174:0/301996349 conn(0x7fd20c0fbe30 msgr2=0x7fd20c0fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:16.068 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:16.051+0000 7fd213897700 1 -- 172.21.15.174:0/301996349 shutdown_connections 2024-02-19T18:54:16.068 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:16.051+0000 7fd213897700 1 -- 172.21.15.174:0/301996349 wait complete. 2024-02-19T18:54:16.068 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:54:16.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:16 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/301996349' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:54:17.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:17 smithi114 ceph-mon[29441]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:17.620 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:54:17.620 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:54:17.816 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:17.861 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:19.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:18 smithi114 ceph-mon[29441]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:19.385 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.368+0000 7f2a96072700 1 -- 172.21.15.174:0/1093908100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a90101eb0 msgr2=0x7f2a901022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:19.386 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.368+0000 7f2a96072700 1 --2- 172.21.15.174:0/1093908100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a90101eb0 0x7f2a901022d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800b0a0 tx=0x7f2a8800b3b0 comp rx=0 tx=0).stop 2024-02-19T18:54:19.386 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.368+0000 7f2a96072700 1 -- 172.21.15.174:0/1093908100 shutdown_connections 2024-02-19T18:54:19.386 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.368+0000 7f2a96072700 1 --2- 172.21.15.174:0/1093908100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a90101eb0 0x7f2a901022d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:19.386 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.368+0000 7f2a96072700 1 -- 172.21.15.174:0/1093908100 >> 172.21.15.174:0/1093908100 conn(0x7f2a900fa4d0 msgr2=0x7f2a900fc910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:19.386 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.368+0000 7f2a96072700 1 -- 172.21.15.174:0/1093908100 shutdown_connections 2024-02-19T18:54:19.386 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.368+0000 7f2a96072700 1 -- 172.21.15.174:0/1093908100 wait complete. 2024-02-19T18:54:19.386 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.369+0000 7f2a96072700 1 Processor -- start 2024-02-19T18:54:19.386 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.369+0000 7f2a96072700 1 -- start start 2024-02-19T18:54:19.386 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.369+0000 7f2a96072700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a90101eb0 0x7f2a901959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:19.386 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.369+0000 7f2a96072700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a90195f10 con 0x7f2a90101eb0 2024-02-19T18:54:19.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.370+0000 7f2a8f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a90101eb0 0x7f2a901959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:19.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.370+0000 7f2a8f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a90101eb0 0x7f2a901959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:34300/0 (socket says 172.21.15.174:34300) 2024-02-19T18:54:19.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.370+0000 7f2a8f7fe700 1 -- 172.21.15.174:0/2234667369 learned_addr learned my addr 172.21.15.174:0/2234667369 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:54:19.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.370+0000 7f2a8f7fe700 1 -- 172.21.15.174:0/2234667369 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a88009d00 con 0x7f2a90101eb0 2024-02-19T18:54:19.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.370+0000 7f2a8f7fe700 1 --2- 172.21.15.174:0/2234667369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a90101eb0 0x7f2a901959d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800b830 tx=0x7f2a8800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:19.388 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.370+0000 7f2a8cff9700 1 -- 172.21.15.174:0/2234667369 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a8800e040 con 0x7f2a90101eb0 2024-02-19T18:54:19.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.370+0000 7f2a8cff9700 1 -- 172.21.15.174:0/2234667369 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2a88012af0 con 0x7f2a90101eb0 2024-02-19T18:54:19.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.370+0000 7f2a96072700 1 -- 172.21.15.174:0/2234667369 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a90196110 con 0x7f2a90101eb0 2024-02-19T18:54:19.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.370+0000 7f2a8cff9700 1 -- 172.21.15.174:0/2234667369 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a8801d9f0 con 0x7f2a90101eb0 2024-02-19T18:54:19.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.370+0000 7f2a96072700 1 -- 172.21.15.174:0/2234667369 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a90196530 con 0x7f2a90101eb0 2024-02-19T18:54:19.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.371+0000 7f2a8cff9700 1 -- 172.21.15.174:0/2234667369 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f2a8801dcb0 con 0x7f2a90101eb0 2024-02-19T18:54:19.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.371+0000 7f2a96072700 1 -- 172.21.15.174:0/2234667369 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a9018f4d0 con 0x7f2a90101eb0 2024-02-19T18:54:19.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.371+0000 7f2a8cff9700 1 --2- 172.21.15.174:0/2234667369 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f2a780383c0 0x7f2a7803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:19.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.371+0000 7f2a8cff9700 1 -- 172.21.15.174:0/2234667369 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2a8804c9f0 con 0x7f2a90101eb0 2024-02-19T18:54:19.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.371+0000 7f2a8effd700 1 --2- 172.21.15.174:0/2234667369 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f2a780383c0 0x7f2a7803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:19.389 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.372+0000 7f2a8effd700 1 --2- 172.21.15.174:0/2234667369 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f2a780383c0 0x7f2a7803a880 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2a84006fd0 tx=0x7f2a84006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:19.392 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.376+0000 7f2a8cff9700 1 -- 172.21.15.174:0/2234667369 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2a8801aa00 con 0x7f2a90101eb0 2024-02-19T18:54:19.653 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.636+0000 7f2a96072700 1 -- 172.21.15.174:0/2234667369 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a900623c0 con 0x7f2a90101eb0 2024-02-19T18:54:19.653 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.637+0000 7f2a8cff9700 1 -- 172.21.15.174:0/2234667369 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f2a8801a820 con 0x7f2a90101eb0 2024-02-19T18:54:19.654 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:54:19.654 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:54:19.656 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.639+0000 7f2a96072700 1 -- 172.21.15.174:0/2234667369 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f2a780383c0 msgr2=0x7f2a7803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:19.656 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.639+0000 7f2a96072700 1 --2- 172.21.15.174:0/2234667369 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f2a780383c0 0x7f2a7803a880 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2a84006fd0 tx=0x7f2a84006e40 comp rx=0 tx=0).stop 2024-02-19T18:54:19.656 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.639+0000 7f2a96072700 1 -- 172.21.15.174:0/2234667369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a90101eb0 msgr2=0x7f2a901959d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:19.656 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.639+0000 7f2a96072700 1 --2- 172.21.15.174:0/2234667369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a90101eb0 0x7f2a901959d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800b830 tx=0x7f2a8800bd40 comp rx=0 tx=0).stop 2024-02-19T18:54:19.656 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.639+0000 7f2a96072700 1 -- 172.21.15.174:0/2234667369 shutdown_connections 2024-02-19T18:54:19.656 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.639+0000 7f2a96072700 1 --2- 172.21.15.174:0/2234667369 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f2a780383c0 0x7f2a7803a880 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:19.656 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.639+0000 7f2a96072700 1 --2- 172.21.15.174:0/2234667369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a90101eb0 0x7f2a901959d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:19.656 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.639+0000 7f2a96072700 1 -- 172.21.15.174:0/2234667369 >> 172.21.15.174:0/2234667369 conn(0x7f2a900fa4d0 msgr2=0x7f2a900fb190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:19.656 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.639+0000 7f2a96072700 1 -- 172.21.15.174:0/2234667369 shutdown_connections 2024-02-19T18:54:19.657 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:19.639+0000 7f2a96072700 1 -- 172.21.15.174:0/2234667369 wait complete. 2024-02-19T18:54:19.657 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:54:20.264 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:19 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/2234667369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:54:21.200 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:54:21.201 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:54:21.255 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:20 smithi114 ceph-mon[29441]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:21.393 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:21.440 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:22.922 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.904+0000 7f61680da700 1 -- 172.21.15.174:0/3699961666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61600fef20 msgr2=0x7f61600ff340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:22.922 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.904+0000 7f61680da700 1 --2- 172.21.15.174:0/3699961666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61600fef20 0x7f61600ff340 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f615c009910 tx=0x7f615c009c20 comp rx=0 tx=0).stop 2024-02-19T18:54:22.922 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.904+0000 7f61680da700 1 -- 172.21.15.174:0/3699961666 shutdown_connections 2024-02-19T18:54:22.922 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.904+0000 7f61680da700 1 --2- 172.21.15.174:0/3699961666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61600fef20 0x7f61600ff340 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:22.922 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.904+0000 7f61680da700 1 -- 172.21.15.174:0/3699961666 >> 172.21.15.174:0/3699961666 conn(0x7f61600fa4a0 msgr2=0x7f61600fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:22.922 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.904+0000 7f61680da700 1 -- 172.21.15.174:0/3699961666 shutdown_connections 2024-02-19T18:54:22.922 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.904+0000 7f61680da700 1 -- 172.21.15.174:0/3699961666 wait complete. 2024-02-19T18:54:22.922 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.904+0000 7f61680da700 1 Processor -- start 2024-02-19T18:54:22.923 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.904+0000 7f61680da700 1 -- start start 2024-02-19T18:54:22.923 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.904+0000 7f61680da700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61600fef20 0x7f616018be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:22.923 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.904+0000 7f61680da700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616018c3b0 con 0x7f61600fef20 2024-02-19T18:54:22.923 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.905+0000 7f6165e76700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61600fef20 0x7f616018be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:22.923 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.905+0000 7f6165e76700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61600fef20 0x7f616018be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:34316/0 (socket says 172.21.15.174:34316) 2024-02-19T18:54:22.923 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.905+0000 7f6165e76700 1 -- 172.21.15.174:0/3034656576 learned_addr learned my addr 172.21.15.174:0/3034656576 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:54:22.923 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.905+0000 7f6165e76700 1 -- 172.21.15.174:0/3034656576 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615c0095f0 con 0x7f61600fef20 2024-02-19T18:54:22.923 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.905+0000 7f6165e76700 1 --2- 172.21.15.174:0/3034656576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61600fef20 0x7f616018be70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f615c005b40 tx=0x7f615c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:22.923 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.905+0000 7f6156ffd700 1 -- 172.21.15.174:0/3034656576 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f615c01d070 con 0x7f61600fef20 2024-02-19T18:54:22.923 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.905+0000 7f6156ffd700 1 -- 172.21.15.174:0/3034656576 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f615c00be10 con 0x7f61600fef20 2024-02-19T18:54:22.923 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.905+0000 7f61680da700 1 -- 172.21.15.174:0/3034656576 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616018a4c0 con 0x7f61600fef20 2024-02-19T18:54:22.923 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.905+0000 7f6156ffd700 1 -- 172.21.15.174:0/3034656576 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f615c00f460 con 0x7f61600fef20 2024-02-19T18:54:22.923 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.905+0000 7f61680da700 1 -- 172.21.15.174:0/3034656576 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616018a960 con 0x7f61600fef20 2024-02-19T18:54:22.924 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.906+0000 7f6156ffd700 1 -- 172.21.15.174:0/3034656576 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f615c00f5c0 con 0x7f61600fef20 2024-02-19T18:54:22.924 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.906+0000 7f61680da700 1 -- 172.21.15.174:0/3034656576 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61600623c0 con 0x7f61600fef20 2024-02-19T18:54:22.924 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.906+0000 7f6156ffd700 1 --2- 172.21.15.174:0/3034656576 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f614c038410 0x7f614c03a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:22.925 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.906+0000 7f6156ffd700 1 -- 172.21.15.174:0/3034656576 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f615c04c4b0 con 0x7f61600fef20 2024-02-19T18:54:22.925 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.906+0000 7f6165675700 1 --2- 172.21.15.174:0/3034656576 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f614c038410 0x7f614c03a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:22.925 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.907+0000 7f6165675700 1 --2- 172.21.15.174:0/3034656576 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f614c038410 0x7f614c03a8d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6150006fd0 tx=0x7f6150006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:22.927 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:22.911+0000 7f6156ffd700 1 -- 172.21.15.174:0/3034656576 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f615c02c7c0 con 0x7f61600fef20 2024-02-19T18:54:23.188 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:23.171+0000 7f61680da700 1 -- 172.21.15.174:0/3034656576 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f616018f6a0 con 0x7f61600fef20 2024-02-19T18:54:23.188 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:23.172+0000 7f6156ffd700 1 -- 172.21.15.174:0/3034656576 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f615c027030 con 0x7f61600fef20 2024-02-19T18:54:23.188 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:54:23.189 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:54:23.190 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:23.174+0000 7f61680da700 1 -- 172.21.15.174:0/3034656576 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f614c038410 msgr2=0x7f614c03a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:23.191 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:23.174+0000 7f61680da700 1 --2- 172.21.15.174:0/3034656576 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f614c038410 0x7f614c03a8d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6150006fd0 tx=0x7f6150006e40 comp rx=0 tx=0).stop 2024-02-19T18:54:23.191 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:23.174+0000 7f61680da700 1 -- 172.21.15.174:0/3034656576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61600fef20 msgr2=0x7f616018be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:23.191 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:23.174+0000 7f61680da700 1 --2- 172.21.15.174:0/3034656576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61600fef20 0x7f616018be70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f615c005b40 tx=0x7f615c004740 comp rx=0 tx=0).stop 2024-02-19T18:54:23.191 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:23.174+0000 7f61680da700 1 -- 172.21.15.174:0/3034656576 shutdown_connections 2024-02-19T18:54:23.191 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:23.174+0000 7f61680da700 1 --2- 172.21.15.174:0/3034656576 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f614c038410 0x7f614c03a8d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:23.191 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:23.174+0000 7f61680da700 1 --2- 172.21.15.174:0/3034656576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61600fef20 0x7f616018be70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:23.191 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:23.174+0000 7f61680da700 1 -- 172.21.15.174:0/3034656576 >> 172.21.15.174:0/3034656576 conn(0x7f61600fa4a0 msgr2=0x7f61600fb180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:23.191 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:23.174+0000 7f61680da700 1 -- 172.21.15.174:0/3034656576 shutdown_connections 2024-02-19T18:54:23.191 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:23.174+0000 7f61680da700 1 -- 172.21.15.174:0/3034656576 wait complete. 2024-02-19T18:54:23.191 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:54:23.428 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:23 smithi114 ceph-mon[29441]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:24.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:24 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/3034656576' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:54:24.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:24 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:54:24.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:24 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:54:24.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:24 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:54:24.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:24 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:54:24.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:24 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:54:24.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:24 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:54:24.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:24 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:54:24.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:24 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:54:24.788 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:54:24.788 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:54:24.993 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:25.040 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:25.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:25 smithi114 ceph-mon[29441]: Deploying daemon prometheus.smithi114 on smithi114 2024-02-19T18:54:25.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:25 smithi114 ceph-mon[29441]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:26.413 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.396+0000 7f787dedf700 1 -- 172.21.15.174:0/557664992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78780ff7a0 msgr2=0x7f78780ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:26.413 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.396+0000 7f787dedf700 1 --2- 172.21.15.174:0/557664992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78780ff7a0 0x7f78780ffbc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7868009910 tx=0x7f7868009c20 comp rx=0 tx=0).stop 2024-02-19T18:54:26.413 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.396+0000 7f787dedf700 1 -- 172.21.15.174:0/557664992 shutdown_connections 2024-02-19T18:54:26.413 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.396+0000 7f787dedf700 1 --2- 172.21.15.174:0/557664992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78780ff7a0 0x7f78780ffbc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:26.413 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.396+0000 7f787dedf700 1 -- 172.21.15.174:0/557664992 >> 172.21.15.174:0/557664992 conn(0x7f78780fad20 msgr2=0x7f78780fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:26.413 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.396+0000 7f787dedf700 1 -- 172.21.15.174:0/557664992 shutdown_connections 2024-02-19T18:54:26.413 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.396+0000 7f787dedf700 1 -- 172.21.15.174:0/557664992 wait complete. 2024-02-19T18:54:26.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.397+0000 7f787dedf700 1 Processor -- start 2024-02-19T18:54:26.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.397+0000 7f787dedf700 1 -- start start 2024-02-19T18:54:26.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.397+0000 7f787dedf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78780ff7a0 0x7f78781948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:26.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.397+0000 7f787dedf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7878194de0 con 0x7f78780ff7a0 2024-02-19T18:54:26.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.398+0000 7f78777fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78780ff7a0 0x7f78781948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:26.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.398+0000 7f78777fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78780ff7a0 0x7f78781948a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:41830/0 (socket says 172.21.15.174:41830) 2024-02-19T18:54:26.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.398+0000 7f78777fe700 1 -- 172.21.15.174:0/432600507 learned_addr learned my addr 172.21.15.174:0/432600507 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:54:26.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.398+0000 7f78777fe700 1 -- 172.21.15.174:0/432600507 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78680095f0 con 0x7f78780ff7a0 2024-02-19T18:54:26.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.398+0000 7f78777fe700 1 --2- 172.21.15.174:0/432600507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78780ff7a0 0x7f78781948a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7868005b40 tx=0x7f7868004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:26.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.398+0000 7f78757fa700 1 -- 172.21.15.174:0/432600507 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f786801d070 con 0x7f78780ff7a0 2024-02-19T18:54:26.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.398+0000 7f78757fa700 1 -- 172.21.15.174:0/432600507 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f786800be10 con 0x7f78780ff7a0 2024-02-19T18:54:26.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.398+0000 7f787dedf700 1 -- 172.21.15.174:0/432600507 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7878194fe0 con 0x7f78780ff7a0 2024-02-19T18:54:26.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.398+0000 7f78757fa700 1 -- 172.21.15.174:0/432600507 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f786800f460 con 0x7f78780ff7a0 2024-02-19T18:54:26.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.398+0000 7f787dedf700 1 -- 172.21.15.174:0/432600507 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7878195480 con 0x7f78780ff7a0 2024-02-19T18:54:26.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.399+0000 7f78757fa700 1 -- 172.21.15.174:0/432600507 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f786800f5c0 con 0x7f78780ff7a0 2024-02-19T18:54:26.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.399+0000 7f787dedf700 1 -- 172.21.15.174:0/432600507 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f787818e3c0 con 0x7f78780ff7a0 2024-02-19T18:54:26.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.399+0000 7f78757fa700 1 --2- 172.21.15.174:0/432600507 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7864038410 0x7f786403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:26.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.399+0000 7f78757fa700 1 -- 172.21.15.174:0/432600507 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f786804c530 con 0x7f78780ff7a0 2024-02-19T18:54:26.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.399+0000 7f786effd700 1 --2- 172.21.15.174:0/432600507 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7864038410 0x7f786403a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:26.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.400+0000 7f786effd700 1 --2- 172.21.15.174:0/432600507 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7864038410 0x7f786403a8d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7860006fd0 tx=0x7f7860006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:26.420 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.404+0000 7f78757fa700 1 -- 172.21.15.174:0/432600507 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f786802a360 con 0x7f78780ff7a0 2024-02-19T18:54:26.685 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.669+0000 7f787dedf700 1 -- 172.21.15.174:0/432600507 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f787802cc70 con 0x7f78780ff7a0 2024-02-19T18:54:26.686 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.669+0000 7f78757fa700 1 -- 172.21.15.174:0/432600507 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f7868027030 con 0x7f78780ff7a0 2024-02-19T18:54:26.686 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:54:26.686 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:54:26.688 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.671+0000 7f787dedf700 1 -- 172.21.15.174:0/432600507 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7864038410 msgr2=0x7f786403a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:26.688 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.671+0000 7f787dedf700 1 --2- 172.21.15.174:0/432600507 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7864038410 0x7f786403a8d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7860006fd0 tx=0x7f7860006e40 comp rx=0 tx=0).stop 2024-02-19T18:54:26.688 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.671+0000 7f787dedf700 1 -- 172.21.15.174:0/432600507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78780ff7a0 msgr2=0x7f78781948a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:26.688 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.671+0000 7f787dedf700 1 --2- 172.21.15.174:0/432600507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78780ff7a0 0x7f78781948a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7868005b40 tx=0x7f7868004740 comp rx=0 tx=0).stop 2024-02-19T18:54:26.688 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.671+0000 7f787dedf700 1 -- 172.21.15.174:0/432600507 shutdown_connections 2024-02-19T18:54:26.688 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.671+0000 7f787dedf700 1 --2- 172.21.15.174:0/432600507 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7864038410 0x7f786403a8d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:26.689 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.671+0000 7f787dedf700 1 --2- 172.21.15.174:0/432600507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78780ff7a0 0x7f78781948a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:26.689 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.671+0000 7f787dedf700 1 -- 172.21.15.174:0/432600507 >> 172.21.15.174:0/432600507 conn(0x7f78780fad20 msgr2=0x7f78780fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:26.689 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.671+0000 7f787dedf700 1 -- 172.21.15.174:0/432600507 shutdown_connections 2024-02-19T18:54:26.689 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:26.671+0000 7f787dedf700 1 -- 172.21.15.174:0/432600507 wait complete. 2024-02-19T18:54:26.689 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:54:27.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:27 smithi114 ceph-mon[29441]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:27.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:27 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/432600507' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:54:28.276 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:54:28.276 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:54:28.478 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:28.524 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:29.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:29 smithi114 ceph-mon[29441]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:29.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:29 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:54:29.965 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.948+0000 7f7acd6ee700 1 -- 172.21.15.174:0/2124956161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac81013d0 msgr2=0x7f7ac81037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:29.965 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.948+0000 7f7acd6ee700 1 --2- 172.21.15.174:0/2124956161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac81013d0 0x7f7ac81037c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8009910 tx=0x7f7ab8009c20 comp rx=0 tx=0).stop 2024-02-19T18:54:29.965 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.948+0000 7f7acd6ee700 1 -- 172.21.15.174:0/2124956161 shutdown_connections 2024-02-19T18:54:29.965 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.948+0000 7f7acd6ee700 1 --2- 172.21.15.174:0/2124956161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac81013d0 0x7f7ac81037c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:29.965 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.948+0000 7f7acd6ee700 1 -- 172.21.15.174:0/2124956161 >> 172.21.15.174:0/2124956161 conn(0x7f7ac80fad20 msgr2=0x7f7ac80fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:29.965 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.948+0000 7f7acd6ee700 1 -- 172.21.15.174:0/2124956161 shutdown_connections 2024-02-19T18:54:29.966 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.948+0000 7f7acd6ee700 1 -- 172.21.15.174:0/2124956161 wait complete. 2024-02-19T18:54:29.967 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.949+0000 7f7acd6ee700 1 Processor -- start 2024-02-19T18:54:29.967 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.949+0000 7f7acd6ee700 1 -- start start 2024-02-19T18:54:29.967 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.949+0000 7f7acd6ee700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac81013d0 0x7f7ac8195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:29.967 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.949+0000 7f7acd6ee700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac8195ea0 con 0x7f7ac81013d0 2024-02-19T18:54:29.967 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.949+0000 7f7ac6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac81013d0 0x7f7ac8195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:29.967 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.949+0000 7f7ac6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac81013d0 0x7f7ac8195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:41850/0 (socket says 172.21.15.174:41850) 2024-02-19T18:54:29.967 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.949+0000 7f7ac6ffd700 1 -- 172.21.15.174:0/3040643438 learned_addr learned my addr 172.21.15.174:0/3040643438 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:54:29.968 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.950+0000 7f7ac6ffd700 1 -- 172.21.15.174:0/3040643438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab80095f0 con 0x7f7ac81013d0 2024-02-19T18:54:29.968 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.950+0000 7f7ac6ffd700 1 --2- 172.21.15.174:0/3040643438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac81013d0 0x7f7ac8195960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8005b40 tx=0x7f7ab8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:29.968 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.950+0000 7f7abffff700 1 -- 172.21.15.174:0/3040643438 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ab801d070 con 0x7f7ac81013d0 2024-02-19T18:54:29.968 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.950+0000 7f7acd6ee700 1 -- 172.21.15.174:0/3040643438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac81960a0 con 0x7f7ac81013d0 2024-02-19T18:54:29.968 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.950+0000 7f7abffff700 1 -- 172.21.15.174:0/3040643438 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7ab800be10 con 0x7f7ac81013d0 2024-02-19T18:54:29.968 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.950+0000 7f7acd6ee700 1 -- 172.21.15.174:0/3040643438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac8196540 con 0x7f7ac81013d0 2024-02-19T18:54:29.968 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.950+0000 7f7abffff700 1 -- 172.21.15.174:0/3040643438 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ab800f460 con 0x7f7ac81013d0 2024-02-19T18:54:29.969 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.951+0000 7f7abffff700 1 -- 172.21.15.174:0/3040643438 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f7ab80227e0 con 0x7f7ac81013d0 2024-02-19T18:54:29.969 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.951+0000 7f7acd6ee700 1 -- 172.21.15.174:0/3040643438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac818f4c0 con 0x7f7ac81013d0 2024-02-19T18:54:29.969 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.951+0000 7f7abffff700 1 --2- 172.21.15.174:0/3040643438 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7ab40383c0 0x7f7ab403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:29.969 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.951+0000 7f7abffff700 1 -- 172.21.15.174:0/3040643438 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7ab804c330 con 0x7f7ac81013d0 2024-02-19T18:54:29.969 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.951+0000 7f7ac67fc700 1 --2- 172.21.15.174:0/3040643438 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7ab40383c0 0x7f7ab403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:29.969 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.952+0000 7f7ac67fc700 1 --2- 172.21.15.174:0/3040643438 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7ab40383c0 0x7f7ab403a880 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0006fd0 tx=0x7f7ab0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:29.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:29.956+0000 7f7abffff700 1 -- 172.21.15.174:0/3040643438 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7ab8022470 con 0x7f7ac81013d0 2024-02-19T18:54:30.234 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:30.217+0000 7f7acd6ee700 1 -- 172.21.15.174:0/3040643438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ac802cc70 con 0x7f7ac81013d0 2024-02-19T18:54:30.235 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:30.218+0000 7f7abffff700 1 -- 172.21.15.174:0/3040643438 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f7ab8027030 con 0x7f7ac81013d0 2024-02-19T18:54:30.235 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:54:30.235 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:54:30.237 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:30.220+0000 7f7acd6ee700 1 -- 172.21.15.174:0/3040643438 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7ab40383c0 msgr2=0x7f7ab403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:30.237 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:30.220+0000 7f7acd6ee700 1 --2- 172.21.15.174:0/3040643438 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7ab40383c0 0x7f7ab403a880 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0006fd0 tx=0x7f7ab0006e40 comp rx=0 tx=0).stop 2024-02-19T18:54:30.237 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:30.220+0000 7f7acd6ee700 1 -- 172.21.15.174:0/3040643438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac81013d0 msgr2=0x7f7ac8195960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:30.237 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:30.220+0000 7f7acd6ee700 1 --2- 172.21.15.174:0/3040643438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac81013d0 0x7f7ac8195960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8005b40 tx=0x7f7ab8004740 comp rx=0 tx=0).stop 2024-02-19T18:54:30.237 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:30.220+0000 7f7acd6ee700 1 -- 172.21.15.174:0/3040643438 shutdown_connections 2024-02-19T18:54:30.237 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:30.220+0000 7f7acd6ee700 1 --2- 172.21.15.174:0/3040643438 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f7ab40383c0 0x7f7ab403a880 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:30.237 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:30.220+0000 7f7acd6ee700 1 --2- 172.21.15.174:0/3040643438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ac81013d0 0x7f7ac8195960 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:30.238 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:30.220+0000 7f7acd6ee700 1 -- 172.21.15.174:0/3040643438 >> 172.21.15.174:0/3040643438 conn(0x7f7ac80fad20 msgr2=0x7f7ac80fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:30.238 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:30.220+0000 7f7acd6ee700 1 -- 172.21.15.174:0/3040643438 shutdown_connections 2024-02-19T18:54:30.238 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:30.220+0000 7f7acd6ee700 1 -- 172.21.15.174:0/3040643438 wait complete. 2024-02-19T18:54:30.238 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:54:31.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:31 smithi114 ceph-mon[29441]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:31.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:31 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/3040643438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:54:31.834 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:54:31.834 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:54:32.026 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:32.072 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:33.455 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.438+0000 7f57d8f4b700 1 -- 172.21.15.174:0/1420248949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57d4101e90 msgr2=0x7f57d41022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:33.455 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.438+0000 7f57d8f4b700 1 --2- 172.21.15.174:0/1420248949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57d4101e90 0x7f57d41022b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f57c4009910 tx=0x7f57c4009c20 comp rx=0 tx=0).stop 2024-02-19T18:54:33.455 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.438+0000 7f57d8f4b700 1 -- 172.21.15.174:0/1420248949 shutdown_connections 2024-02-19T18:54:33.455 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.438+0000 7f57d8f4b700 1 --2- 172.21.15.174:0/1420248949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57d4101e90 0x7f57d41022b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:33.455 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.438+0000 7f57d8f4b700 1 -- 172.21.15.174:0/1420248949 >> 172.21.15.174:0/1420248949 conn(0x7f57d40fa4b0 msgr2=0x7f57d40fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:33.455 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.438+0000 7f57d8f4b700 1 -- 172.21.15.174:0/1420248949 shutdown_connections 2024-02-19T18:54:33.455 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.438+0000 7f57d8f4b700 1 -- 172.21.15.174:0/1420248949 wait complete. 2024-02-19T18:54:33.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.439+0000 7f57d8f4b700 1 Processor -- start 2024-02-19T18:54:33.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.439+0000 7f57d8f4b700 1 -- start start 2024-02-19T18:54:33.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.439+0000 7f57d8f4b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57d4101e90 0x7f57d41959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:33.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.439+0000 7f57d8f4b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d4195ee0 con 0x7f57d4101e90 2024-02-19T18:54:33.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.439+0000 7f57d259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57d4101e90 0x7f57d41959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:33.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.439+0000 7f57d259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57d4101e90 0x7f57d41959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:41860/0 (socket says 172.21.15.174:41860) 2024-02-19T18:54:33.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.439+0000 7f57d259c700 1 -- 172.21.15.174:0/418956172 learned_addr learned my addr 172.21.15.174:0/418956172 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:54:33.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.439+0000 7f57d259c700 1 -- 172.21.15.174:0/418956172 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c40095f0 con 0x7f57d4101e90 2024-02-19T18:54:33.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.440+0000 7f57d259c700 1 --2- 172.21.15.174:0/418956172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57d4101e90 0x7f57d41959a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f57c4005b40 tx=0x7f57c4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:33.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.440+0000 7f57c37fe700 1 -- 172.21.15.174:0/418956172 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57c401d070 con 0x7f57d4101e90 2024-02-19T18:54:33.458 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.440+0000 7f57d8f4b700 1 -- 172.21.15.174:0/418956172 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d41960e0 con 0x7f57d4101e90 2024-02-19T18:54:33.458 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.440+0000 7f57d8f4b700 1 -- 172.21.15.174:0/418956172 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d4196580 con 0x7f57d4101e90 2024-02-19T18:54:33.458 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.440+0000 7f57c37fe700 1 -- 172.21.15.174:0/418956172 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f57c400be10 con 0x7f57d4101e90 2024-02-19T18:54:33.458 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.440+0000 7f57c37fe700 1 -- 172.21.15.174:0/418956172 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57c400f460 con 0x7f57d4101e90 2024-02-19T18:54:33.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.441+0000 7f57c37fe700 1 -- 172.21.15.174:0/418956172 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f57c400f5c0 con 0x7f57d4101e90 2024-02-19T18:54:33.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.441+0000 7f57c37fe700 1 --2- 172.21.15.174:0/418956172 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f57bc038370 0x7f57bc03a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:33.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.441+0000 7f57d8f4b700 1 -- 172.21.15.174:0/418956172 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b4005320 con 0x7f57d4101e90 2024-02-19T18:54:33.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.441+0000 7f57c37fe700 1 -- 172.21.15.174:0/418956172 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f57c404c3d0 con 0x7f57d4101e90 2024-02-19T18:54:33.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.441+0000 7f57d1d9b700 1 --2- 172.21.15.174:0/418956172 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f57bc038370 0x7f57bc03a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:33.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.442+0000 7f57d1d9b700 1 --2- 172.21.15.174:0/418956172 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f57bc038370 0x7f57bc03a830 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f57c8006fd0 tx=0x7f57c8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:33.464 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.446+0000 7f57c37fe700 1 -- 172.21.15.174:0/418956172 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f57c4021c00 con 0x7f57d4101e90 2024-02-19T18:54:33.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:33 smithi114 ceph-mon[29441]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:33.726 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.709+0000 7f57d8f4b700 1 -- 172.21.15.174:0/418956172 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57b4005190 con 0x7f57d4101e90 2024-02-19T18:54:33.727 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.710+0000 7f57c37fe700 1 -- 172.21.15.174:0/418956172 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f57c40144c0 con 0x7f57d4101e90 2024-02-19T18:54:33.727 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:54:33.727 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:54:33.729 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.712+0000 7f57d8f4b700 1 -- 172.21.15.174:0/418956172 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f57bc038370 msgr2=0x7f57bc03a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:33.729 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.712+0000 7f57d8f4b700 1 --2- 172.21.15.174:0/418956172 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f57bc038370 0x7f57bc03a830 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f57c8006fd0 tx=0x7f57c8006e40 comp rx=0 tx=0).stop 2024-02-19T18:54:33.729 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.712+0000 7f57d8f4b700 1 -- 172.21.15.174:0/418956172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57d4101e90 msgr2=0x7f57d41959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:33.729 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.712+0000 7f57d8f4b700 1 --2- 172.21.15.174:0/418956172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57d4101e90 0x7f57d41959a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f57c4005b40 tx=0x7f57c4004740 comp rx=0 tx=0).stop 2024-02-19T18:54:33.729 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.712+0000 7f57d8f4b700 1 -- 172.21.15.174:0/418956172 shutdown_connections 2024-02-19T18:54:33.729 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.712+0000 7f57d8f4b700 1 --2- 172.21.15.174:0/418956172 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f57bc038370 0x7f57bc03a830 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:33.729 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.712+0000 7f57d8f4b700 1 --2- 172.21.15.174:0/418956172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57d4101e90 0x7f57d41959a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:33.729 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.712+0000 7f57d8f4b700 1 -- 172.21.15.174:0/418956172 >> 172.21.15.174:0/418956172 conn(0x7f57d40fa4b0 msgr2=0x7f57d40fb190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:33.729 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.712+0000 7f57d8f4b700 1 -- 172.21.15.174:0/418956172 shutdown_connections 2024-02-19T18:54:33.730 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:33.712+0000 7f57d8f4b700 1 -- 172.21.15.174:0/418956172 wait complete. 2024-02-19T18:54:33.730 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:54:34.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:34 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/418956172' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:54:35.340 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:54:35.340 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:54:35.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:35 smithi114 ceph-mon[29441]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:35.540 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:35.585 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:37.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.067+0000 7f60a927f700 1 -- 172.21.15.174:0/3692549536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a40ff7a0 msgr2=0x7f60a40ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:37.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.067+0000 7f60a927f700 1 --2- 172.21.15.174:0/3692549536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a40ff7a0 0x7f60a40ffbc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6094009910 tx=0x7f6094009c20 comp rx=0 tx=0).stop 2024-02-19T18:54:37.084 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.067+0000 7f60a927f700 1 -- 172.21.15.174:0/3692549536 shutdown_connections 2024-02-19T18:54:37.085 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.067+0000 7f60a927f700 1 --2- 172.21.15.174:0/3692549536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a40ff7a0 0x7f60a40ffbc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:37.085 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.067+0000 7f60a927f700 1 -- 172.21.15.174:0/3692549536 >> 172.21.15.174:0/3692549536 conn(0x7f60a40fad20 msgr2=0x7f60a40fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:37.085 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.067+0000 7f60a927f700 1 -- 172.21.15.174:0/3692549536 shutdown_connections 2024-02-19T18:54:37.085 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.067+0000 7f60a927f700 1 -- 172.21.15.174:0/3692549536 wait complete. 2024-02-19T18:54:37.085 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.067+0000 7f60a927f700 1 Processor -- start 2024-02-19T18:54:37.085 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.068+0000 7f60a927f700 1 -- start start 2024-02-19T18:54:37.085 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.068+0000 7f60a927f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a40ff7a0 0x7f60a40fcce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:37.085 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.068+0000 7f60a927f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a40fd220 con 0x7f60a40ff7a0 2024-02-19T18:54:37.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.068+0000 7f60a2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a40ff7a0 0x7f60a40fcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:37.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.068+0000 7f60a2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a40ff7a0 0x7f60a40fcce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:57326/0 (socket says 172.21.15.174:57326) 2024-02-19T18:54:37.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.068+0000 7f60a2ffd700 1 -- 172.21.15.174:0/3395522096 learned_addr learned my addr 172.21.15.174:0/3395522096 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:54:37.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.068+0000 7f60a2ffd700 1 -- 172.21.15.174:0/3395522096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60940095f0 con 0x7f60a40ff7a0 2024-02-19T18:54:37.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.069+0000 7f60a2ffd700 1 --2- 172.21.15.174:0/3395522096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a40ff7a0 0x7f60a40fcce0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6094005b40 tx=0x7f6094004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:37.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.069+0000 7f608bfff700 1 -- 172.21.15.174:0/3395522096 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f609401d070 con 0x7f60a40ff7a0 2024-02-19T18:54:37.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.069+0000 7f60a927f700 1 -- 172.21.15.174:0/3395522096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a40fb330 con 0x7f60a40ff7a0 2024-02-19T18:54:37.087 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.069+0000 7f608bfff700 1 -- 172.21.15.174:0/3395522096 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f609400be10 con 0x7f60a40ff7a0 2024-02-19T18:54:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.069+0000 7f608bfff700 1 -- 172.21.15.174:0/3395522096 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f609400f460 con 0x7f60a40ff7a0 2024-02-19T18:54:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.069+0000 7f60a927f700 1 -- 172.21.15.174:0/3395522096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a40fb7d0 con 0x7f60a40ff7a0 2024-02-19T18:54:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.070+0000 7f608bfff700 1 -- 172.21.15.174:0/3395522096 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f609400b960 con 0x7f60a40ff7a0 2024-02-19T18:54:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.070+0000 7f60a927f700 1 -- 172.21.15.174:0/3395522096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60a40623c0 con 0x7f60a40ff7a0 2024-02-19T18:54:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.070+0000 7f608bfff700 1 --2- 172.21.15.174:0/3395522096 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f608c0383c0 0x7f608c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.070+0000 7f608bfff700 1 -- 172.21.15.174:0/3395522096 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f609404c4b0 con 0x7f60a40ff7a0 2024-02-19T18:54:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.070+0000 7f60a27fc700 1 --2- 172.21.15.174:0/3395522096 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f608c0383c0 0x7f608c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:37.088 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.071+0000 7f60a27fc700 1 --2- 172.21.15.174:0/3395522096 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f608c0383c0 0x7f608c03a880 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6098006fd0 tx=0x7f6098006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:37.091 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.075+0000 7f608bfff700 1 -- 172.21.15.174:0/3395522096 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f609402a360 con 0x7f60a40ff7a0 2024-02-19T18:54:37.358 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.341+0000 7f60a927f700 1 -- 172.21.15.174:0/3395522096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60a418edb0 con 0x7f60a40ff7a0 2024-02-19T18:54:37.359 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.342+0000 7f608bfff700 1 -- 172.21.15.174:0/3395522096 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f6094027070 con 0x7f60a40ff7a0 2024-02-19T18:54:37.359 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:54:37.359 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:54:37.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.344+0000 7f60a927f700 1 -- 172.21.15.174:0/3395522096 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f608c0383c0 msgr2=0x7f608c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:37.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.344+0000 7f60a927f700 1 --2- 172.21.15.174:0/3395522096 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f608c0383c0 0x7f608c03a880 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6098006fd0 tx=0x7f6098006e40 comp rx=0 tx=0).stop 2024-02-19T18:54:37.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.344+0000 7f60a927f700 1 -- 172.21.15.174:0/3395522096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a40ff7a0 msgr2=0x7f60a40fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:37.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.344+0000 7f60a927f700 1 --2- 172.21.15.174:0/3395522096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a40ff7a0 0x7f60a40fcce0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6094005b40 tx=0x7f6094004740 comp rx=0 tx=0).stop 2024-02-19T18:54:37.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.344+0000 7f60a927f700 1 -- 172.21.15.174:0/3395522096 shutdown_connections 2024-02-19T18:54:37.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.344+0000 7f60a927f700 1 --2- 172.21.15.174:0/3395522096 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f608c0383c0 0x7f608c03a880 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:37.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.344+0000 7f60a927f700 1 --2- 172.21.15.174:0/3395522096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60a40ff7a0 0x7f60a40fcce0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:37.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.344+0000 7f60a927f700 1 -- 172.21.15.174:0/3395522096 >> 172.21.15.174:0/3395522096 conn(0x7f60a40fad20 msgr2=0x7f60a4068810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:37.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.344+0000 7f60a927f700 1 -- 172.21.15.174:0/3395522096 shutdown_connections 2024-02-19T18:54:37.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:37.344+0000 7f60a927f700 1 -- 172.21.15.174:0/3395522096 wait complete. 2024-02-19T18:54:37.362 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:54:37.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:37 smithi114 ceph-mon[29441]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:38.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:38 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/3395522096' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:54:38.947 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:54:38.948 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:54:39.148 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:39.194 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:39.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:39 smithi114 ceph-mon[29441]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:40.744 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.728+0000 7f30f60d7700 1 -- 172.21.15.174:0/1113644205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30f01008b0 msgr2=0x7f30f0100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:40.745 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.728+0000 7f30f60d7700 1 --2- 172.21.15.174:0/1113644205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30f01008b0 0x7f30f0100cd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f30e0009910 tx=0x7f30e0009c20 comp rx=0 tx=0).stop 2024-02-19T18:54:40.745 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.728+0000 7f30f60d7700 1 -- 172.21.15.174:0/1113644205 shutdown_connections 2024-02-19T18:54:40.745 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.728+0000 7f30f60d7700 1 --2- 172.21.15.174:0/1113644205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30f01008b0 0x7f30f0100cd0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:40.745 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.728+0000 7f30f60d7700 1 -- 172.21.15.174:0/1113644205 >> 172.21.15.174:0/1113644205 conn(0x7f30f00fbe30 msgr2=0x7f30f00fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:40.745 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.728+0000 7f30f60d7700 1 -- 172.21.15.174:0/1113644205 shutdown_connections 2024-02-19T18:54:40.745 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.728+0000 7f30f60d7700 1 -- 172.21.15.174:0/1113644205 wait complete. 2024-02-19T18:54:40.746 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.728+0000 7f30f60d7700 1 Processor -- start 2024-02-19T18:54:40.747 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.729+0000 7f30f60d7700 1 -- start start 2024-02-19T18:54:40.747 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.729+0000 7f30f60d7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30f01008b0 0x7f30f01959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:40.747 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.729+0000 7f30f60d7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f0195f00 con 0x7f30f01008b0 2024-02-19T18:54:40.747 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.729+0000 7f30ef7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30f01008b0 0x7f30f01959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:40.747 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.729+0000 7f30ef7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30f01008b0 0x7f30f01959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:57350/0 (socket says 172.21.15.174:57350) 2024-02-19T18:54:40.747 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.729+0000 7f30ef7fe700 1 -- 172.21.15.174:0/1088317763 learned_addr learned my addr 172.21.15.174:0/1088317763 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:54:40.747 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.729+0000 7f30ef7fe700 1 -- 172.21.15.174:0/1088317763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e00095f0 con 0x7f30f01008b0 2024-02-19T18:54:40.747 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.729+0000 7f30ef7fe700 1 --2- 172.21.15.174:0/1088317763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30f01008b0 0x7f30f01959c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f30e0005b40 tx=0x7f30e0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:40.747 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.729+0000 7f30ecff9700 1 -- 172.21.15.174:0/1088317763 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30e001d070 con 0x7f30f01008b0 2024-02-19T18:54:40.747 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.729+0000 7f30f60d7700 1 -- 172.21.15.174:0/1088317763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f0196100 con 0x7f30f01008b0 2024-02-19T18:54:40.748 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.730+0000 7f30ecff9700 1 -- 172.21.15.174:0/1088317763 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f30e000be10 con 0x7f30f01008b0 2024-02-19T18:54:40.748 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.730+0000 7f30f60d7700 1 -- 172.21.15.174:0/1088317763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f01965a0 con 0x7f30f01008b0 2024-02-19T18:54:40.748 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.730+0000 7f30ecff9700 1 -- 172.21.15.174:0/1088317763 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30e000f460 con 0x7f30f01008b0 2024-02-19T18:54:40.749 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.730+0000 7f30f60d7700 1 -- 172.21.15.174:0/1088317763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f018f4d0 con 0x7f30f01008b0 2024-02-19T18:54:40.749 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.730+0000 7f30ecff9700 1 -- 172.21.15.174:0/1088317763 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f30e00227e0 con 0x7f30f01008b0 2024-02-19T18:54:40.749 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.731+0000 7f30ecff9700 1 --2- 172.21.15.174:0/1088317763 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f30d80383c0 0x7f30d803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:40.749 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.731+0000 7f30ecff9700 1 -- 172.21.15.174:0/1088317763 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f30e004c3d0 con 0x7f30f01008b0 2024-02-19T18:54:40.749 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.731+0000 7f30eeffd700 1 --2- 172.21.15.174:0/1088317763 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f30d80383c0 0x7f30d803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:40.749 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.731+0000 7f30eeffd700 1 --2- 172.21.15.174:0/1088317763 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f30d80383c0 0x7f30d803a880 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f30e4006fd0 tx=0x7f30e4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:40.752 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.736+0000 7f30ecff9700 1 -- 172.21.15.174:0/1088317763 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f30e000f5c0 con 0x7f30f01008b0 2024-02-19T18:54:41.016 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:40.999+0000 7f30f60d7700 1 -- 172.21.15.174:0/1088317763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30f002cc70 con 0x7f30f01008b0 2024-02-19T18:54:41.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:41.000+0000 7f30ecff9700 1 -- 172.21.15.174:0/1088317763 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f30e0027030 con 0x7f30f01008b0 2024-02-19T18:54:41.017 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:54:41.017 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:54:41.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:41.002+0000 7f30f60d7700 1 -- 172.21.15.174:0/1088317763 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f30d80383c0 msgr2=0x7f30d803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:41.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:41.002+0000 7f30f60d7700 1 --2- 172.21.15.174:0/1088317763 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f30d80383c0 0x7f30d803a880 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f30e4006fd0 tx=0x7f30e4006e40 comp rx=0 tx=0).stop 2024-02-19T18:54:41.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:41.002+0000 7f30f60d7700 1 -- 172.21.15.174:0/1088317763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30f01008b0 msgr2=0x7f30f01959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:41.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:41.002+0000 7f30f60d7700 1 --2- 172.21.15.174:0/1088317763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30f01008b0 0x7f30f01959c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f30e0005b40 tx=0x7f30e0004740 comp rx=0 tx=0).stop 2024-02-19T18:54:41.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:41.002+0000 7f30f60d7700 1 -- 172.21.15.174:0/1088317763 shutdown_connections 2024-02-19T18:54:41.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:41.002+0000 7f30f60d7700 1 --2- 172.21.15.174:0/1088317763 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f30d80383c0 0x7f30d803a880 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:41.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:41.002+0000 7f30f60d7700 1 --2- 172.21.15.174:0/1088317763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f30f01008b0 0x7f30f01959c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:41.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:41.002+0000 7f30f60d7700 1 -- 172.21.15.174:0/1088317763 >> 172.21.15.174:0/1088317763 conn(0x7f30f00fbe30 msgr2=0x7f30f00fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:41.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:41.002+0000 7f30f60d7700 1 -- 172.21.15.174:0/1088317763 shutdown_connections 2024-02-19T18:54:41.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:41.002+0000 7f30f60d7700 1 -- 172.21.15.174:0/1088317763 wait complete. 2024-02-19T18:54:41.020 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:54:41.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:41 smithi114 ceph-mon[29441]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:41.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:41 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/1088317763' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:54:42.653 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:54:42.653 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:54:42.859 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:42.907 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:43.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:43 smithi114 ceph-mon[29441]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:44.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.195+0000 7fee6be48700 1 -- 172.21.15.174:0/4194786080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee640ff790 msgr2=0x7fee640ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:44.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.195+0000 7fee6be48700 1 --2- 172.21.15.174:0/4194786080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee640ff790 0x7fee640ffbb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fee54009910 tx=0x7fee54009c20 comp rx=0 tx=0).stop 2024-02-19T18:54:44.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.195+0000 7fee6be48700 1 -- 172.21.15.174:0/4194786080 shutdown_connections 2024-02-19T18:54:44.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.195+0000 7fee6be48700 1 --2- 172.21.15.174:0/4194786080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee640ff790 0x7fee640ffbb0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:44.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.195+0000 7fee6be48700 1 -- 172.21.15.174:0/4194786080 >> 172.21.15.174:0/4194786080 conn(0x7fee640fad30 msgr2=0x7fee640fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:44.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.195+0000 7fee6be48700 1 -- 172.21.15.174:0/4194786080 shutdown_connections 2024-02-19T18:54:44.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.195+0000 7fee6be48700 1 -- 172.21.15.174:0/4194786080 wait complete. 2024-02-19T18:54:44.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.195+0000 7fee6be48700 1 Processor -- start 2024-02-19T18:54:44.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.195+0000 7fee6be48700 1 -- start start 2024-02-19T18:54:44.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.196+0000 7fee6be48700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee640ff790 0x7fee641959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:44.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.196+0000 7fee6be48700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee64195ee0 con 0x7fee640ff790 2024-02-19T18:54:44.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.196+0000 7fee69be4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee640ff790 0x7fee641959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:44.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.196+0000 7fee69be4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee640ff790 0x7fee641959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:57368/0 (socket says 172.21.15.174:57368) 2024-02-19T18:54:44.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.196+0000 7fee69be4700 1 -- 172.21.15.174:0/442342733 learned_addr learned my addr 172.21.15.174:0/442342733 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:54:44.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.196+0000 7fee69be4700 1 -- 172.21.15.174:0/442342733 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee540095f0 con 0x7fee640ff790 2024-02-19T18:54:44.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.196+0000 7fee69be4700 1 --2- 172.21.15.174:0/442342733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee640ff790 0x7fee641959a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fee54005b40 tx=0x7fee54004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:44.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.196+0000 7fee5affd700 1 -- 172.21.15.174:0/442342733 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee5401d070 con 0x7fee640ff790 2024-02-19T18:54:44.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.196+0000 7fee6be48700 1 -- 172.21.15.174:0/442342733 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee641960e0 con 0x7fee640ff790 2024-02-19T18:54:44.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.196+0000 7fee5affd700 1 -- 172.21.15.174:0/442342733 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fee5400be10 con 0x7fee640ff790 2024-02-19T18:54:44.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.196+0000 7fee5affd700 1 -- 172.21.15.174:0/442342733 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee5400f460 con 0x7fee640ff790 2024-02-19T18:54:44.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.196+0000 7fee6be48700 1 -- 172.21.15.174:0/442342733 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee64196580 con 0x7fee640ff790 2024-02-19T18:54:44.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.197+0000 7fee5affd700 1 -- 172.21.15.174:0/442342733 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fee5400f6d0 con 0x7fee640ff790 2024-02-19T18:54:44.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.197+0000 7fee5affd700 1 --2- 172.21.15.174:0/442342733 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fee50038410 0x7fee5003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:44.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.197+0000 7fee5affd700 1 -- 172.21.15.174:0/442342733 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fee5404c4b0 con 0x7fee640ff790 2024-02-19T18:54:44.216 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.197+0000 7fee6be48700 1 -- 172.21.15.174:0/442342733 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee641055d0 con 0x7fee640ff790 2024-02-19T18:54:44.216 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.198+0000 7fee693e3700 1 --2- 172.21.15.174:0/442342733 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fee50038410 0x7fee5003a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:44.216 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.198+0000 7fee693e3700 1 --2- 172.21.15.174:0/442342733 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fee50038410 0x7fee5003a8d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fee60006fd0 tx=0x7fee60006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:44.219 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.202+0000 7fee5affd700 1 -- 172.21.15.174:0/442342733 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fee54027070 con 0x7fee640ff790 2024-02-19T18:54:44.488 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.471+0000 7fee6be48700 1 -- 172.21.15.174:0/442342733 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee641057c0 con 0x7fee640ff790 2024-02-19T18:54:44.488 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.471+0000 7fee5affd700 1 -- 172.21.15.174:0/442342733 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fee5402c440 con 0x7fee640ff790 2024-02-19T18:54:44.488 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:54:44.488 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:54:44.490 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.473+0000 7fee6be48700 1 -- 172.21.15.174:0/442342733 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fee50038410 msgr2=0x7fee5003a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:44.490 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.473+0000 7fee6be48700 1 --2- 172.21.15.174:0/442342733 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fee50038410 0x7fee5003a8d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fee60006fd0 tx=0x7fee60006e40 comp rx=0 tx=0).stop 2024-02-19T18:54:44.490 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.473+0000 7fee6be48700 1 -- 172.21.15.174:0/442342733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee640ff790 msgr2=0x7fee641959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:44.490 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.473+0000 7fee6be48700 1 --2- 172.21.15.174:0/442342733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee640ff790 0x7fee641959a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fee54005b40 tx=0x7fee54004740 comp rx=0 tx=0).stop 2024-02-19T18:54:44.490 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.473+0000 7fee6be48700 1 -- 172.21.15.174:0/442342733 shutdown_connections 2024-02-19T18:54:44.490 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.473+0000 7fee6be48700 1 --2- 172.21.15.174:0/442342733 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fee50038410 0x7fee5003a8d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:44.490 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.473+0000 7fee6be48700 1 --2- 172.21.15.174:0/442342733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee640ff790 0x7fee641959a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:44.490 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.473+0000 7fee6be48700 1 -- 172.21.15.174:0/442342733 >> 172.21.15.174:0/442342733 conn(0x7fee640fad30 msgr2=0x7fee640fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:44.490 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.473+0000 7fee6be48700 1 -- 172.21.15.174:0/442342733 shutdown_connections 2024-02-19T18:54:44.490 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:44.473+0000 7fee6be48700 1 -- 172.21.15.174:0/442342733 wait complete. 2024-02-19T18:54:44.491 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:54:45.479 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:45 smithi114 ceph-mon[29441]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:45.479 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:45 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/442342733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:54:46.089 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:54:46.090 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:54:46.292 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:46.339 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:47.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:47 smithi114 ceph-mon[29441]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:47.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.932+0000 7f52fa081700 1 -- 172.21.15.174:0/3301421651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f40ff110 msgr2=0x7f52f4101540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:47.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.932+0000 7f52fa081700 1 --2- 172.21.15.174:0/3301421651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f40ff110 0x7f52f4101540 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f52e4009910 tx=0x7f52e4009c20 comp rx=0 tx=0).stop 2024-02-19T18:54:47.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.932+0000 7f52fa081700 1 -- 172.21.15.174:0/3301421651 shutdown_connections 2024-02-19T18:54:47.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.932+0000 7f52fa081700 1 --2- 172.21.15.174:0/3301421651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f40ff110 0x7f52f4101540 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:47.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.932+0000 7f52fa081700 1 -- 172.21.15.174:0/3301421651 >> 172.21.15.174:0/3301421651 conn(0x7f52f40fad30 msgr2=0x7f52f40fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:47.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.932+0000 7f52fa081700 1 -- 172.21.15.174:0/3301421651 shutdown_connections 2024-02-19T18:54:47.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.932+0000 7f52fa081700 1 -- 172.21.15.174:0/3301421651 wait complete. 2024-02-19T18:54:47.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.932+0000 7f52fa081700 1 Processor -- start 2024-02-19T18:54:47.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.932+0000 7f52fa081700 1 -- start start 2024-02-19T18:54:47.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.932+0000 7f52fa081700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f40ff110 0x7f52f419bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:47.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.932+0000 7f52fa081700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f419c4d0 con 0x7f52f40ff110 2024-02-19T18:54:47.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.933+0000 7f52f37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f40ff110 0x7f52f419bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:47.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.933+0000 7f52f37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f40ff110 0x7f52f419bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:57634/0 (socket says 172.21.15.174:57634) 2024-02-19T18:54:47.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.933+0000 7f52f37fe700 1 -- 172.21.15.174:0/2755419644 learned_addr learned my addr 172.21.15.174:0/2755419644 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:54:47.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.933+0000 7f52f37fe700 1 -- 172.21.15.174:0/2755419644 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e40095f0 con 0x7f52f40ff110 2024-02-19T18:54:47.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.933+0000 7f52f37fe700 1 --2- 172.21.15.174:0/2755419644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f40ff110 0x7f52f419bf90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f52e4005b40 tx=0x7f52e4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:47.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.933+0000 7f52f17fa700 1 -- 172.21.15.174:0/2755419644 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52e401d070 con 0x7f52f40ff110 2024-02-19T18:54:47.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.933+0000 7f52fa081700 1 -- 172.21.15.174:0/2755419644 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f419c6d0 con 0x7f52f40ff110 2024-02-19T18:54:47.951 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.933+0000 7f52f17fa700 1 -- 172.21.15.174:0/2755419644 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f52e400be10 con 0x7f52f40ff110 2024-02-19T18:54:47.952 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.933+0000 7f52f17fa700 1 -- 172.21.15.174:0/2755419644 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52e400f460 con 0x7f52f40ff110 2024-02-19T18:54:47.952 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.933+0000 7f52fa081700 1 -- 172.21.15.174:0/2755419644 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f419cb70 con 0x7f52f40ff110 2024-02-19T18:54:47.953 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.934+0000 7f52f17fa700 1 -- 172.21.15.174:0/2755419644 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f52e40227e0 con 0x7f52f40ff110 2024-02-19T18:54:47.953 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.934+0000 7f52fa081700 1 -- 172.21.15.174:0/2755419644 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f4196400 con 0x7f52f40ff110 2024-02-19T18:54:47.953 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.934+0000 7f52f17fa700 1 --2- 172.21.15.174:0/2755419644 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f52e00383c0 0x7f52e003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:47.953 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.934+0000 7f52f17fa700 1 -- 172.21.15.174:0/2755419644 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f52e404c3f0 con 0x7f52f40ff110 2024-02-19T18:54:47.953 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.935+0000 7f52ebfff700 1 --2- 172.21.15.174:0/2755419644 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f52e00383c0 0x7f52e003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:47.953 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.935+0000 7f52ebfff700 1 --2- 172.21.15.174:0/2755419644 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f52e00383c0 0x7f52e003a880 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f52dc006fd0 tx=0x7f52dc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:47.956 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:47.940+0000 7f52f17fa700 1 -- 172.21.15.174:0/2755419644 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f52e400f5c0 con 0x7f52f40ff110 2024-02-19T18:54:48.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:48.206+0000 7f52fa081700 1 -- 172.21.15.174:0/2755419644 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52f402cc70 con 0x7f52f40ff110 2024-02-19T18:54:48.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:48.207+0000 7f52f17fa700 1 -- 172.21.15.174:0/2755419644 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f52e4027030 con 0x7f52f40ff110 2024-02-19T18:54:48.223 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:54:48.223 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:54:48.225 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:48.209+0000 7f52fa081700 1 -- 172.21.15.174:0/2755419644 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f52e00383c0 msgr2=0x7f52e003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:48.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:48.209+0000 7f52fa081700 1 --2- 172.21.15.174:0/2755419644 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f52e00383c0 0x7f52e003a880 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f52dc006fd0 tx=0x7f52dc006e40 comp rx=0 tx=0).stop 2024-02-19T18:54:48.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:48.209+0000 7f52fa081700 1 -- 172.21.15.174:0/2755419644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f40ff110 msgr2=0x7f52f419bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:48.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:48.209+0000 7f52fa081700 1 --2- 172.21.15.174:0/2755419644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f40ff110 0x7f52f419bf90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f52e4005b40 tx=0x7f52e4004740 comp rx=0 tx=0).stop 2024-02-19T18:54:48.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:48.209+0000 7f52fa081700 1 -- 172.21.15.174:0/2755419644 shutdown_connections 2024-02-19T18:54:48.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:48.209+0000 7f52fa081700 1 --2- 172.21.15.174:0/2755419644 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f52e00383c0 0x7f52e003a880 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:48.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:48.209+0000 7f52fa081700 1 --2- 172.21.15.174:0/2755419644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52f40ff110 0x7f52f419bf90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:48.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:48.209+0000 7f52fa081700 1 -- 172.21.15.174:0/2755419644 >> 172.21.15.174:0/2755419644 conn(0x7f52f40fad30 msgr2=0x7f52f40fb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:48.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:48.209+0000 7f52fa081700 1 -- 172.21.15.174:0/2755419644 shutdown_connections 2024-02-19T18:54:48.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:48.209+0000 7f52fa081700 1 -- 172.21.15.174:0/2755419644 wait complete. 2024-02-19T18:54:48.227 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:54:49.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:48 smithi114 ceph-mon[29441]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:49.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:48 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/2755419644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:54:49.868 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:54:49.868 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:54:50.062 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:50.109 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:51.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:51 smithi114 ceph-mon[29441]: pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:51.546 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.529+0000 7fcd81ab5700 1 -- 172.21.15.174:0/3066283008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd7c0ff7a0 msgr2=0x7fcd7c0ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:51.546 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.529+0000 7fcd81ab5700 1 --2- 172.21.15.174:0/3066283008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd7c0ff7a0 0x7fcd7c0ffbc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c009910 tx=0x7fcd6c009c20 comp rx=0 tx=0).stop 2024-02-19T18:54:51.546 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.529+0000 7fcd81ab5700 1 -- 172.21.15.174:0/3066283008 shutdown_connections 2024-02-19T18:54:51.547 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.529+0000 7fcd81ab5700 1 --2- 172.21.15.174:0/3066283008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd7c0ff7a0 0x7fcd7c0ffbc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:51.547 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.529+0000 7fcd81ab5700 1 -- 172.21.15.174:0/3066283008 >> 172.21.15.174:0/3066283008 conn(0x7fcd7c0fad20 msgr2=0x7fcd7c0fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:51.547 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.529+0000 7fcd81ab5700 1 -- 172.21.15.174:0/3066283008 shutdown_connections 2024-02-19T18:54:51.547 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.529+0000 7fcd81ab5700 1 -- 172.21.15.174:0/3066283008 wait complete. 2024-02-19T18:54:51.547 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.530+0000 7fcd81ab5700 1 Processor -- start 2024-02-19T18:54:51.549 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.530+0000 7fcd81ab5700 1 -- start start 2024-02-19T18:54:51.549 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.530+0000 7fcd81ab5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd7c0ff7a0 0x7fcd7c0fcce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:51.549 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.530+0000 7fcd81ab5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c0fd220 con 0x7fcd7c0ff7a0 2024-02-19T18:54:51.549 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.530+0000 7fcd7b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd7c0ff7a0 0x7fcd7c0fcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:51.549 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.531+0000 7fcd7b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd7c0ff7a0 0x7fcd7c0fcce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:57638/0 (socket says 172.21.15.174:57638) 2024-02-19T18:54:51.549 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.531+0000 7fcd7b7fe700 1 -- 172.21.15.174:0/4105469896 learned_addr learned my addr 172.21.15.174:0/4105469896 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:54:51.550 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.531+0000 7fcd7b7fe700 1 -- 172.21.15.174:0/4105469896 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c0095f0 con 0x7fcd7c0ff7a0 2024-02-19T18:54:51.550 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.531+0000 7fcd7b7fe700 1 --2- 172.21.15.174:0/4105469896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd7c0ff7a0 0x7fcd7c0fcce0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c005b40 tx=0x7fcd6c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:51.550 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.531+0000 7fcd78ff9700 1 -- 172.21.15.174:0/4105469896 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd6c01d070 con 0x7fcd7c0ff7a0 2024-02-19T18:54:51.550 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.531+0000 7fcd81ab5700 1 -- 172.21.15.174:0/4105469896 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7c0fb330 con 0x7fcd7c0ff7a0 2024-02-19T18:54:51.550 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.531+0000 7fcd78ff9700 1 -- 172.21.15.174:0/4105469896 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcd6c00be10 con 0x7fcd7c0ff7a0 2024-02-19T18:54:51.550 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.531+0000 7fcd78ff9700 1 -- 172.21.15.174:0/4105469896 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd6c00f460 con 0x7fcd7c0ff7a0 2024-02-19T18:54:51.550 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.531+0000 7fcd81ab5700 1 -- 172.21.15.174:0/4105469896 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7c0fb7d0 con 0x7fcd7c0ff7a0 2024-02-19T18:54:51.550 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.532+0000 7fcd81ab5700 1 -- 172.21.15.174:0/4105469896 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7c0623c0 con 0x7fcd7c0ff7a0 2024-02-19T18:54:51.550 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.532+0000 7fcd78ff9700 1 -- 172.21.15.174:0/4105469896 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fcd6c0227e0 con 0x7fcd7c0ff7a0 2024-02-19T18:54:51.550 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.532+0000 7fcd78ff9700 1 --2- 172.21.15.174:0/4105469896 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fcd640383c0 0x7fcd6403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:51.550 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.532+0000 7fcd78ff9700 1 -- 172.21.15.174:0/4105469896 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcd6c04c430 con 0x7fcd7c0ff7a0 2024-02-19T18:54:51.551 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.533+0000 7fcd7affd700 1 --2- 172.21.15.174:0/4105469896 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fcd640383c0 0x7fcd6403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:51.551 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.533+0000 7fcd7affd700 1 --2- 172.21.15.174:0/4105469896 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fcd640383c0 0x7fcd6403a880 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcd70006fd0 tx=0x7fcd70006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:51.554 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.537+0000 7fcd78ff9700 1 -- 172.21.15.174:0/4105469896 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fcd6c02bba0 con 0x7fcd7c0ff7a0 2024-02-19T18:54:51.817 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.800+0000 7fcd81ab5700 1 -- 172.21.15.174:0/4105469896 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd7c18edb0 con 0x7fcd7c0ff7a0 2024-02-19T18:54:51.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.801+0000 7fcd78ff9700 1 -- 172.21.15.174:0/4105469896 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fcd6c027030 con 0x7fcd7c0ff7a0 2024-02-19T18:54:51.818 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:54:51.818 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:54:51.820 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.803+0000 7fcd81ab5700 1 -- 172.21.15.174:0/4105469896 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fcd640383c0 msgr2=0x7fcd6403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:51.820 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.803+0000 7fcd81ab5700 1 --2- 172.21.15.174:0/4105469896 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fcd640383c0 0x7fcd6403a880 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcd70006fd0 tx=0x7fcd70006e40 comp rx=0 tx=0).stop 2024-02-19T18:54:51.820 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.803+0000 7fcd81ab5700 1 -- 172.21.15.174:0/4105469896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd7c0ff7a0 msgr2=0x7fcd7c0fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:51.820 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.803+0000 7fcd81ab5700 1 --2- 172.21.15.174:0/4105469896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd7c0ff7a0 0x7fcd7c0fcce0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c005b40 tx=0x7fcd6c004740 comp rx=0 tx=0).stop 2024-02-19T18:54:51.820 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.803+0000 7fcd81ab5700 1 -- 172.21.15.174:0/4105469896 shutdown_connections 2024-02-19T18:54:51.820 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.803+0000 7fcd81ab5700 1 --2- 172.21.15.174:0/4105469896 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fcd640383c0 0x7fcd6403a880 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:51.820 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.803+0000 7fcd81ab5700 1 --2- 172.21.15.174:0/4105469896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd7c0ff7a0 0x7fcd7c0fcce0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:51.820 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.803+0000 7fcd81ab5700 1 -- 172.21.15.174:0/4105469896 >> 172.21.15.174:0/4105469896 conn(0x7fcd7c0fad20 msgr2=0x7fcd7c069100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:51.821 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.803+0000 7fcd81ab5700 1 -- 172.21.15.174:0/4105469896 shutdown_connections 2024-02-19T18:54:51.821 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:51.803+0000 7fcd81ab5700 1 -- 172.21.15.174:0/4105469896 wait complete. 2024-02-19T18:54:51.821 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:54:52.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:52 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/4105469896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:54:53.436 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:53 smithi114 ceph-mon[29441]: pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:53.546 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:54:53.546 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:54:53.747 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:53.794 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:55.246 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.229+0000 7fc200cb4700 1 -- 172.21.15.174:0/2594204706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1fc1008a0 msgr2=0x7fc1fc100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:55.246 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.229+0000 7fc200cb4700 1 --2- 172.21.15.174:0/2594204706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1fc1008a0 0x7fc1fc100cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0009910 tx=0x7fc1f0009c20 comp rx=0 tx=0).stop 2024-02-19T18:54:55.246 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.229+0000 7fc200cb4700 1 -- 172.21.15.174:0/2594204706 shutdown_connections 2024-02-19T18:54:55.246 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.229+0000 7fc200cb4700 1 --2- 172.21.15.174:0/2594204706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1fc1008a0 0x7fc1fc100cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:55.246 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.229+0000 7fc200cb4700 1 -- 172.21.15.174:0/2594204706 >> 172.21.15.174:0/2594204706 conn(0x7fc1fc0fbe60 msgr2=0x7fc1fc0fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:55.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.229+0000 7fc200cb4700 1 -- 172.21.15.174:0/2594204706 shutdown_connections 2024-02-19T18:54:55.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.229+0000 7fc200cb4700 1 -- 172.21.15.174:0/2594204706 wait complete. 2024-02-19T18:54:55.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.229+0000 7fc200cb4700 1 Processor -- start 2024-02-19T18:54:55.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.230+0000 7fc200cb4700 1 -- start start 2024-02-19T18:54:55.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.230+0000 7fc200cb4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1fc1008a0 0x7fc1fc1959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:55.247 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.230+0000 7fc200cb4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1fc195f00 con 0x7fc1fc1008a0 2024-02-19T18:54:55.248 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.230+0000 7fc1fa59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1fc1008a0 0x7fc1fc1959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:55.248 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.230+0000 7fc1fa59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1fc1008a0 0x7fc1fc1959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:60954/0 (socket says 172.21.15.174:60954) 2024-02-19T18:54:55.248 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.230+0000 7fc1fa59c700 1 -- 172.21.15.174:0/3017950911 learned_addr learned my addr 172.21.15.174:0/3017950911 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:54:55.249 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.230+0000 7fc1fa59c700 1 -- 172.21.15.174:0/3017950911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f00095f0 con 0x7fc1fc1008a0 2024-02-19T18:54:55.249 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.230+0000 7fc1fa59c700 1 --2- 172.21.15.174:0/3017950911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1fc1008a0 0x7fc1fc1959c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0005950 tx=0x7fc1f0009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:55.249 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.230+0000 7fc1eb7fe700 1 -- 172.21.15.174:0/3017950911 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1f001d070 con 0x7fc1fc1008a0 2024-02-19T18:54:55.249 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.231+0000 7fc200cb4700 1 -- 172.21.15.174:0/3017950911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1fc196100 con 0x7fc1fc1008a0 2024-02-19T18:54:55.249 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.231+0000 7fc1eb7fe700 1 -- 172.21.15.174:0/3017950911 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc1f000be10 con 0x7fc1fc1008a0 2024-02-19T18:54:55.249 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.231+0000 7fc1eb7fe700 1 -- 172.21.15.174:0/3017950911 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1f000f460 con 0x7fc1fc1008a0 2024-02-19T18:54:55.249 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.231+0000 7fc200cb4700 1 -- 172.21.15.174:0/3017950911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1fc1965a0 con 0x7fc1fc1008a0 2024-02-19T18:54:55.249 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.231+0000 7fc1eb7fe700 1 -- 172.21.15.174:0/3017950911 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fc1f00227c0 con 0x7fc1fc1008a0 2024-02-19T18:54:55.249 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.231+0000 7fc200cb4700 1 -- 172.21.15.174:0/3017950911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1fc18f4c0 con 0x7fc1fc1008a0 2024-02-19T18:54:55.249 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.232+0000 7fc1eb7fe700 1 --2- 172.21.15.174:0/3017950911 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fc1e40383c0 0x7fc1e403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:55.249 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.232+0000 7fc1eb7fe700 1 -- 172.21.15.174:0/3017950911 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc1f004c4e0 con 0x7fc1fc1008a0 2024-02-19T18:54:55.249 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.232+0000 7fc1f9d9b700 1 --2- 172.21.15.174:0/3017950911 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fc1e40383c0 0x7fc1e403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:55.250 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.232+0000 7fc1f9d9b700 1 --2- 172.21.15.174:0/3017950911 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fc1e40383c0 0x7fc1e403a880 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec006fd0 tx=0x7fc1ec006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:55.253 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.237+0000 7fc1eb7fe700 1 -- 172.21.15.174:0/3017950911 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc1f002bba0 con 0x7fc1fc1008a0 2024-02-19T18:54:55.515 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.498+0000 7fc200cb4700 1 -- 172.21.15.174:0/3017950911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1fc02cc70 con 0x7fc1fc1008a0 2024-02-19T18:54:55.515 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.499+0000 7fc1eb7fe700 1 -- 172.21.15.174:0/3017950911 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fc1f0027030 con 0x7fc1fc1008a0 2024-02-19T18:54:55.515 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:54:55.516 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:54:55.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.501+0000 7fc200cb4700 1 -- 172.21.15.174:0/3017950911 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fc1e40383c0 msgr2=0x7fc1e403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:55.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.501+0000 7fc200cb4700 1 --2- 172.21.15.174:0/3017950911 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fc1e40383c0 0x7fc1e403a880 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec006fd0 tx=0x7fc1ec006e40 comp rx=0 tx=0).stop 2024-02-19T18:54:55.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.501+0000 7fc200cb4700 1 -- 172.21.15.174:0/3017950911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1fc1008a0 msgr2=0x7fc1fc1959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:55.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.501+0000 7fc200cb4700 1 --2- 172.21.15.174:0/3017950911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1fc1008a0 0x7fc1fc1959c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0005950 tx=0x7fc1f0009e60 comp rx=0 tx=0).stop 2024-02-19T18:54:55.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.501+0000 7fc200cb4700 1 -- 172.21.15.174:0/3017950911 shutdown_connections 2024-02-19T18:54:55.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.501+0000 7fc200cb4700 1 --2- 172.21.15.174:0/3017950911 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7fc1e40383c0 0x7fc1e403a880 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:55.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.501+0000 7fc200cb4700 1 --2- 172.21.15.174:0/3017950911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1fc1008a0 0x7fc1fc1959c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:55.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.501+0000 7fc200cb4700 1 -- 172.21.15.174:0/3017950911 >> 172.21.15.174:0/3017950911 conn(0x7fc1fc0fbe60 msgr2=0x7fc1fc0fcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:55.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.501+0000 7fc200cb4700 1 -- 172.21.15.174:0/3017950911 shutdown_connections 2024-02-19T18:54:55.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:55.501+0000 7fc200cb4700 1 -- 172.21.15.174:0/3017950911 wait complete. 2024-02-19T18:54:55.519 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:54:55.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:55 smithi114 ceph-mon[29441]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:55.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:55 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:54:55.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:55 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:54:55.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:55 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:54:55.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:55 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-19T18:54:56.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:56 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/3017950911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:54:56.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:56 smithi114 ceph-mon[29441]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-19T18:54:56.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:56 smithi114 ceph-mon[29441]: mgrmap e14: smithi114.pcvtrk(active, since 2m) 2024-02-19T18:54:57.209 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:54:57.210 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:54:57.414 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:57.460 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:54:58.969 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.951+0000 7f2556e26700 1 -- 172.21.15.174:0/1927463031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25500ff120 msgr2=0x7f2550101550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:58.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.951+0000 7f2556e26700 1 --2- 172.21.15.174:0/1927463031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25500ff120 0x7f2550101550 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f254c009910 tx=0x7f254c009c20 comp rx=0 tx=0).stop 2024-02-19T18:54:58.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.951+0000 7f2556e26700 1 -- 172.21.15.174:0/1927463031 shutdown_connections 2024-02-19T18:54:58.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.951+0000 7f2556e26700 1 --2- 172.21.15.174:0/1927463031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25500ff120 0x7f2550101550 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:58.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.951+0000 7f2556e26700 1 -- 172.21.15.174:0/1927463031 >> 172.21.15.174:0/1927463031 conn(0x7f25500fad20 msgr2=0x7f25500fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:58.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.951+0000 7f2556e26700 1 -- 172.21.15.174:0/1927463031 shutdown_connections 2024-02-19T18:54:58.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.951+0000 7f2556e26700 1 -- 172.21.15.174:0/1927463031 wait complete. 2024-02-19T18:54:58.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.952+0000 7f2556e26700 1 Processor -- start 2024-02-19T18:54:58.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.952+0000 7f2556e26700 1 -- start start 2024-02-19T18:54:58.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.952+0000 7f2556e26700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25500ff120 0x7f255019bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:58.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.952+0000 7f2556e26700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255019c4a0 con 0x7f25500ff120 2024-02-19T18:54:58.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.952+0000 7f2554bc2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25500ff120 0x7f255019bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:58.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.952+0000 7f2554bc2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25500ff120 0x7f255019bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:60976/0 (socket says 172.21.15.174:60976) 2024-02-19T18:54:58.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.952+0000 7f2554bc2700 1 -- 172.21.15.174:0/3525426355 learned_addr learned my addr 172.21.15.174:0/3525426355 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:54:58.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.952+0000 7f2554bc2700 1 -- 172.21.15.174:0/3525426355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254c0095f0 con 0x7f25500ff120 2024-02-19T18:54:58.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.952+0000 7f2554bc2700 1 --2- 172.21.15.174:0/3525426355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25500ff120 0x7f255019bf60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f254c005b40 tx=0x7f254c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:58.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.953+0000 7f2545ffb700 1 -- 172.21.15.174:0/3525426355 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f254c01d070 con 0x7f25500ff120 2024-02-19T18:54:58.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.953+0000 7f2545ffb700 1 -- 172.21.15.174:0/3525426355 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f254c00be10 con 0x7f25500ff120 2024-02-19T18:54:58.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.953+0000 7f2545ffb700 1 -- 172.21.15.174:0/3525426355 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f254c00f460 con 0x7f25500ff120 2024-02-19T18:54:58.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.953+0000 7f2556e26700 1 -- 172.21.15.174:0/3525426355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f255019c6a0 con 0x7f25500ff120 2024-02-19T18:54:58.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.953+0000 7f2556e26700 1 -- 172.21.15.174:0/3525426355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f255019cb40 con 0x7f25500ff120 2024-02-19T18:54:58.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.954+0000 7f2545ffb700 1 -- 172.21.15.174:0/3525426355 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 14) v1 ==== 45354+0+0 (secure 0 0 0) 0x7f254c0227e0 con 0x7f25500ff120 2024-02-19T18:54:58.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.954+0000 7f2556e26700 1 -- 172.21.15.174:0/3525426355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2550196410 con 0x7f25500ff120 2024-02-19T18:54:58.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.954+0000 7f2545ffb700 1 --2- 172.21.15.174:0/3525426355 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f2538038460 0x7f253803a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:58.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.954+0000 7f2545ffb700 1 -- 172.21.15.174:0/3525426355 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f254c04c4d0 con 0x7f25500ff120 2024-02-19T18:54:58.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.954+0000 7f2547fff700 1 -- 172.21.15.174:0/3525426355 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f2538038460 msgr2=0x7f253803a920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/1719431365 2024-02-19T18:54:58.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.954+0000 7f2547fff700 1 --2- 172.21.15.174:0/3525426355 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f2538038460 0x7f253803a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:54:58.975 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:58.959+0000 7f2545ffb700 1 -- 172.21.15.174:0/3525426355 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f254c02ba00 con 0x7f25500ff120 2024-02-19T18:54:59.172 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:59.155+0000 7f2547fff700 1 -- 172.21.15.174:0/3525426355 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f2538038460 msgr2=0x7f253803a920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/1719431365 2024-02-19T18:54:59.172 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:59.155+0000 7f2547fff700 1 --2- 172.21.15.174:0/3525426355 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f2538038460 0x7f253803a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T18:54:59.241 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:59.224+0000 7f2556e26700 1 -- 172.21.15.174:0/3525426355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f255002cc70 con 0x7f25500ff120 2024-02-19T18:54:59.241 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:59.225+0000 7f2545ffb700 1 -- 172.21.15.174:0/3525426355 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f254c027030 con 0x7f25500ff120 2024-02-19T18:54:59.242 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:54:59.242 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:54:59.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:59.227+0000 7f2556e26700 1 -- 172.21.15.174:0/3525426355 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f2538038460 msgr2=0x7f253803a920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:54:59.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:59.227+0000 7f2556e26700 1 --2- 172.21.15.174:0/3525426355 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f2538038460 0x7f253803a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:59.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:59.227+0000 7f2556e26700 1 -- 172.21.15.174:0/3525426355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25500ff120 msgr2=0x7f255019bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:59.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:59.227+0000 7f2556e26700 1 --2- 172.21.15.174:0/3525426355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25500ff120 0x7f255019bf60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f254c005b40 tx=0x7f254c004740 comp rx=0 tx=0).stop 2024-02-19T18:54:59.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:59.227+0000 7f2556e26700 1 -- 172.21.15.174:0/3525426355 shutdown_connections 2024-02-19T18:54:59.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:59.227+0000 7f2556e26700 1 --2- 172.21.15.174:0/3525426355 >> [v2:172.21.15.114:6800/1719431365,v1:172.21.15.114:6801/1719431365] conn(0x7f2538038460 0x7f253803a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:59.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:59.227+0000 7f2556e26700 1 --2- 172.21.15.174:0/3525426355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25500ff120 0x7f255019bf60 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:59.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:59.227+0000 7f2556e26700 1 -- 172.21.15.174:0/3525426355 >> 172.21.15.174:0/3525426355 conn(0x7f25500fad20 msgr2=0x7f25500fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:59.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:59.227+0000 7f2556e26700 1 -- 172.21.15.174:0/3525426355 shutdown_connections 2024-02-19T18:54:59.244 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:54:59.227+0000 7f2556e26700 1 -- 172.21.15.174:0/3525426355 wait complete. 2024-02-19T18:54:59.245 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:54:59.617 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:54:59 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/3525426355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:00.900 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:55:00.901 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:55:01.103 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:55:01.149 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:55:02.637 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.620+0000 7fb66cd16700 1 -- 172.21.15.174:0/1397259079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6680feb50 msgr2=0x7fb6680fef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:02.637 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.620+0000 7fb66cd16700 1 --2- 172.21.15.174:0/1397259079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6680feb50 0x7fb6680fef70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb660009910 tx=0x7fb660009c20 comp rx=0 tx=0).stop 2024-02-19T18:55:02.637 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.620+0000 7fb66cd16700 1 -- 172.21.15.174:0/1397259079 shutdown_connections 2024-02-19T18:55:02.638 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.620+0000 7fb66cd16700 1 --2- 172.21.15.174:0/1397259079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6680feb50 0x7fb6680fef70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:02.638 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.620+0000 7fb66cd16700 1 -- 172.21.15.174:0/1397259079 >> 172.21.15.174:0/1397259079 conn(0x7fb6680fa4b0 msgr2=0x7fb6680fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:02.638 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.620+0000 7fb66cd16700 1 -- 172.21.15.174:0/1397259079 shutdown_connections 2024-02-19T18:55:02.638 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.620+0000 7fb66cd16700 1 -- 172.21.15.174:0/1397259079 wait complete. 2024-02-19T18:55:02.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.621+0000 7fb66cd16700 1 Processor -- start 2024-02-19T18:55:02.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.621+0000 7fb66cd16700 1 -- start start 2024-02-19T18:55:02.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.621+0000 7fb66cd16700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6680feb50 0x7fb6681959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:02.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.621+0000 7fb66cd16700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb668195f00 con 0x7fb6680feb50 2024-02-19T18:55:02.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.622+0000 7fb66659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6680feb50 0x7fb6681959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:02.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.622+0000 7fb66659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6680feb50 0x7fb6681959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:60998/0 (socket says 172.21.15.174:60998) 2024-02-19T18:55:02.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.622+0000 7fb66659c700 1 -- 172.21.15.174:0/1163781147 learned_addr learned my addr 172.21.15.174:0/1163781147 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:55:02.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.622+0000 7fb66659c700 1 -- 172.21.15.174:0/1163781147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6600095f0 con 0x7fb6680feb50 2024-02-19T18:55:02.640 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.622+0000 7fb66659c700 1 --2- 172.21.15.174:0/1163781147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6680feb50 0x7fb6681959c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb660005950 tx=0x7fb660009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:02.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.622+0000 7fb6577fe700 1 -- 172.21.15.174:0/1163781147 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb66001d070 con 0x7fb6680feb50 2024-02-19T18:55:02.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.622+0000 7fb6577fe700 1 -- 172.21.15.174:0/1163781147 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb66000be10 con 0x7fb6680feb50 2024-02-19T18:55:02.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.622+0000 7fb66cd16700 1 -- 172.21.15.174:0/1163781147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb668196100 con 0x7fb6680feb50 2024-02-19T18:55:02.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.622+0000 7fb6577fe700 1 -- 172.21.15.174:0/1163781147 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb66000f460 con 0x7fb6680feb50 2024-02-19T18:55:02.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.622+0000 7fb66cd16700 1 -- 172.21.15.174:0/1163781147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6681965a0 con 0x7fb6680feb50 2024-02-19T18:55:02.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.623+0000 7fb6577fe700 1 -- 172.21.15.174:0/1163781147 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 15) v1 ==== 45077+0+0 (secure 0 0 0) 0x7fb66000f690 con 0x7fb6680feb50 2024-02-19T18:55:02.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.623+0000 7fb6577fe700 1 -- 172.21.15.174:0/1163781147 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fb66004bfe0 con 0x7fb6680feb50 2024-02-19T18:55:02.641 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.623+0000 7fb66cd16700 1 -- 172.21.15.174:0/1163781147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6680ff680 con 0x7fb6680feb50 2024-02-19T18:55:02.645 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.628+0000 7fb6577fe700 1 -- 172.21.15.174:0/1163781147 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb6680ff680 con 0x7fb6680feb50 2024-02-19T18:55:02.917 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.899+0000 7fb66cd16700 1 -- 172.21.15.174:0/1163781147 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6680623c0 con 0x7fb6680feb50 2024-02-19T18:55:02.917 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.900+0000 7fb6577fe700 1 -- 172.21.15.174:0/1163781147 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fb660027030 con 0x7fb6680feb50 2024-02-19T18:55:02.917 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:55:02.917 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:55:02.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.902+0000 7fb66cd16700 1 -- 172.21.15.174:0/1163781147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6680feb50 msgr2=0x7fb6681959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:02.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.902+0000 7fb66cd16700 1 --2- 172.21.15.174:0/1163781147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6680feb50 0x7fb6681959c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb660005950 tx=0x7fb660009e60 comp rx=0 tx=0).stop 2024-02-19T18:55:02.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.902+0000 7fb66cd16700 1 -- 172.21.15.174:0/1163781147 shutdown_connections 2024-02-19T18:55:02.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.902+0000 7fb66cd16700 1 --2- 172.21.15.174:0/1163781147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb6680feb50 0x7fb6681959c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:02.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.902+0000 7fb66cd16700 1 -- 172.21.15.174:0/1163781147 >> 172.21.15.174:0/1163781147 conn(0x7fb6680fa4b0 msgr2=0x7fb6680fb190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:02.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.902+0000 7fb66cd16700 1 -- 172.21.15.174:0/1163781147 shutdown_connections 2024-02-19T18:55:02.919 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:02.902+0000 7fb66cd16700 1 -- 172.21.15.174:0/1163781147 wait complete. 2024-02-19T18:55:02.920 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:55:03.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:02 smithi114 ceph-mon[29441]: Active manager daemon smithi114.pcvtrk restarted 2024-02-19T18:55:03.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:02 smithi114 ceph-mon[29441]: Activating manager daemon smithi114.pcvtrk 2024-02-19T18:55:03.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:02 smithi114 ceph-mon[29441]: osdmap e5: 0 total, 0 up, 0 in 2024-02-19T18:55:03.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:02 smithi114 ceph-mon[29441]: mgrmap e15: smithi114.pcvtrk(active, starting, since 0.00324487s) 2024-02-19T18:55:03.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:02 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T18:55:03.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:02 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr metadata", "who": "smithi114.pcvtrk", "id": "smithi114.pcvtrk"}]: dispatch 2024-02-19T18:55:03.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:02 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T18:55:03.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:02 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T18:55:03.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:02 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T18:55:03.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:02 smithi114 ceph-mon[29441]: Manager daemon smithi114.pcvtrk is now available 2024-02-19T18:55:03.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:02 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:03.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:02 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:04.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:03 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:04.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:03 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:55:04.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:03 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T18:55:04.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:03 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/trash_purge_schedule"}]: dispatch 2024-02-19T18:55:04.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:03 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/1163781147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:04.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:03 smithi114 ceph-mon[29441]: mgrmap e16: smithi114.pcvtrk(active, since 1.00527s) 2024-02-19T18:55:04.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:03 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:04.538 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:55:04.538 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:55:04.736 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:55:04.790 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:55:05.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:04 smithi114 ceph-mon[29441]: [19/Feb/2024:18:55:03] ENGINE Bus STARTING 2024-02-19T18:55:05.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:04 smithi114 ceph-mon[29441]: [19/Feb/2024:18:55:03] ENGINE Serving on http://172.21.15.114:8765 2024-02-19T18:55:05.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:04 smithi114 ceph-mon[29441]: [19/Feb/2024:18:55:03] ENGINE Serving on https://172.21.15.114:7150 2024-02-19T18:55:05.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:04 smithi114 ceph-mon[29441]: [19/Feb/2024:18:55:03] ENGINE Bus STARTED 2024-02-19T18:55:05.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:04 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:05.862 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:05 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:05.862 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:05 smithi114 ceph-mon[29441]: mgrmap e17: smithi114.pcvtrk(active, since 2s) 2024-02-19T18:55:06.527 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.509+0000 7fcc3597b700 1 -- 172.21.15.174:0/1291244107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc301008a0 msgr2=0x7fcc30100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:06.527 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.509+0000 7fcc3597b700 1 --2- 172.21.15.174:0/1291244107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc301008a0 0x7fcc30100cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcc20009910 tx=0x7fcc20009c20 comp rx=0 tx=0).stop 2024-02-19T18:55:06.527 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.509+0000 7fcc3597b700 1 -- 172.21.15.174:0/1291244107 shutdown_connections 2024-02-19T18:55:06.527 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.509+0000 7fcc3597b700 1 --2- 172.21.15.174:0/1291244107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc301008a0 0x7fcc30100cc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:06.528 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.509+0000 7fcc3597b700 1 -- 172.21.15.174:0/1291244107 >> 172.21.15.174:0/1291244107 conn(0x7fcc300fbe60 msgr2=0x7fcc300fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:06.528 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.509+0000 7fcc3597b700 1 -- 172.21.15.174:0/1291244107 shutdown_connections 2024-02-19T18:55:06.528 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.509+0000 7fcc3597b700 1 -- 172.21.15.174:0/1291244107 wait complete. 2024-02-19T18:55:06.528 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.509+0000 7fcc3597b700 1 Processor -- start 2024-02-19T18:55:06.528 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.510+0000 7fcc3597b700 1 -- start start 2024-02-19T18:55:06.528 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.510+0000 7fcc3597b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc301008a0 0x7fcc301959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:06.528 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.510+0000 7fcc3597b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc30195f00 con 0x7fcc301008a0 2024-02-19T18:55:06.528 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.510+0000 7fcc2effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc301008a0 0x7fcc301959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:06.528 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.510+0000 7fcc2effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc301008a0 0x7fcc301959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:33584/0 (socket says 172.21.15.174:33584) 2024-02-19T18:55:06.528 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.510+0000 7fcc2effd700 1 -- 172.21.15.174:0/3750725451 learned_addr learned my addr 172.21.15.174:0/3750725451 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:55:06.528 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.510+0000 7fcc2effd700 1 -- 172.21.15.174:0/3750725451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc200095f0 con 0x7fcc301008a0 2024-02-19T18:55:06.528 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.510+0000 7fcc2effd700 1 --2- 172.21.15.174:0/3750725451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc301008a0 0x7fcc301959c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcc20005b40 tx=0x7fcc20004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:06.529 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.510+0000 7fcc34979700 1 -- 172.21.15.174:0/3750725451 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc2001d070 con 0x7fcc301008a0 2024-02-19T18:55:06.529 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.510+0000 7fcc3597b700 1 -- 172.21.15.174:0/3750725451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc30196100 con 0x7fcc301008a0 2024-02-19T18:55:06.529 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.510+0000 7fcc34979700 1 -- 172.21.15.174:0/3750725451 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcc2000be10 con 0x7fcc301008a0 2024-02-19T18:55:06.529 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.511+0000 7fcc34979700 1 -- 172.21.15.174:0/3750725451 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc2000f460 con 0x7fcc301008a0 2024-02-19T18:55:06.529 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.511+0000 7fcc3597b700 1 -- 172.21.15.174:0/3750725451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc301965a0 con 0x7fcc301008a0 2024-02-19T18:55:06.530 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.511+0000 7fcc34979700 1 -- 172.21.15.174:0/3750725451 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7fcc2000f6d0 con 0x7fcc301008a0 2024-02-19T18:55:06.530 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.511+0000 7fcc3597b700 1 -- 172.21.15.174:0/3750725451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3018f4c0 con 0x7fcc301008a0 2024-02-19T18:55:06.530 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.511+0000 7fcc34979700 1 --2- 172.21.15.174:0/3750725451 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fcc180384b0 0x7fcc1803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:06.530 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.512+0000 7fcc34979700 1 -- 172.21.15.174:0/3750725451 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fcc2004c720 con 0x7fcc301008a0 2024-02-19T18:55:06.530 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.512+0000 7fcc2e7fc700 1 --2- 172.21.15.174:0/3750725451 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fcc180384b0 0x7fcc1803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:06.530 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.512+0000 7fcc2e7fc700 1 --2- 172.21.15.174:0/3750725451 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fcc180384b0 0x7fcc1803a970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcc24006fd0 tx=0x7fcc24006e40 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:06.533 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.516+0000 7fcc34979700 1 -- 172.21.15.174:0/3750725451 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fcc20020d40 con 0x7fcc301008a0 2024-02-19T18:55:06.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.788+0000 7fcc3597b700 1 -- 172.21.15.174:0/3750725451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc3002cc70 con 0x7fcc301008a0 2024-02-19T18:55:06.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.788+0000 7fcc34979700 1 -- 172.21.15.174:0/3750725451 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fcc20027030 con 0x7fcc301008a0 2024-02-19T18:55:06.805 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:55:06.805 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:55:06.807 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.790+0000 7fcc3597b700 1 -- 172.21.15.174:0/3750725451 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fcc180384b0 msgr2=0x7fcc1803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:06.807 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.790+0000 7fcc3597b700 1 --2- 172.21.15.174:0/3750725451 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fcc180384b0 0x7fcc1803a970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcc24006fd0 tx=0x7fcc24006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:06.807 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.790+0000 7fcc3597b700 1 -- 172.21.15.174:0/3750725451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc301008a0 msgr2=0x7fcc301959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:06.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.790+0000 7fcc3597b700 1 --2- 172.21.15.174:0/3750725451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc301008a0 0x7fcc301959c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcc20005b40 tx=0x7fcc20004740 comp rx=0 tx=0).stop 2024-02-19T18:55:06.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.791+0000 7fcc3597b700 1 -- 172.21.15.174:0/3750725451 shutdown_connections 2024-02-19T18:55:06.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.791+0000 7fcc3597b700 1 --2- 172.21.15.174:0/3750725451 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fcc180384b0 0x7fcc1803a970 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:06.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.791+0000 7fcc3597b700 1 --2- 172.21.15.174:0/3750725451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc301008a0 0x7fcc301959c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:06.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.791+0000 7fcc3597b700 1 -- 172.21.15.174:0/3750725451 >> 172.21.15.174:0/3750725451 conn(0x7fcc300fbe60 msgr2=0x7fcc300fcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:06.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.791+0000 7fcc3597b700 1 -- 172.21.15.174:0/3750725451 shutdown_connections 2024-02-19T18:55:06.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:06.791+0000 7fcc3597b700 1 -- 172.21.15.174:0/3750725451 wait complete. 2024-02-19T18:55:06.808 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:55:07.150 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:06 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/3750725451' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:08.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:07 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:08.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:07 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:08.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:07 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:08.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:07 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:08.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:07 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:55:08.410 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:55:08.411 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:55:08.614 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:55:08.670 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:55:10.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:10 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:10.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:10 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:10.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:10 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:10.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:10 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:10.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:10 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:55:10.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:10 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:10.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:10 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:55:10.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:10 smithi114 ceph-mon[29441]: Updating smithi114:/etc/ceph/ceph.conf 2024-02-19T18:55:10.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:10 smithi114 ceph-mon[29441]: Updating smithi174:/etc/ceph/ceph.conf 2024-02-19T18:55:10.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:10 smithi114 ceph-mon[29441]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:55:10.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:10 smithi114 ceph-mon[29441]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:55:11.008 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.985+0000 7f10be62b700 1 -- 172.21.15.174:0/1407848236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b8105740 msgr2=0x7f10b8105b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:11.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.985+0000 7f10be62b700 1 --2- 172.21.15.174:0/1407848236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b8105740 0x7f10b8105b20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f10ac009910 tx=0x7f10ac009c20 comp rx=0 tx=0).stop 2024-02-19T18:55:11.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.985+0000 7f10be62b700 1 -- 172.21.15.174:0/1407848236 shutdown_connections 2024-02-19T18:55:11.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.985+0000 7f10be62b700 1 --2- 172.21.15.174:0/1407848236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b8105740 0x7f10b8105b20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:11.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.985+0000 7f10be62b700 1 -- 172.21.15.174:0/1407848236 >> 172.21.15.174:0/1407848236 conn(0x7f10b80fb250 msgr2=0x7f10b80fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:11.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.985+0000 7f10be62b700 1 -- 172.21.15.174:0/1407848236 shutdown_connections 2024-02-19T18:55:11.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.985+0000 7f10be62b700 1 -- 172.21.15.174:0/1407848236 wait complete. 2024-02-19T18:55:11.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.985+0000 7f10be62b700 1 Processor -- start 2024-02-19T18:55:11.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.985+0000 7f10be62b700 1 -- start start 2024-02-19T18:55:11.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.986+0000 7f10be62b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b8105740 0x7f10b8195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:11.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.986+0000 7f10be62b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b81962f0 con 0x7f10b8105740 2024-02-19T18:55:11.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.986+0000 7f10b7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b8105740 0x7f10b8195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:11.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.986+0000 7f10b7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b8105740 0x7f10b8195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:33604/0 (socket says 172.21.15.174:33604) 2024-02-19T18:55:11.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.986+0000 7f10b7fff700 1 -- 172.21.15.174:0/1998393146 learned_addr learned my addr 172.21.15.174:0/1998393146 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:55:11.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.986+0000 7f10b7fff700 1 -- 172.21.15.174:0/1998393146 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ac0095f0 con 0x7f10b8105740 2024-02-19T18:55:11.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.986+0000 7f10b7fff700 1 --2- 172.21.15.174:0/1998393146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b8105740 0x7f10b8195db0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f10ac005b40 tx=0x7f10ac004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:11.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.987+0000 7f10b57fa700 1 -- 172.21.15.174:0/1998393146 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10ac01d070 con 0x7f10b8105740 2024-02-19T18:55:11.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.987+0000 7f10b57fa700 1 -- 172.21.15.174:0/1998393146 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f10ac0044d0 con 0x7f10b8105740 2024-02-19T18:55:11.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.987+0000 7f10be62b700 1 -- 172.21.15.174:0/1998393146 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b81964f0 con 0x7f10b8105740 2024-02-19T18:55:11.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.987+0000 7f10b57fa700 1 -- 172.21.15.174:0/1998393146 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10ac003910 con 0x7f10b8105740 2024-02-19T18:55:11.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.987+0000 7f10be62b700 1 -- 172.21.15.174:0/1998393146 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b8196970 con 0x7f10b8105740 2024-02-19T18:55:11.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.987+0000 7f10b57fa700 1 -- 172.21.15.174:0/1998393146 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f10ac003b40 con 0x7f10b8105740 2024-02-19T18:55:11.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.988+0000 7f10b57fa700 1 --2- 172.21.15.174:0/1998393146 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f10a0038500 0x7f10a003a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:11.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.988+0000 7f10b57fa700 1 -- 172.21.15.174:0/1998393146 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f10ac04c3c0 con 0x7f10b8105740 2024-02-19T18:55:11.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.988+0000 7f10be62b700 1 -- 172.21.15.174:0/1998393146 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b80623c0 con 0x7f10b8105740 2024-02-19T18:55:11.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.988+0000 7f10b77fe700 1 --2- 172.21.15.174:0/1998393146 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f10a0038500 0x7f10a003a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:11.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.988+0000 7f10b77fe700 1 --2- 172.21.15.174:0/1998393146 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f10a0038500 0x7f10a003a9c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f10a8006fd0 tx=0x7f10a8006e40 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:11.011 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:10.993+0000 7f10b57fa700 1 -- 172.21.15.174:0/1998393146 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f10ac02a950 con 0x7f10b8105740 2024-02-19T18:55:11.296 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:11.279+0000 7f10be62b700 1 -- 172.21.15.174:0/1998393146 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10b819a7c0 con 0x7f10b8105740 2024-02-19T18:55:11.297 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:11.280+0000 7f10b57fa700 1 -- 172.21.15.174:0/1998393146 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f10ac02a360 con 0x7f10b8105740 2024-02-19T18:55:11.297 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:55:11.297 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:55:11.299 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:11.282+0000 7f10be62b700 1 -- 172.21.15.174:0/1998393146 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f10a0038500 msgr2=0x7f10a003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:11.299 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:11.282+0000 7f10be62b700 1 --2- 172.21.15.174:0/1998393146 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f10a0038500 0x7f10a003a9c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f10a8006fd0 tx=0x7f10a8006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:11.299 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:11.282+0000 7f10be62b700 1 -- 172.21.15.174:0/1998393146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b8105740 msgr2=0x7f10b8195db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:11.299 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:11.282+0000 7f10be62b700 1 --2- 172.21.15.174:0/1998393146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b8105740 0x7f10b8195db0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f10ac005b40 tx=0x7f10ac004c80 comp rx=0 tx=0).stop 2024-02-19T18:55:11.299 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:11.282+0000 7f10be62b700 1 -- 172.21.15.174:0/1998393146 shutdown_connections 2024-02-19T18:55:11.299 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:11.282+0000 7f10be62b700 1 --2- 172.21.15.174:0/1998393146 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f10a0038500 0x7f10a003a9c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:11.299 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:11.282+0000 7f10be62b700 1 --2- 172.21.15.174:0/1998393146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10b8105740 0x7f10b8195db0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:11.300 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:11.282+0000 7f10be62b700 1 -- 172.21.15.174:0/1998393146 >> 172.21.15.174:0/1998393146 conn(0x7f10b80fb250 msgr2=0x7f10b8107b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:11.300 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:11.282+0000 7f10be62b700 1 -- 172.21.15.174:0/1998393146 shutdown_connections 2024-02-19T18:55:11.300 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:11.282+0000 7f10be62b700 1 -- 172.21.15.174:0/1998393146 wait complete. 2024-02-19T18:55:11.300 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:55:12.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:11 smithi114 ceph-mon[29441]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-02-19T18:55:12.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:11 smithi114 ceph-mon[29441]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-02-19T18:55:12.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:11 smithi114 ceph-mon[29441]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T18:55:12.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:11 smithi114 ceph-mon[29441]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T18:55:12.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:11 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:12.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:11 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:12.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:11 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:12.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:11 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:12.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:11 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:12.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:11 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi174", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T18:55:12.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:11 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi174", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-02-19T18:55:12.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:11 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:12.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:11 smithi114 ceph-mon[29441]: Deploying daemon ceph-exporter.smithi174 on smithi174 2024-02-19T18:55:12.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:11 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/1998393146' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:13.354 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:55:13.354 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:55:13.671 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:55:15.194 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.176+0000 7f477e47f700 1 -- 172.21.15.174:0/1044907122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4778106830 msgr2=0x7f4778106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:15.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.176+0000 7f477e47f700 1 --2- 172.21.15.174:0/1044907122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4778106830 0x7f4778106c10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4768009910 tx=0x7f4768009c20 comp rx=0 tx=0).stop 2024-02-19T18:55:15.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.176+0000 7f477e47f700 1 -- 172.21.15.174:0/1044907122 shutdown_connections 2024-02-19T18:55:15.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.176+0000 7f477e47f700 1 --2- 172.21.15.174:0/1044907122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4778106830 0x7f4778106c10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:15.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.176+0000 7f477e47f700 1 -- 172.21.15.174:0/1044907122 >> 172.21.15.174:0/1044907122 conn(0x7f47780fc380 msgr2=0x7f47780fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:15.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.176+0000 7f477e47f700 1 -- 172.21.15.174:0/1044907122 shutdown_connections 2024-02-19T18:55:15.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.176+0000 7f477e47f700 1 -- 172.21.15.174:0/1044907122 wait complete. 2024-02-19T18:55:15.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.176+0000 7f477e47f700 1 Processor -- start 2024-02-19T18:55:15.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.176+0000 7f477e47f700 1 -- start start 2024-02-19T18:55:15.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.177+0000 7f477e47f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4778106830 0x7f4778073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:15.195 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.177+0000 7f477e47f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4778075010 con 0x7f4778106830 2024-02-19T18:55:15.196 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.177+0000 7f4777fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4778106830 0x7f4778073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:15.196 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.177+0000 7f4777fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4778106830 0x7f4778073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:41750/0 (socket says 172.21.15.174:41750) 2024-02-19T18:55:15.196 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.177+0000 7f4777fff700 1 -- 172.21.15.174:0/678344833 learned_addr learned my addr 172.21.15.174:0/678344833 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:55:15.196 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.178+0000 7f4777fff700 1 -- 172.21.15.174:0/678344833 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47680095f0 con 0x7f4778106830 2024-02-19T18:55:15.197 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.178+0000 7f4777fff700 1 --2- 172.21.15.174:0/678344833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4778106830 0x7f4778073090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4768005b40 tx=0x7f4768004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:15.197 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.178+0000 7f47757fa700 1 -- 172.21.15.174:0/678344833 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f476801d070 con 0x7f4778106830 2024-02-19T18:55:15.197 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.178+0000 7f47757fa700 1 -- 172.21.15.174:0/678344833 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f47680044d0 con 0x7f4778106830 2024-02-19T18:55:15.197 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.178+0000 7f477e47f700 1 -- 172.21.15.174:0/678344833 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47780735d0 con 0x7f4778106830 2024-02-19T18:55:15.197 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.178+0000 7f47757fa700 1 -- 172.21.15.174:0/678344833 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4768003930 con 0x7f4778106830 2024-02-19T18:55:15.197 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.178+0000 7f477e47f700 1 -- 172.21.15.174:0/678344833 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4778073a50 con 0x7f4778106830 2024-02-19T18:55:15.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.179+0000 7f47757fa700 1 -- 172.21.15.174:0/678344833 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f4768003bf0 con 0x7f4778106830 2024-02-19T18:55:15.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.179+0000 7f477e47f700 1 -- 172.21.15.174:0/678344833 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4778104ae0 con 0x7f4778106830 2024-02-19T18:55:15.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.179+0000 7f47757fa700 1 --2- 172.21.15.174:0/678344833 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f47600384b0 0x7f476003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:15.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.179+0000 7f47757fa700 1 -- 172.21.15.174:0/678344833 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f476804c6e0 con 0x7f4778106830 2024-02-19T18:55:15.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.180+0000 7f47777fe700 1 --2- 172.21.15.174:0/678344833 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f47600384b0 0x7f476003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:15.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.180+0000 7f47777fe700 1 --2- 172.21.15.174:0/678344833 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f47600384b0 0x7f476003a970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f476c006fd0 tx=0x7f476c006e40 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:15.202 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.184+0000 7f47757fa700 1 -- 172.21.15.174:0/678344833 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f476802a950 con 0x7f4778106830 2024-02-19T18:55:15.484 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.465+0000 7f477e47f700 1 -- 172.21.15.174:0/678344833 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4778073db0 con 0x7f4778106830 2024-02-19T18:55:15.485 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.466+0000 7f47757fa700 1 -- 172.21.15.174:0/678344833 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f476802a360 con 0x7f4778106830 2024-02-19T18:55:15.485 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:55:15.485 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:55:15.486 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.468+0000 7f477e47f700 1 -- 172.21.15.174:0/678344833 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f47600384b0 msgr2=0x7f476003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:15.486 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.468+0000 7f477e47f700 1 --2- 172.21.15.174:0/678344833 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f47600384b0 0x7f476003a970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f476c006fd0 tx=0x7f476c006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:15.486 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.468+0000 7f477e47f700 1 -- 172.21.15.174:0/678344833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4778106830 msgr2=0x7f4778073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:15.486 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.468+0000 7f477e47f700 1 --2- 172.21.15.174:0/678344833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4778106830 0x7f4778073090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4768005b40 tx=0x7f4768004c80 comp rx=0 tx=0).stop 2024-02-19T18:55:15.488 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.468+0000 7f477e47f700 1 -- 172.21.15.174:0/678344833 shutdown_connections 2024-02-19T18:55:15.488 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.468+0000 7f477e47f700 1 --2- 172.21.15.174:0/678344833 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f47600384b0 0x7f476003a970 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:15.489 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.468+0000 7f477e47f700 1 --2- 172.21.15.174:0/678344833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4778106830 0x7f4778073090 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:15.489 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.468+0000 7f477e47f700 1 -- 172.21.15.174:0/678344833 >> 172.21.15.174:0/678344833 conn(0x7f47780fc380 msgr2=0x7f4778108c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:15.489 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.468+0000 7f477e47f700 1 -- 172.21.15.174:0/678344833 shutdown_connections 2024-02-19T18:55:15.489 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:15.468+0000 7f477e47f700 1 -- 172.21.15.174:0/678344833 wait complete. 2024-02-19T18:55:15.490 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:55:15.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:15 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/678344833' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:17.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:17.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:17.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:17.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:17.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi174", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:55:17.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi174", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-19T18:55:17.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:17.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:16 smithi114 ceph-mon[29441]: Deploying daemon crash.smithi174 on smithi174 2024-02-19T18:55:17.267 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:55:17.267 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:55:17.478 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:55:18.892 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.873+0000 7f0361bbb700 1 -- 172.21.15.174:0/4157090113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f035c0ff120 msgr2=0x7f035c0ff500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:18.892 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.873+0000 7f0361bbb700 1 --2- 172.21.15.174:0/4157090113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f035c0ff120 0x7f035c0ff500 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f034c009910 tx=0x7f034c009c20 comp rx=0 tx=0).stop 2024-02-19T18:55:18.892 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.873+0000 7f0361bbb700 1 -- 172.21.15.174:0/4157090113 shutdown_connections 2024-02-19T18:55:18.892 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.873+0000 7f0361bbb700 1 --2- 172.21.15.174:0/4157090113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f035c0ff120 0x7f035c0ff500 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:18.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.873+0000 7f0361bbb700 1 -- 172.21.15.174:0/4157090113 >> 172.21.15.174:0/4157090113 conn(0x7f035c075df0 msgr2=0x7f035c076200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:18.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.873+0000 7f0361bbb700 1 -- 172.21.15.174:0/4157090113 shutdown_connections 2024-02-19T18:55:18.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.873+0000 7f0361bbb700 1 -- 172.21.15.174:0/4157090113 wait complete. 2024-02-19T18:55:18.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.873+0000 7f0361bbb700 1 Processor -- start 2024-02-19T18:55:18.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.874+0000 7f0361bbb700 1 -- start start 2024-02-19T18:55:18.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.874+0000 7f0361bbb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f035c0ff120 0x7f035c195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:18.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.874+0000 7f0361bbb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035c196310 con 0x7f035c0ff120 2024-02-19T18:55:18.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.874+0000 7f035b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f035c0ff120 0x7f035c195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:18.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.874+0000 7f035b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f035c0ff120 0x7f035c195dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:41772/0 (socket says 172.21.15.174:41772) 2024-02-19T18:55:18.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.874+0000 7f035b7fe700 1 -- 172.21.15.174:0/2587139701 learned_addr learned my addr 172.21.15.174:0/2587139701 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:55:18.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.874+0000 7f035b7fe700 1 -- 172.21.15.174:0/2587139701 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c0095f0 con 0x7f035c0ff120 2024-02-19T18:55:18.893 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.875+0000 7f035b7fe700 1 --2- 172.21.15.174:0/2587139701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f035c0ff120 0x7f035c195dd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f034c005b40 tx=0x7f034c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:18.894 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.875+0000 7f0358ff9700 1 -- 172.21.15.174:0/2587139701 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f034c01d070 con 0x7f035c0ff120 2024-02-19T18:55:18.895 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.875+0000 7f0361bbb700 1 -- 172.21.15.174:0/2587139701 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f035c196510 con 0x7f035c0ff120 2024-02-19T18:55:18.895 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.875+0000 7f0358ff9700 1 -- 172.21.15.174:0/2587139701 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f034c0044d0 con 0x7f035c0ff120 2024-02-19T18:55:18.895 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.875+0000 7f0361bbb700 1 -- 172.21.15.174:0/2587139701 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f035c1968d0 con 0x7f035c0ff120 2024-02-19T18:55:18.895 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.875+0000 7f0358ff9700 1 -- 172.21.15.174:0/2587139701 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f034c003930 con 0x7f035c0ff120 2024-02-19T18:55:18.895 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.876+0000 7f0361bbb700 1 -- 172.21.15.174:0/2587139701 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f035c0623c0 con 0x7f035c0ff120 2024-02-19T18:55:18.895 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.876+0000 7f0358ff9700 1 -- 172.21.15.174:0/2587139701 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f034c003c50 con 0x7f035c0ff120 2024-02-19T18:55:18.895 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.876+0000 7f0358ff9700 1 --2- 172.21.15.174:0/2587139701 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f03440384b0 0x7f034403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:18.895 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.876+0000 7f0358ff9700 1 -- 172.21.15.174:0/2587139701 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f034c04c4b0 con 0x7f035c0ff120 2024-02-19T18:55:18.896 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.876+0000 7f035affd700 1 --2- 172.21.15.174:0/2587139701 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f03440384b0 0x7f034403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:18.896 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.877+0000 7f035affd700 1 --2- 172.21.15.174:0/2587139701 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f03440384b0 0x7f034403a970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0350006fd0 tx=0x7f0350006e40 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:18.899 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:18.881+0000 7f0358ff9700 1 -- 172.21.15.174:0/2587139701 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f034c020b60 con 0x7f035c0ff120 2024-02-19T18:55:19.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:18 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:19.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:18 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:55:19.176 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:19.158+0000 7f0361bbb700 1 -- 172.21.15.174:0/2587139701 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f035c108850 con 0x7f035c0ff120 2024-02-19T18:55:19.176 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:19.158+0000 7f0358ff9700 1 -- 172.21.15.174:0/2587139701 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f034c020b60 con 0x7f035c0ff120 2024-02-19T18:55:19.177 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:55:19.177 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:55:19.178 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:19.161+0000 7f0361bbb700 1 -- 172.21.15.174:0/2587139701 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f03440384b0 msgr2=0x7f034403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:19.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:19.161+0000 7f0361bbb700 1 --2- 172.21.15.174:0/2587139701 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f03440384b0 0x7f034403a970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0350006fd0 tx=0x7f0350006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:19.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:19.161+0000 7f0361bbb700 1 -- 172.21.15.174:0/2587139701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f035c0ff120 msgr2=0x7f035c195dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:19.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:19.161+0000 7f0361bbb700 1 --2- 172.21.15.174:0/2587139701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f035c0ff120 0x7f035c195dd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f034c005b40 tx=0x7f034c004c80 comp rx=0 tx=0).stop 2024-02-19T18:55:19.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:19.161+0000 7f0361bbb700 1 -- 172.21.15.174:0/2587139701 shutdown_connections 2024-02-19T18:55:19.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:19.161+0000 7f0361bbb700 1 --2- 172.21.15.174:0/2587139701 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f03440384b0 0x7f034403a970 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:19.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:19.161+0000 7f0361bbb700 1 --2- 172.21.15.174:0/2587139701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f035c0ff120 0x7f035c195dd0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:19.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:19.161+0000 7f0361bbb700 1 -- 172.21.15.174:0/2587139701 >> 172.21.15.174:0/2587139701 conn(0x7f035c075df0 msgr2=0x7f035c104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:19.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:19.161+0000 7f0361bbb700 1 -- 172.21.15.174:0/2587139701 shutdown_connections 2024-02-19T18:55:19.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:19.161+0000 7f0361bbb700 1 -- 172.21.15.174:0/2587139701 wait complete. 2024-02-19T18:55:19.179 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:55:20.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:19 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/2587139701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:20.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:19 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:20.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:19 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:20.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:19 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:20.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:19 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:21.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:20 smithi114 ceph-mon[29441]: Deploying daemon node-exporter.smithi174 on smithi174 2024-02-19T18:55:21.039 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:55:21.039 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:55:21.239 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:55:23.222 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.161+0000 7f1c3e1ff700 1 -- 172.21.15.174:0/3781805934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c38105780 msgr2=0x7f1c38105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:23.222 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.161+0000 7f1c3e1ff700 1 --2- 172.21.15.174:0/3781805934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c38105780 0x7f1c38105b60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c009910 tx=0x7f1c2c009c20 comp rx=0 tx=0).stop 2024-02-19T18:55:23.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.161+0000 7f1c3e1ff700 1 -- 172.21.15.174:0/3781805934 shutdown_connections 2024-02-19T18:55:23.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.161+0000 7f1c3e1ff700 1 --2- 172.21.15.174:0/3781805934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c38105780 0x7f1c38105b60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:23.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.161+0000 7f1c3e1ff700 1 -- 172.21.15.174:0/3781805934 >> 172.21.15.174:0/3781805934 conn(0x7f1c380752b0 msgr2=0x7f1c380756c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:23.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.161+0000 7f1c3e1ff700 1 -- 172.21.15.174:0/3781805934 shutdown_connections 2024-02-19T18:55:23.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.161+0000 7f1c3e1ff700 1 -- 172.21.15.174:0/3781805934 wait complete. 2024-02-19T18:55:23.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.162+0000 7f1c3e1ff700 1 Processor -- start 2024-02-19T18:55:23.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.162+0000 7f1c3e1ff700 1 -- start start 2024-02-19T18:55:23.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.162+0000 7f1c3e1ff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c38105780 0x7f1c38195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:23.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.162+0000 7f1c3e1ff700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c381962e0 con 0x7f1c38105780 2024-02-19T18:55:23.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.162+0000 7f1c377fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c38105780 0x7f1c38195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:23.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.162+0000 7f1c377fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c38105780 0x7f1c38195da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:41806/0 (socket says 172.21.15.174:41806) 2024-02-19T18:55:23.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.162+0000 7f1c377fe700 1 -- 172.21.15.174:0/1515106916 learned_addr learned my addr 172.21.15.174:0/1515106916 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:55:23.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.163+0000 7f1c377fe700 1 -- 172.21.15.174:0/1515106916 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c0095f0 con 0x7f1c38105780 2024-02-19T18:55:23.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.163+0000 7f1c377fe700 1 --2- 172.21.15.174:0/1515106916 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c38105780 0x7f1c38195da0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c005b40 tx=0x7f1c2c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:23.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.163+0000 7f1c34ff9700 1 -- 172.21.15.174:0/1515106916 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c2c01d070 con 0x7f1c38105780 2024-02-19T18:55:23.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.163+0000 7f1c3e1ff700 1 -- 172.21.15.174:0/1515106916 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c381964e0 con 0x7f1c38105780 2024-02-19T18:55:23.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.163+0000 7f1c34ff9700 1 -- 172.21.15.174:0/1515106916 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1c2c0044d0 con 0x7f1c38105780 2024-02-19T18:55:23.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.163+0000 7f1c3e1ff700 1 -- 172.21.15.174:0/1515106916 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c381968a0 con 0x7f1c38105780 2024-02-19T18:55:23.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.163+0000 7f1c34ff9700 1 -- 172.21.15.174:0/1515106916 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c2c0039a0 con 0x7f1c38105780 2024-02-19T18:55:23.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.164+0000 7f1c34ff9700 1 -- 172.21.15.174:0/1515106916 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f1c2c003b00 con 0x7f1c38105780 2024-02-19T18:55:23.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.164+0000 7f1c3e1ff700 1 -- 172.21.15.174:0/1515106916 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c380623c0 con 0x7f1c38105780 2024-02-19T18:55:23.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.164+0000 7f1c34ff9700 1 --2- 172.21.15.174:0/1515106916 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f1c200384b0 0x7f1c2003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:23.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.164+0000 7f1c34ff9700 1 -- 172.21.15.174:0/1515106916 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f1c2c04c4d0 con 0x7f1c38105780 2024-02-19T18:55:23.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.164+0000 7f1c36ffd700 1 --2- 172.21.15.174:0/1515106916 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f1c200384b0 0x7f1c2003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:23.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.165+0000 7f1c36ffd700 1 --2- 172.21.15.174:0/1515106916 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f1c200384b0 0x7f1c2003a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1c28006fd0 tx=0x7f1c28006e40 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:23.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.170+0000 7f1c34ff9700 1 -- 172.21.15.174:0/1515106916 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1c2c02a540 con 0x7f1c38105780 2024-02-19T18:55:23.516 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.497+0000 7f1c3e1ff700 1 -- 172.21.15.174:0/1515106916 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c3819a890 con 0x7f1c38105780 2024-02-19T18:55:23.516 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.498+0000 7f1c34ff9700 1 -- 172.21.15.174:0/1515106916 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f1c2c02a360 con 0x7f1c38105780 2024-02-19T18:55:23.517 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:55:23.517 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:55:23.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.500+0000 7f1c3e1ff700 1 -- 172.21.15.174:0/1515106916 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f1c200384b0 msgr2=0x7f1c2003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:23.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.500+0000 7f1c3e1ff700 1 --2- 172.21.15.174:0/1515106916 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f1c200384b0 0x7f1c2003a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1c28006fd0 tx=0x7f1c28006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:23.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.500+0000 7f1c3e1ff700 1 -- 172.21.15.174:0/1515106916 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c38105780 msgr2=0x7f1c38195da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:23.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.500+0000 7f1c3e1ff700 1 --2- 172.21.15.174:0/1515106916 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c38105780 0x7f1c38195da0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c005b40 tx=0x7f1c2c004c80 comp rx=0 tx=0).stop 2024-02-19T18:55:23.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.500+0000 7f1c3e1ff700 1 -- 172.21.15.174:0/1515106916 shutdown_connections 2024-02-19T18:55:23.518 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.500+0000 7f1c3e1ff700 1 --2- 172.21.15.174:0/1515106916 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f1c200384b0 0x7f1c2003a970 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:23.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.500+0000 7f1c3e1ff700 1 --2- 172.21.15.174:0/1515106916 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c38105780 0x7f1c38195da0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:23.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.500+0000 7f1c3e1ff700 1 -- 172.21.15.174:0/1515106916 >> 172.21.15.174:0/1515106916 conn(0x7f1c380752b0 msgr2=0x7f1c38067fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:23.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.500+0000 7f1c3e1ff700 1 -- 172.21.15.174:0/1515106916 shutdown_connections 2024-02-19T18:55:23.519 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:23.500+0000 7f1c3e1ff700 1 -- 172.21.15.174:0/1515106916 wait complete. 2024-02-19T18:55:23.519 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:55:24.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:23 smithi114 ceph-mon[29441]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:24.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:24.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:23 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/1515106916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:25.293 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:55:25.294 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:55:25.507 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:55:25.935 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:25 smithi114 ceph-mon[29441]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:27.846 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.827+0000 7ff532909700 1 -- 172.21.15.174:0/1867197809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff52c105730 msgr2=0x7ff52c105b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:27.846 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.827+0000 7ff532909700 1 --2- 172.21.15.174:0/1867197809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff52c105730 0x7ff52c105b10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009910 tx=0x7ff51c009c20 comp rx=0 tx=0).stop 2024-02-19T18:55:27.846 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.827+0000 7ff532909700 1 -- 172.21.15.174:0/1867197809 shutdown_connections 2024-02-19T18:55:27.846 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.827+0000 7ff532909700 1 --2- 172.21.15.174:0/1867197809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff52c105730 0x7ff52c105b10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:27.846 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.827+0000 7ff532909700 1 -- 172.21.15.174:0/1867197809 >> 172.21.15.174:0/1867197809 conn(0x7ff52c0fb240 msgr2=0x7ff52c0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:27.846 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.827+0000 7ff532909700 1 -- 172.21.15.174:0/1867197809 shutdown_connections 2024-02-19T18:55:27.846 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.827+0000 7ff532909700 1 -- 172.21.15.174:0/1867197809 wait complete. 2024-02-19T18:55:27.846 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.828+0000 7ff532909700 1 Processor -- start 2024-02-19T18:55:27.846 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.828+0000 7ff532909700 1 -- start start 2024-02-19T18:55:27.847 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.828+0000 7ff532909700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff52c105730 0x7ff52c195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:27.847 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.828+0000 7ff532909700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c196250 con 0x7ff52c105730 2024-02-19T18:55:27.847 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.828+0000 7ff52bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff52c105730 0x7ff52c195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:27.847 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.828+0000 7ff52bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff52c105730 0x7ff52c195d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:39854/0 (socket says 172.21.15.174:39854) 2024-02-19T18:55:27.847 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.828+0000 7ff52bfff700 1 -- 172.21.15.174:0/1202793726 learned_addr learned my addr 172.21.15.174:0/1202793726 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:55:27.847 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.829+0000 7ff52bfff700 1 -- 172.21.15.174:0/1202793726 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c0095f0 con 0x7ff52c105730 2024-02-19T18:55:27.847 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.829+0000 7ff52bfff700 1 --2- 172.21.15.174:0/1202793726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff52c105730 0x7ff52c195d10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0092e0 tx=0x7ff51c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:27.847 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.829+0000 7ff5297fa700 1 -- 172.21.15.174:0/1202793726 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff51c01d070 con 0x7ff52c105730 2024-02-19T18:55:27.847 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.829+0000 7ff532909700 1 -- 172.21.15.174:0/1202793726 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff52c1964b0 con 0x7ff52c105730 2024-02-19T18:55:27.847 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.829+0000 7ff5297fa700 1 -- 172.21.15.174:0/1202793726 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff51c00be10 con 0x7ff52c105730 2024-02-19T18:55:27.847 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.829+0000 7ff5297fa700 1 -- 172.21.15.174:0/1202793726 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff51c022e10 con 0x7ff52c105730 2024-02-19T18:55:27.847 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.829+0000 7ff532909700 1 -- 172.21.15.174:0/1202793726 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff52c196930 con 0x7ff52c105730 2024-02-19T18:55:27.848 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.830+0000 7ff532909700 1 -- 172.21.15.174:0/1202793726 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff52c0623c0 con 0x7ff52c105730 2024-02-19T18:55:27.848 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.830+0000 7ff5297fa700 1 -- 172.21.15.174:0/1202793726 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7ff51c00b960 con 0x7ff52c105730 2024-02-19T18:55:27.848 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.830+0000 7ff5297fa700 1 --2- 172.21.15.174:0/1202793726 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff514038460 0x7ff51403a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:27.848 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.830+0000 7ff5297fa700 1 -- 172.21.15.174:0/1202793726 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7ff51c04c440 con 0x7ff52c105730 2024-02-19T18:55:27.848 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.830+0000 7ff52b7fe700 1 --2- 172.21.15.174:0/1202793726 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff514038460 0x7ff51403a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:27.849 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.831+0000 7ff52b7fe700 1 --2- 172.21.15.174:0/1202793726 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff514038460 0x7ff51403a920 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff520006fd0 tx=0x7ff520006e40 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:27.852 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:27.835+0000 7ff5297fa700 1 -- 172.21.15.174:0/1202793726 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff51c00f460 con 0x7ff52c105730 2024-02-19T18:55:28.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:27 smithi114 ceph-mon[29441]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:28.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:28.104+0000 7ff532909700 1 -- 172.21.15.174:0/1202793726 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff52c19a8b0 con 0x7ff52c105730 2024-02-19T18:55:28.121 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:28.104+0000 7ff5297fa700 1 -- 172.21.15.174:0/1202793726 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7ff51c027030 con 0x7ff52c105730 2024-02-19T18:55:28.121 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:55:28.122 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:55:28.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:28.107+0000 7ff532909700 1 -- 172.21.15.174:0/1202793726 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff514038460 msgr2=0x7ff51403a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:28.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:28.107+0000 7ff532909700 1 --2- 172.21.15.174:0/1202793726 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff514038460 0x7ff51403a920 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff520006fd0 tx=0x7ff520006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:28.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:28.107+0000 7ff532909700 1 -- 172.21.15.174:0/1202793726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff52c105730 msgr2=0x7ff52c195d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:28.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:28.107+0000 7ff532909700 1 --2- 172.21.15.174:0/1202793726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff52c105730 0x7ff52c195d10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0092e0 tx=0x7ff51c004c80 comp rx=0 tx=0).stop 2024-02-19T18:55:28.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:28.107+0000 7ff532909700 1 -- 172.21.15.174:0/1202793726 shutdown_connections 2024-02-19T18:55:28.124 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:28.107+0000 7ff532909700 1 --2- 172.21.15.174:0/1202793726 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff514038460 0x7ff51403a920 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:28.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:28.107+0000 7ff532909700 1 --2- 172.21.15.174:0/1202793726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff52c105730 0x7ff52c195d10 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:28.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:28.107+0000 7ff532909700 1 -- 172.21.15.174:0/1202793726 >> 172.21.15.174:0/1202793726 conn(0x7ff52c0fb240 msgr2=0x7ff52c069120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:28.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:28.107+0000 7ff532909700 1 -- 172.21.15.174:0/1202793726 shutdown_connections 2024-02-19T18:55:28.125 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:28.107+0000 7ff532909700 1 -- 172.21.15.174:0/1202793726 wait complete. 2024-02-19T18:55:28.125 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:55:29.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:29 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:29.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:29 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:29.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:29 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/1202793726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:29.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:29 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:29.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:29 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:29.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:29 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi174.lydvnj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T18:55:29.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:29 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi174.lydvnj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-19T18:55:29.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:29 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T18:55:29.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:29 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:29.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:29 smithi114 ceph-mon[29441]: Deploying daemon mgr.smithi174.lydvnj on smithi174 2024-02-19T18:55:29.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:29 smithi114 ceph-mon[29441]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:29.687 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:55:29.687 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:55:29.905 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:55:31.466 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.447+0000 7fd8c1197700 1 -- 172.21.15.174:0/1592611772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8bc106810 msgr2=0x7fd8bc106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:31.467 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.447+0000 7fd8c1197700 1 --2- 172.21.15.174:0/1592611772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8bc106810 0x7fd8bc106bf0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009910 tx=0x7fd8b0009c20 comp rx=0 tx=0).stop 2024-02-19T18:55:31.467 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.447+0000 7fd8c1197700 1 -- 172.21.15.174:0/1592611772 shutdown_connections 2024-02-19T18:55:31.467 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.447+0000 7fd8c1197700 1 --2- 172.21.15.174:0/1592611772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8bc106810 0x7fd8bc106bf0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:31.467 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.447+0000 7fd8c1197700 1 -- 172.21.15.174:0/1592611772 >> 172.21.15.174:0/1592611772 conn(0x7fd8bc0fc320 msgr2=0x7fd8bc0fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:31.467 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.447+0000 7fd8c1197700 1 -- 172.21.15.174:0/1592611772 shutdown_connections 2024-02-19T18:55:31.467 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.447+0000 7fd8c1197700 1 -- 172.21.15.174:0/1592611772 wait complete. 2024-02-19T18:55:31.467 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.448+0000 7fd8c1197700 1 Processor -- start 2024-02-19T18:55:31.467 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.448+0000 7fd8c1197700 1 -- start start 2024-02-19T18:55:31.467 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.448+0000 7fd8c1197700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8bc106810 0x7fd8bc195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:31.467 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.448+0000 7fd8c1197700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc1962a0 con 0x7fd8bc106810 2024-02-19T18:55:31.467 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.449+0000 7fd8bad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8bc106810 0x7fd8bc195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:31.468 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.449+0000 7fd8bad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8bc106810 0x7fd8bc195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:39878/0 (socket says 172.21.15.174:39878) 2024-02-19T18:55:31.468 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.449+0000 7fd8bad9d700 1 -- 172.21.15.174:0/100998210 learned_addr learned my addr 172.21.15.174:0/100998210 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:55:31.468 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.449+0000 7fd8bad9d700 1 -- 172.21.15.174:0/100998210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b00095f0 con 0x7fd8bc106810 2024-02-19T18:55:31.468 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.449+0000 7fd8bad9d700 1 --2- 172.21.15.174:0/100998210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8bc106810 0x7fd8bc195d60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0005b40 tx=0x7fd8b0004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:31.468 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.449+0000 7fd8a3fff700 1 -- 172.21.15.174:0/100998210 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8b001d070 con 0x7fd8bc106810 2024-02-19T18:55:31.468 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.449+0000 7fd8c1197700 1 -- 172.21.15.174:0/100998210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8bc1964a0 con 0x7fd8bc106810 2024-02-19T18:55:31.468 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.449+0000 7fd8a3fff700 1 -- 172.21.15.174:0/100998210 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd8b00044d0 con 0x7fd8bc106810 2024-02-19T18:55:31.468 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.449+0000 7fd8a3fff700 1 -- 172.21.15.174:0/100998210 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8b0003910 con 0x7fd8bc106810 2024-02-19T18:55:31.468 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.449+0000 7fd8c1197700 1 -- 172.21.15.174:0/100998210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8bc196920 con 0x7fd8bc106810 2024-02-19T18:55:31.469 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.450+0000 7fd8c1197700 1 -- 172.21.15.174:0/100998210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8bc0623c0 con 0x7fd8bc106810 2024-02-19T18:55:31.469 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.450+0000 7fd8a3fff700 1 -- 172.21.15.174:0/100998210 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7fd8b0003bb0 con 0x7fd8bc106810 2024-02-19T18:55:31.469 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.451+0000 7fd8a3fff700 1 --2- 172.21.15.174:0/100998210 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd8a40384b0 0x7fd8a403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:31.469 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.451+0000 7fd8a3fff700 1 -- 172.21.15.174:0/100998210 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fd8b004c6f0 con 0x7fd8bc106810 2024-02-19T18:55:31.469 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.451+0000 7fd8ba59c700 1 --2- 172.21.15.174:0/100998210 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd8a40384b0 0x7fd8a403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:31.469 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.451+0000 7fd8ba59c700 1 --2- 172.21.15.174:0/100998210 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd8a40384b0 0x7fd8a403a970 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac006fd0 tx=0x7fd8ac006e40 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:31.473 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.456+0000 7fd8a3fff700 1 -- 172.21.15.174:0/100998210 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd8b002a950 con 0x7fd8bc106810 2024-02-19T18:55:31.749 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.732+0000 7fd8c1197700 1 -- 172.21.15.174:0/100998210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd8bc1970c0 con 0x7fd8bc106810 2024-02-19T18:55:31.749 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.732+0000 7fd8a3fff700 1 -- 172.21.15.174:0/100998210 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fd8b002a360 con 0x7fd8bc106810 2024-02-19T18:55:31.749 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:55:31.750 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:55:31.752 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.735+0000 7fd8c1197700 1 -- 172.21.15.174:0/100998210 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd8a40384b0 msgr2=0x7fd8a403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:31.752 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.735+0000 7fd8c1197700 1 --2- 172.21.15.174:0/100998210 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd8a40384b0 0x7fd8a403a970 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac006fd0 tx=0x7fd8ac006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:31.752 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.735+0000 7fd8c1197700 1 -- 172.21.15.174:0/100998210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8bc106810 msgr2=0x7fd8bc195d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:31.752 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.735+0000 7fd8c1197700 1 --2- 172.21.15.174:0/100998210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8bc106810 0x7fd8bc195d60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0005b40 tx=0x7fd8b0004c80 comp rx=0 tx=0).stop 2024-02-19T18:55:31.752 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.735+0000 7fd8c1197700 1 -- 172.21.15.174:0/100998210 shutdown_connections 2024-02-19T18:55:31.752 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.735+0000 7fd8c1197700 1 --2- 172.21.15.174:0/100998210 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd8a40384b0 0x7fd8a403a970 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:31.752 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.735+0000 7fd8c1197700 1 --2- 172.21.15.174:0/100998210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd8bc106810 0x7fd8bc195d60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:31.752 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.735+0000 7fd8c1197700 1 -- 172.21.15.174:0/100998210 >> 172.21.15.174:0/100998210 conn(0x7fd8bc0fc320 msgr2=0x7fd8bc108bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:31.752 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.735+0000 7fd8c1197700 1 -- 172.21.15.174:0/100998210 shutdown_connections 2024-02-19T18:55:31.752 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:31.735+0000 7fd8c1197700 1 -- 172.21.15.174:0/100998210 wait complete. 2024-02-19T18:55:31.753 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:55:32.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:31 smithi114 ceph-mon[29441]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:33.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:32 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/100998210' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:33.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:32 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:33.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:32 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:33.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:32 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:33.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:32 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:33.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:32 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T18:55:33.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:32 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:33.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:32 smithi114 ceph-mon[29441]: Deploying daemon mon.smithi174 on smithi174 2024-02-19T18:55:33.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:32 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:55:33.497 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:55:33.497 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:55:34.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:33 smithi114 ceph-mon[29441]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:34.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:34.621 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi174/config 2024-02-19T18:55:36.005 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:35 smithi114 ceph-mon[29441]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:36.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.396+0000 7f861da56700 1 -- 172.21.15.174:0/2927478160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86181003b0 msgr2=0x7f8618100790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:36.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.396+0000 7f861da56700 1 --2- 172.21.15.174:0/2927478160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86181003b0 0x7f8618100790 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f860c009910 tx=0x7f860c009c20 comp rx=0 tx=0).stop 2024-02-19T18:55:36.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.396+0000 7f861da56700 1 -- 172.21.15.174:0/2927478160 shutdown_connections 2024-02-19T18:55:36.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.396+0000 7f861da56700 1 --2- 172.21.15.174:0/2927478160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86181003b0 0x7f8618100790 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:36.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.396+0000 7f861da56700 1 -- 172.21.15.174:0/2927478160 >> 172.21.15.174:0/2927478160 conn(0x7f8618074ae0 msgr2=0x7f8618074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:36.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.396+0000 7f861da56700 1 -- 172.21.15.174:0/2927478160 shutdown_connections 2024-02-19T18:55:36.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.396+0000 7f861da56700 1 -- 172.21.15.174:0/2927478160 wait complete. 2024-02-19T18:55:36.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.397+0000 7f861da56700 1 Processor -- start 2024-02-19T18:55:36.414 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.397+0000 7f861da56700 1 -- start start 2024-02-19T18:55:36.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.397+0000 7f861da56700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86181003b0 0x7f86180fc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:36.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.397+0000 7f861da56700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86180fe070 con 0x7f86181003b0 2024-02-19T18:55:36.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.398+0000 7f8616ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86181003b0 0x7f86180fc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:36.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.398+0000 7f8616ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86181003b0 0x7f86180fc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:48302/0 (socket says 172.21.15.174:48302) 2024-02-19T18:55:36.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.398+0000 7f8616ffd700 1 -- 172.21.15.174:0/2231035950 learned_addr learned my addr 172.21.15.174:0/2231035950 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:55:36.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.398+0000 7f8616ffd700 1 -- 172.21.15.174:0/2231035950 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c0095f0 con 0x7f86181003b0 2024-02-19T18:55:36.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.398+0000 7f8616ffd700 1 --2- 172.21.15.174:0/2231035950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86181003b0 0x7f86180fc0f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f860c005b40 tx=0x7f860c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:36.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.398+0000 7f861ca54700 1 -- 172.21.15.174:0/2231035950 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f860c01d070 con 0x7f86181003b0 2024-02-19T18:55:36.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.398+0000 7f861da56700 1 -- 172.21.15.174:0/2231035950 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86180fc630 con 0x7f86181003b0 2024-02-19T18:55:36.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.398+0000 7f861ca54700 1 -- 172.21.15.174:0/2231035950 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f860c0044d0 con 0x7f86181003b0 2024-02-19T18:55:36.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.398+0000 7f861ca54700 1 -- 172.21.15.174:0/2231035950 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f860c003930 con 0x7f86181003b0 2024-02-19T18:55:36.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.398+0000 7f861da56700 1 -- 172.21.15.174:0/2231035950 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86180fcab0 con 0x7f86181003b0 2024-02-19T18:55:36.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.399+0000 7f861da56700 1 -- 172.21.15.174:0/2231035950 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86180623c0 con 0x7f86181003b0 2024-02-19T18:55:36.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.399+0000 7f861ca54700 1 -- 172.21.15.174:0/2231035950 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f860c003ba0 con 0x7f86181003b0 2024-02-19T18:55:36.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.400+0000 7f861ca54700 1 --2- 172.21.15.174:0/2231035950 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8600038500 0x7f860003a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:36.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.400+0000 7f861ca54700 1 -- 172.21.15.174:0/2231035950 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f860c04c3b0 con 0x7f86181003b0 2024-02-19T18:55:36.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.400+0000 7f86167fc700 1 --2- 172.21.15.174:0/2231035950 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8600038500 0x7f860003a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:36.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.400+0000 7f86167fc700 1 --2- 172.21.15.174:0/2231035950 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8600038500 0x7f860003a9c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8608006fd0 tx=0x7f8608006e40 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:36.421 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.405+0000 7f861ca54700 1 -- 172.21.15.174:0/2231035950 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f860c022d10 con 0x7f86181003b0 2024-02-19T18:55:36.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:36 smithi174 podman[41020]: 2024-02-19 18:55:36.611227726 +0000 UTC m=+0.124547830 container create 70dd7bf69fd96add53ce87c186fe201334b327ce7670fe8543f403bfc2a040bf (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi174, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240214, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1) 2024-02-19T18:55:36.709 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.691+0000 7f861da56700 1 -- 172.21.15.174:0/2231035950 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86180fcea0 con 0x7f86181003b0 2024-02-19T18:55:36.709 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.692+0000 7f861ca54700 1 -- 172.21.15.174:0/2231035950 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f860c020b20 con 0x7f86181003b0 2024-02-19T18:55:36.709 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:55:36.709 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:51:30.435772Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:55:36.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.694+0000 7f861da56700 1 -- 172.21.15.174:0/2231035950 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8600038500 msgr2=0x7f860003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:36.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.694+0000 7f861da56700 1 --2- 172.21.15.174:0/2231035950 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8600038500 0x7f860003a9c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8608006fd0 tx=0x7f8608006e40 comp rx=0 tx=0).stop 2024-02-19T18:55:36.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.694+0000 7f861da56700 1 -- 172.21.15.174:0/2231035950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86181003b0 msgr2=0x7f86180fc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:36.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.694+0000 7f861da56700 1 --2- 172.21.15.174:0/2231035950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86181003b0 0x7f86180fc0f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f860c005b40 tx=0x7f860c004c80 comp rx=0 tx=0).stop 2024-02-19T18:55:36.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.694+0000 7f861da56700 1 -- 172.21.15.174:0/2231035950 shutdown_connections 2024-02-19T18:55:36.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.694+0000 7f861da56700 1 --2- 172.21.15.174:0/2231035950 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8600038500 0x7f860003a9c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:36.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.694+0000 7f861da56700 1 --2- 172.21.15.174:0/2231035950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86181003b0 0x7f86180fc0f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:36.711 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.694+0000 7f861da56700 1 -- 172.21.15.174:0/2231035950 >> 172.21.15.174:0/2231035950 conn(0x7f8618074ae0 msgr2=0x7f8618103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:36.712 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.694+0000 7f861da56700 1 -- 172.21.15.174:0/2231035950 shutdown_connections 2024-02-19T18:55:36.712 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:36.694+0000 7f861da56700 1 -- 172.21.15.174:0/2231035950 wait complete. 2024-02-19T18:55:36.712 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-02-19T18:55:37.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 podman[41020]: 2024-02-19 18:55:37.392818345 +0000 UTC m=+0.906138465 container init 70dd7bf69fd96add53ce87c186fe201334b327ce7670fe8543f403bfc2a040bf (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi174, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-19T18:55:37.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 podman[41020]: 2024-02-19 18:55:37.396013615 +0000 UTC m=+0.909333728 container start 70dd7bf69fd96add53ce87c186fe201334b327ce7670fe8543f403bfc2a040bf (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi174, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-19T18:55:37.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: set uid:gid to 167:167 (ceph:ceph) 2024-02-19T18:55:37.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable), process ceph-mon, pid 2 2024-02-19T18:55:37.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pidfile_write: ignore empty --pid-file 2024-02-19T18:55:37.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: load: jerasure load: lrc 2024-02-19T18:55:37.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: RocksDB version: 7.9.2 2024-02-19T18:55:37.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Git sha 0 2024-02-19T18:55:37.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Compile date 2023-12-11 22:07:34 2024-02-19T18:55:37.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: DB SUMMARY 2024-02-19T18:55:37.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: DB Session ID: CQKB73Y8YMCBBAKXF68J 2024-02-19T18:55:37.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: CURRENT file: CURRENT 2024-02-19T18:55:37.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: IDENTITY file: IDENTITY 2024-02-19T18:55:37.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-19T18:55:37.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi174/store.db dir, Total Num: 0, files: 2024-02-19T18:55:37.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi174/store.db: 000004.log size: 511 ; 2024-02-19T18:55:37.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.error_if_exists: 0 2024-02-19T18:55:37.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.create_if_missing: 0 2024-02-19T18:55:37.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.paranoid_checks: 1 2024-02-19T18:55:37.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-19T18:55:37.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-19T18:55:37.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-19T18:55:37.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.env: 0x55ff2bda3720 2024-02-19T18:55:37.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.fs: PosixFileSystem 2024-02-19T18:55:37.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.info_log: 0x55ff2de212e0 2024-02-19T18:55:37.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_file_opening_threads: 16 2024-02-19T18:55:37.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.statistics: (nil) 2024-02-19T18:55:37.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.use_fsync: 0 2024-02-19T18:55:37.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_log_file_size: 0 2024-02-19T18:55:37.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T18:55:37.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T18:55:37.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.keep_log_file_num: 1000 2024-02-19T18:55:37.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.recycle_log_file_num: 0 2024-02-19T18:55:37.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.allow_fallocate: 1 2024-02-19T18:55:37.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.allow_mmap_reads: 0 2024-02-19T18:55:37.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.allow_mmap_writes: 0 2024-02-19T18:55:37.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.use_direct_reads: 0 2024-02-19T18:55:37.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T18:55:37.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.create_missing_column_families: 0 2024-02-19T18:55:37.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.db_log_dir: 2024-02-19T18:55:37.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.wal_dir: 2024-02-19T18:55:37.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T18:55:37.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T18:55:37.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T18:55:37.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T18:55:37.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T18:55:37.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T18:55:37.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.advise_random_on_open: 1 2024-02-19T18:55:37.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.db_write_buffer_size: 0 2024-02-19T18:55:37.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.write_buffer_manager: 0x55ff2d092140 2024-02-19T18:55:37.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T18:55:37.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T18:55:37.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T18:55:37.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.rate_limiter: (nil) 2024-02-19T18:55:37.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T18:55:37.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.wal_recovery_mode: 2 2024-02-19T18:55:37.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.enable_thread_tracking: 0 2024-02-19T18:55:37.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.enable_pipelined_write: 0 2024-02-19T18:55:37.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.unordered_write: 0 2024-02-19T18:55:37.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T18:55:37.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T18:55:37.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T18:55:37.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T18:55:37.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.row_cache: None 2024-02-19T18:55:37.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.wal_filter: None 2024-02-19T18:55:37.647 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T18:55:37.649 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.allow_ingest_behind: 0 2024-02-19T18:55:37.649 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.two_write_queues: 0 2024-02-19T18:55:37.649 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.manual_wal_flush: 0 2024-02-19T18:55:37.649 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.wal_compression: 0 2024-02-19T18:55:37.649 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.atomic_flush: 0 2024-02-19T18:55:37.649 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T18:55:37.649 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T18:55:37.649 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T18:55:37.649 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.log_readahead_size: 0 2024-02-19T18:55:37.649 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-19T18:55:37.649 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.best_efforts_recovery: 0 2024-02-19T18:55:37.649 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-19T18:55:37.650 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-19T18:55:37.650 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.allow_data_in_errors: 0 2024-02-19T18:55:37.650 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.db_host_id: __hostname__ 2024-02-19T18:55:37.650 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-19T18:55:37.650 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_background_jobs: 2 2024-02-19T18:55:37.650 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_background_compactions: -1 2024-02-19T18:55:37.650 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_subcompactions: 1 2024-02-19T18:55:37.650 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T18:55:37.650 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T18:55:37.650 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T18:55:37.650 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_total_wal_size: 0 2024-02-19T18:55:37.650 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T18:55:37.650 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T18:55:37.650 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T18:55:37.651 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T18:55:37.651 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_open_files: -1 2024-02-19T18:55:37.651 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.bytes_per_sync: 0 2024-02-19T18:55:37.651 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T18:55:37.651 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T18:55:37.651 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compaction_readahead_size: 0 2024-02-19T18:55:37.651 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_background_flushes: -1 2024-02-19T18:55:37.651 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Compression algorithms supported: 2024-02-19T18:55:37.651 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T18:55:37.651 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: kZSTD supported: 0 2024-02-19T18:55:37.651 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: kXpressCompression supported: 0 2024-02-19T18:55:37.651 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: kLZ4HCCompression supported: 1 2024-02-19T18:55:37.651 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: kZlibCompression supported: 1 2024-02-19T18:55:37.651 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: kSnappyCompression supported: 1 2024-02-19T18:55:37.652 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: kLZ4Compression supported: 1 2024-02-19T18:55:37.652 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: kBZip2Compression supported: 0 2024-02-19T18:55:37.652 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T18:55:37.652 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-19T18:55:37.652 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi174/store.db/MANIFEST-000005 2024-02-19T18:55:37.652 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-19T18:55:37.652 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T18:55:37.652 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.merge_operator: 2024-02-19T18:55:37.652 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compaction_filter: None 2024-02-19T18:55:37.652 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compaction_filter_factory: None 2024-02-19T18:55:37.652 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.sst_partitioner_factory: None 2024-02-19T18:55:37.652 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T18:55:37.652 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T18:55:37.652 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ff2de213e0) 2024-02-19T18:55:37.653 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: cache_index_and_filter_blocks: 1 2024-02-19T18:55:37.653 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T18:55:37.653 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T18:55:37.653 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: pin_top_level_index_and_filter: 1 2024-02-19T18:55:37.653 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: index_type: 0 2024-02-19T18:55:37.653 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: data_block_index_type: 0 2024-02-19T18:55:37.653 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: index_shortening: 1 2024-02-19T18:55:37.653 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-19T18:55:37.653 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: checksum: 4 2024-02-19T18:55:37.653 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: no_block_cache: 0 2024-02-19T18:55:37.653 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: block_cache: 0x55ff2d1331f0 2024-02-19T18:55:37.653 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: block_cache_name: BinnedLRUCache 2024-02-19T18:55:37.653 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: block_cache_options: 2024-02-19T18:55:37.653 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: capacity : 536870912 2024-02-19T18:55:37.653 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: num_shard_bits : 4 2024-02-19T18:55:37.653 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: strict_capacity_limit : 0 2024-02-19T18:55:37.654 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: high_pri_pool_ratio: 0.000 2024-02-19T18:55:37.654 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: block_cache_compressed: (nil) 2024-02-19T18:55:37.654 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: persistent_cache: (nil) 2024-02-19T18:55:37.654 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: block_size: 4096 2024-02-19T18:55:37.654 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: block_size_deviation: 10 2024-02-19T18:55:37.654 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: block_restart_interval: 16 2024-02-19T18:55:37.654 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: index_block_restart_interval: 1 2024-02-19T18:55:37.654 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: metadata_block_size: 4096 2024-02-19T18:55:37.654 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: partition_filters: 0 2024-02-19T18:55:37.654 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: use_delta_encoding: 1 2024-02-19T18:55:37.654 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: filter_policy: bloomfilter 2024-02-19T18:55:37.654 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: whole_key_filtering: 1 2024-02-19T18:55:37.654 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: verify_compression: 0 2024-02-19T18:55:37.654 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: read_amp_bytes_per_bit: 0 2024-02-19T18:55:37.655 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: format_version: 5 2024-02-19T18:55:37.655 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: enable_index_compression: 1 2024-02-19T18:55:37.655 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: block_align: 0 2024-02-19T18:55:37.655 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: max_auto_readahead_size: 262144 2024-02-19T18:55:37.655 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: prepopulate_block_cache: 0 2024-02-19T18:55:37.655 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: initial_auto_readahead_size: 8192 2024-02-19T18:55:37.655 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: num_file_reads_for_auto_readahead: 2 2024-02-19T18:55:37.655 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.write_buffer_size: 33554432 2024-02-19T18:55:37.655 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_write_buffer_number: 2 2024-02-19T18:55:37.655 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compression: NoCompression 2024-02-19T18:55:37.655 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.bottommost_compression: Disabled 2024-02-19T18:55:37.655 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.prefix_extractor: nullptr 2024-02-19T18:55:37.655 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T18:55:37.655 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.num_levels: 7 2024-02-19T18:55:37.655 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T18:55:37.656 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T18:55:37.656 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T18:55:37.656 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T18:55:37.656 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T18:55:37.656 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T18:55:37.656 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T18:55:37.656 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T18:55:37.656 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-19T18:55:37.656 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T18:55:37.656 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-19T18:55:37.656 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-19T18:55:37.656 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T18:55:37.656 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compression_opts.level: 32767 2024-02-19T18:55:37.657 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compression_opts.strategy: 0 2024-02-19T18:55:37.657 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T18:55:37.657 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T18:55:37.657 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-19T18:55:37.657 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-19T18:55:37.657 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compression_opts.enabled: false 2024-02-19T18:55:37.657 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-19T18:55:37.657 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T18:55:37.657 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T18:55:37.657 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T18:55:37.657 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.target_file_size_base: 67108864 2024-02-19T18:55:37.657 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T18:55:37.657 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T18:55:37.657 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T18:55:37.657 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T18:55:37.658 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T18:55:37.658 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T18:55:37.658 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T18:55:37.658 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T18:55:37.658 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T18:55:37.658 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T18:55:37.658 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T18:55:37.658 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T18:55:37.658 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T18:55:37.658 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-19T18:55:37.658 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.arena_block_size: 1048576 2024-02-19T18:55:37.658 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T18:55:37.658 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T18:55:37.658 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.disable_auto_compactions: 0 2024-02-19T18:55:37.658 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T18:55:37.658 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T18:55:37.659 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T18:55:37.659 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T18:55:37.659 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T18:55:37.659 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T18:55:37.659 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T18:55:37.659 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T18:55:37.659 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T18:55:37.659 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T18:55:37.659 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.table_properties_collectors: 2024-02-19T18:55:37.659 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.inplace_update_support: 0 2024-02-19T18:55:37.659 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T18:55:37.659 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T18:55:37.659 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T18:55:37.659 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T18:55:37.659 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.bloom_locality: 0 2024-02-19T18:55:37.660 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.max_successive_merges: 0 2024-02-19T18:55:37.660 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T18:55:37.660 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.paranoid_file_checks: 0 2024-02-19T18:55:37.660 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.force_consistency_checks: 1 2024-02-19T18:55:37.660 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.report_bg_io_stats: 0 2024-02-19T18:55:37.660 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.ttl: 2592000 2024-02-19T18:55:37.660 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T18:55:37.660 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-19T18:55:37.660 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-19T18:55:37.660 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.enable_blob_files: false 2024-02-19T18:55:37.660 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.min_blob_size: 0 2024-02-19T18:55:37.660 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.blob_file_size: 268435456 2024-02-19T18:55:37.660 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-19T18:55:37.661 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-19T18:55:37.661 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-19T18:55:37.661 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-19T18:55:37.661 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-19T18:55:37.661 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.blob_file_starting_level: 0 2024-02-19T18:55:37.661 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-19T18:55:37.661 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi174/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T18:55:37.661 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-19T18:55:37.661 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fcaa4927-5ace-4f3c-a090-68d2f97f0518 2024-02-19T18:55:37.661 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708368937425137, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-19T18:55:37.661 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-19T18:55:37.661 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708368937425565, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1708368937, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fcaa4927-5ace-4f3c-a090-68d2f97f0518", "db_session_id": "CQKB73Y8YMCBBAKXF68J", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-19T18:55:37.661 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708368937425644, "job": 1, "event": "recovery_finished"} 2024-02-19T18:55:37.661 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-19T18:55:37.661 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi174/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-19T18:55:37.662 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ff2d1d0000 2024-02-19T18:55:37.662 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: DB pointer 0x55ff2d1bc000 2024-02-19T18:55:37.662 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-19T18:55:37.662 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-19T18:55:37.662 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: ** DB Stats ** 2024-02-19T18:55:37.662 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T18:55:37.662 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T18:55:37.662 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T18:55:37.662 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T18:55:37.662 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T18:55:37.662 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T18:55:37.662 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T18:55:37.662 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: 2024-02-19T18:55:37.662 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: ** Compaction Stats [default] ** 2024-02-19T18:55:37.663 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-19T18:55:37.663 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-19T18:55:37.663 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-19T18:55:37.663 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-19T18:55:37.663 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-19T18:55:37.663 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: 2024-02-19T18:55:37.663 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: ** Compaction Stats [default] ** 2024-02-19T18:55:37.663 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-19T18:55:37.663 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T18:55:37.663 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-19T18:55:37.663 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: 2024-02-19T18:55:37.663 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-19T18:55:37.663 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: 2024-02-19T18:55:37.663 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T18:55:37.664 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T18:55:37.664 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T18:55:37.664 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T18:55:37.664 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T18:55:37.664 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-19T18:55:37.664 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T18:55:37.664 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T18:55:37.664 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T18:55:37.664 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Block cache BinnedLRUCache@0x55ff2d1331f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-02-19T18:55:37.664 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-19T18:55:37.664 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: 2024-02-19T18:55:37.664 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-19T18:55:37.664 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mon.smithi174 does not exist in monmap, will attempt to join an existing cluster 2024-02-19T18:55:37.664 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: using public_addr v2:172.21.15.174:0/0 -> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-02-19T18:55:37.665 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: starting mon.smithi174 rank -1 at public addrs [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] at bind addrs [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi174 fsid be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T18:55:37.665 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mon.smithi174@-1(???) e0 preinit fsid be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T18:55:37.665 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mon.smithi174@-1(synchronizing).mds e1 new map 2024-02-19T18:55:37.665 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mon.smithi174@-1(synchronizing).mds e1 print_map 2024-02-19T18:55:37.665 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: e1 2024-02-19T18:55:37.665 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T18:55:37.665 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T18:55:37.665 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: legacy client fscid: -1 2024-02-19T18:55:37.665 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: 2024-02-19T18:55:37.665 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: No filesystems configured 2024-02-19T18:55:37.665 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mon.smithi174@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-19T18:55:37.665 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mon.smithi174@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-19T18:55:37.665 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mon.smithi174@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-19T18:55:37.666 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mon.smithi174@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-19T18:55:37.666 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mon.smithi174@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-19T18:55:37.666 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mon.smithi174@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-19T18:55:37.666 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mon.smithi174@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-02-19T18:55:37.666 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mon.smithi174@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-19T18:55:37.666 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mon.smithi174@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T18:55:37.666 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mon.smithi174@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T18:55:37.666 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mon.smithi174@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T18:55:37.666 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.666 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.666 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/2234667369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.666 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.666 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.666 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/3034656576' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.666 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.667 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.667 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.667 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.667 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.667 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.667 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.667 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.667 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: Deploying daemon prometheus.smithi114 on smithi114 2024-02-19T18:55:37.667 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.667 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.667 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/432600507' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.667 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.667 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.667 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.667 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/3040643438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.668 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.668 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/418956172' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.668 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.668 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.668 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/3395522096' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.668 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.668 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.668 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/1088317763' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.668 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.668 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.668 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/442342733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.668 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.668 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.668 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/2755419644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.669 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.669 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/4105469896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.669 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.669 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.669 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.669 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.669 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.669 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-19T18:55:37.669 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/3017950911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.669 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14162 172.21.15.114:0/205366855' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-19T18:55:37.669 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mgrmap e14: smithi114.pcvtrk(active, since 2m) 2024-02-19T18:55:37.669 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/3525426355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.669 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: Active manager daemon smithi114.pcvtrk restarted 2024-02-19T18:55:37.669 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: Activating manager daemon smithi114.pcvtrk 2024-02-19T18:55:37.670 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: osdmap e5: 0 total, 0 up, 0 in 2024-02-19T18:55:37.670 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mgrmap e15: smithi114.pcvtrk(active, starting, since 0.00324487s) 2024-02-19T18:55:37.670 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T18:55:37.670 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr metadata", "who": "smithi114.pcvtrk", "id": "smithi114.pcvtrk"}]: dispatch 2024-02-19T18:55:37.670 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T18:55:37.670 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T18:55:37.670 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T18:55:37.670 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: Manager daemon smithi114.pcvtrk is now available 2024-02-19T18:55:37.670 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.670 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.670 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.670 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:55:37.670 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T18:55:37.670 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/trash_purge_schedule"}]: dispatch 2024-02-19T18:55:37.670 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/1163781147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.671 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mgrmap e16: smithi114.pcvtrk(active, since 1.00527s) 2024-02-19T18:55:37.671 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.671 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: [19/Feb/2024:18:55:03] ENGINE Bus STARTING 2024-02-19T18:55:37.671 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: [19/Feb/2024:18:55:03] ENGINE Serving on http://172.21.15.114:8765 2024-02-19T18:55:37.671 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: [19/Feb/2024:18:55:03] ENGINE Serving on https://172.21.15.114:7150 2024-02-19T18:55:37.671 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: [19/Feb/2024:18:55:03] ENGINE Bus STARTED 2024-02-19T18:55:37.671 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.671 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.671 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mgrmap e17: smithi114.pcvtrk(active, since 2s) 2024-02-19T18:55:37.671 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/3750725451' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.671 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.671 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.671 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.671 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.672 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:55:37.672 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.672 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.672 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.672 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.672 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:55:37.672 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:37.672 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:55:37.672 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: Updating smithi114:/etc/ceph/ceph.conf 2024-02-19T18:55:37.672 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: Updating smithi174:/etc/ceph/ceph.conf 2024-02-19T18:55:37.672 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:55:37.672 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:55:37.672 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-02-19T18:55:37.672 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-02-19T18:55:37.673 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T18:55:37.673 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T18:55:37.673 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.673 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.673 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.673 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.673 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.673 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi174", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T18:55:37.673 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi174", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-02-19T18:55:37.673 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:37.673 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: Deploying daemon ceph-exporter.smithi174 on smithi174 2024-02-19T18:55:37.673 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/1998393146' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.673 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/678344833' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.673 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.674 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.674 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.674 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.674 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi174", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:55:37.674 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi174", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-19T18:55:37.674 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:37.674 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: Deploying daemon crash.smithi174 on smithi174 2024-02-19T18:55:37.674 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.674 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:55:37.674 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/2587139701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.674 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.674 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.674 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.674 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.674 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: Deploying daemon node-exporter.smithi174 on smithi174 2024-02-19T18:55:37.675 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.675 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.675 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/1515106916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.675 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.675 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.675 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.675 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.675 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/1202793726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.675 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.675 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.675 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi174.lydvnj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T18:55:37.675 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi174.lydvnj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-19T18:55:37.675 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T18:55:37.675 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:37.675 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: Deploying daemon mgr.smithi174.lydvnj on smithi174 2024-02-19T18:55:37.676 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.676 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.676 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/100998210' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:37.676 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.676 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.676 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.676 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.676 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T18:55:37.676 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:37.676 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: Deploying daemon mon.smithi174 on smithi174 2024-02-19T18:55:37.676 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:55:37.676 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.676 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:37.676 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:37.677 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mon.smithi174@-1(synchronizing).paxosservice(auth 1..8) refresh upgraded, format 0 -> 3 2024-02-19T18:55:37.677 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: expand_channel_meta expand map: {default=false} 2024-02-19T18:55:37.677 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: expand_channel_meta from 'false' to 'false' 2024-02-19T18:55:37.677 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: expand_channel_meta expanded map: {default=false} 2024-02-19T18:55:37.677 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: expand_channel_meta expand map: {default=info} 2024-02-19T18:55:37.677 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: expand_channel_meta from 'info' to 'info' 2024-02-19T18:55:37.677 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: expand_channel_meta expanded map: {default=info} 2024-02-19T18:55:37.677 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: expand_channel_meta expand map: {default=daemon} 2024-02-19T18:55:37.677 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: expand_channel_meta from 'daemon' to 'daemon' 2024-02-19T18:55:37.677 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: expand_channel_meta expanded map: {default=daemon} 2024-02-19T18:55:37.677 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: expand_channel_meta expand map: {default=debug} 2024-02-19T18:55:37.677 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: expand_channel_meta from 'debug' to 'debug' 2024-02-19T18:55:37.677 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: expand_channel_meta expanded map: {default=debug} 2024-02-19T18:55:37.677 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 ceph-mon[41061]: mon.smithi174@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-02-19T18:55:37.678 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 bash[41020]: 70dd7bf69fd96add53ce87c186fe201334b327ce7670fe8543f403bfc2a040bf 2024-02-19T18:55:37.678 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:37 smithi174 systemd[1]: Started Ceph mon.smithi174 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T18:55:38.305 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:55:38.306 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:55:38.530 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi174/config 2024-02-19T18:55:42.891 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:55:42.891 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T18:55:42.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: mon.smithi114 calling monitor election 2024-02-19T18:55:42.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:55:42.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/313262388' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/crt"}]: dispatch 2024-02-19T18:55:42.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:42.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:55:42.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: mon.smithi174 calling monitor election 2024-02-19T18:55:42.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:55:42.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:42.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:55:42.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:55:42.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: mon.smithi114 is new leader, mons smithi114,smithi174 in quorum (ranks 0,1) 2024-02-19T18:55:42.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: monmap e2: 2 mons at {smithi114=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],smithi174=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-19T18:55:42.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: fsmap 2024-02-19T18:55:42.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: osdmap e5: 0 total, 0 up, 0 in 2024-02-19T18:55:42.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: mgrmap e17: smithi114.pcvtrk(active, since 39s) 2024-02-19T18:55:42.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: Standby manager daemon smithi174.lydvnj started 2024-02-19T18:55:42.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/313262388' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T18:55:42.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: overall HEALTH_OK 2024-02-19T18:55:42.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/313262388' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/key"}]: dispatch 2024-02-19T18:55:42.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/313262388' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T18:55:42.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:42.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:42.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:42.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:42.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:42 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:43.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:55:43.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T18:55:43.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: mon.smithi114 calling monitor election 2024-02-19T18:55:43.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:55:43.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.174:0/313262388' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/crt"}]: dispatch 2024-02-19T18:55:43.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:43.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:55:43.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: mon.smithi174 calling monitor election 2024-02-19T18:55:43.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:55:43.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:43.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:55:43.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:55:43.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: mon.smithi114 is new leader, mons smithi114,smithi174 in quorum (ranks 0,1) 2024-02-19T18:55:43.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: monmap e2: 2 mons at {smithi114=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],smithi174=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-19T18:55:43.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: fsmap 2024-02-19T18:55:43.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: osdmap e5: 0 total, 0 up, 0 in 2024-02-19T18:55:43.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: mgrmap e17: smithi114.pcvtrk(active, since 39s) 2024-02-19T18:55:43.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: Standby manager daemon smithi174.lydvnj started 2024-02-19T18:55:43.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.174:0/313262388' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T18:55:43.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: overall HEALTH_OK 2024-02-19T18:55:43.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.174:0/313262388' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/key"}]: dispatch 2024-02-19T18:55:43.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.174:0/313262388' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T18:55:43.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:43.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:43.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:43.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:43.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:42 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:43.043 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.025+0000 7f5bb70ed700 1 -- 172.21.15.174:0/416139721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ba00035c0 msgr2=0x7f5ba0005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:43.043 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.025+0000 7f5bb70ed700 1 --2- 172.21.15.174:0/416139721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ba00035c0 0x7f5ba0005a50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0076780 tx=0x7f5ba400feb0 comp rx=0 tx=0).stop 2024-02-19T18:55:43.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.025+0000 7f5bb70ed700 1 -- 172.21.15.174:0/416139721 shutdown_connections 2024-02-19T18:55:43.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.025+0000 7f5bb70ed700 1 --2- 172.21.15.174:0/416139721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ba00035c0 0x7f5ba0005a50 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:43.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.025+0000 7f5bb70ed700 1 -- 172.21.15.174:0/416139721 >> 172.21.15.174:0/416139721 conn(0x7f5bb0075340 msgr2=0x7f5bb0075750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:43.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.025+0000 7f5bb70ed700 1 -- 172.21.15.174:0/416139721 shutdown_connections 2024-02-19T18:55:43.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.025+0000 7f5bb70ed700 1 -- 172.21.15.174:0/416139721 wait complete. 2024-02-19T18:55:43.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.025+0000 7f5bb70ed700 1 Processor -- start 2024-02-19T18:55:43.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.025+0000 7f5bb70ed700 1 -- start start 2024-02-19T18:55:43.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.025+0000 7f5bb70ed700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ba00035c0 0x7f5bb0101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:43.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.025+0000 7f5bb70ed700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb0101950 0x7f5bb0101dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:43.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.025+0000 7f5bb70ed700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb01a21d0 con 0x7f5ba00035c0 2024-02-19T18:55:43.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.025+0000 7f5bb70ed700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb01a2310 con 0x7f5bb0101950 2024-02-19T18:55:43.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.026+0000 7f5baffff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb0101950 0x7f5bb0101dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:43.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.026+0000 7f5bb4e89700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ba00035c0 0x7f5bb0101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:43.044 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.026+0000 7f5baffff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb0101950 0x7f5bb0101dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:41588/0 (socket says 172.21.15.174:41588) 2024-02-19T18:55:43.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.026+0000 7f5baffff700 1 -- 172.21.15.174:0/924973409 learned_addr learned my addr 172.21.15.174:0/924973409 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:55:43.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.026+0000 7f5baffff700 1 -- 172.21.15.174:0/924973409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb0101950 msgr2=0x7f5bb0101dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-02-19T18:55:43.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.026+0000 7f5baffff700 1 -- 172.21.15.174:0/924973409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb0101950 msgr2=0x7f5bb0101dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-19T18:55:43.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.026+0000 7f5baffff700 1 --2- 172.21.15.174:0/924973409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb0101950 0x7f5bb0101dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-19T18:55:43.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.026+0000 7f5baffff700 1 --2- 172.21.15.174:0/924973409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb0101950 0x7f5bb0101dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:55:43.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.026+0000 7f5bb4e89700 1 -- 172.21.15.174:0/924973409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb0101950 msgr2=0x7f5bb0101dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:55:43.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.026+0000 7f5bb4e89700 1 --2- 172.21.15.174:0/924973409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb0101950 0x7f5bb0101dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:43.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.026+0000 7f5bb4e89700 1 -- 172.21.15.174:0/924973409 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba400f900 con 0x7f5ba00035c0 2024-02-19T18:55:43.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.026+0000 7f5bb4e89700 1 --2- 172.21.15.174:0/924973409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ba00035c0 0x7f5bb0101410 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0076780 tx=0x7f5ba4004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:43.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.026+0000 7f5badffb700 1 -- 172.21.15.174:0/924973409 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ba4004b50 con 0x7f5ba00035c0 2024-02-19T18:55:43.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.026+0000 7f5bb70ed700 1 -- 172.21.15.174:0/924973409 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb01a2450 con 0x7f5ba00035c0 2024-02-19T18:55:43.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.026+0000 7f5badffb700 1 -- 172.21.15.174:0/924973409 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5ba4013e10 con 0x7f5ba00035c0 2024-02-19T18:55:43.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.026+0000 7f5badffb700 1 -- 172.21.15.174:0/924973409 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ba4011cd0 con 0x7f5ba00035c0 2024-02-19T18:55:43.045 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.027+0000 7f5bb70ed700 1 -- 172.21.15.174:0/924973409 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb01a2940 con 0x7f5ba00035c0 2024-02-19T18:55:43.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.028+0000 7f5bb70ed700 1 -- 172.21.15.174:0/924973409 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb00623c0 con 0x7f5ba00035c0 2024-02-19T18:55:43.046 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.028+0000 7f5badffb700 1 -- 172.21.15.174:0/924973409 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f5ba4013960 con 0x7f5ba00035c0 2024-02-19T18:55:43.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.029+0000 7f5badffb700 1 --2- 172.21.15.174:0/924973409 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5b9406c560 0x7f5b9406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:43.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.029+0000 7f5badffb700 1 -- 172.21.15.174:0/924973409 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f5ba408d070 con 0x7f5ba00035c0 2024-02-19T18:55:43.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.029+0000 7f5baffff700 1 --2- 172.21.15.174:0/924973409 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5b9406c560 0x7f5b9406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:43.047 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.029+0000 7f5baffff700 1 --2- 172.21.15.174:0/924973409 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5b9406c560 0x7f5b9406ea20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009180 tx=0x7f5b9c008040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:43.051 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.034+0000 7f5badffb700 1 -- 172.21.15.174:0/924973409 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5ba40588d0 con 0x7f5ba00035c0 2024-02-19T18:55:43.339 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.322+0000 7f5bb70ed700 1 -- 172.21.15.174:0/924973409 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5bb01a2d90 con 0x7f5ba00035c0 2024-02-19T18:55:43.339 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.323+0000 7f5badffb700 1 -- 172.21.15.174:0/924973409 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1026 (secure 0 0 0) 0x7f5ba4031090 con 0x7f5ba00035c0 2024-02-19T18:55:43.340 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-02-19T18:55:43.340 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":2,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:55:37.455016Z","created":"2024-02-19T18:51:30.435772Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi114","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi174","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-19T18:55:43.342 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.324+0000 7f5bb70ed700 1 -- 172.21.15.174:0/924973409 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5b9406c560 msgr2=0x7f5b9406ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:43.342 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.325+0000 7f5bb70ed700 1 --2- 172.21.15.174:0/924973409 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5b9406c560 0x7f5b9406ea20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009180 tx=0x7f5b9c008040 comp rx=0 tx=0).stop 2024-02-19T18:55:43.342 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.325+0000 7f5bb70ed700 1 -- 172.21.15.174:0/924973409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ba00035c0 msgr2=0x7f5bb0101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:43.342 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.325+0000 7f5bb70ed700 1 --2- 172.21.15.174:0/924973409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ba00035c0 0x7f5bb0101410 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0076780 tx=0x7f5ba4004f90 comp rx=0 tx=0).stop 2024-02-19T18:55:43.342 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.325+0000 7f5bb70ed700 1 -- 172.21.15.174:0/924973409 shutdown_connections 2024-02-19T18:55:43.342 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.325+0000 7f5bb70ed700 1 --2- 172.21.15.174:0/924973409 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5b9406c560 0x7f5b9406ea20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:43.342 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.325+0000 7f5bb70ed700 1 --2- 172.21.15.174:0/924973409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ba00035c0 0x7f5bb0101410 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:43.342 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.325+0000 7f5bb70ed700 1 --2- 172.21.15.174:0/924973409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb0101950 0x7f5bb0101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:43.342 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.325+0000 7f5bb70ed700 1 -- 172.21.15.174:0/924973409 >> 172.21.15.174:0/924973409 conn(0x7f5bb0075340 msgr2=0x7f5bb0103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:43.342 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.325+0000 7f5bb70ed700 1 -- 172.21.15.174:0/924973409 shutdown_connections 2024-02-19T18:55:43.342 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:55:43.325+0000 7f5bb70ed700 1 -- 172.21.15.174:0/924973409 wait complete. 2024-02-19T18:55:43.343 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 2 2024-02-19T18:55:43.641 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:43 smithi174 ceph-mon[41061]: mgrmap e18: smithi114.pcvtrk(active, since 39s), standbys: smithi174.lydvnj 2024-02-19T18:55:43.641 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:43 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr metadata", "who": "smithi174.lydvnj", "id": "smithi174.lydvnj"}]: dispatch 2024-02-19T18:55:43.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:43 smithi174 ceph-mon[41061]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:43.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:43 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:43.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:43 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:43.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:43 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/924973409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:43.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:43 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:55:44.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:43 smithi114 ceph-mon[29441]: mgrmap e18: smithi114.pcvtrk(active, since 39s), standbys: smithi174.lydvnj 2024-02-19T18:55:44.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:43 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr metadata", "who": "smithi174.lydvnj", "id": "smithi174.lydvnj"}]: dispatch 2024-02-19T18:55:44.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:43 smithi114 ceph-mon[29441]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:44.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:43 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:44.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:43 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:44.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:43 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/924973409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:55:44.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:43 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:55:44.044 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-19T18:55:44.045 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph config generate-minimal-conf 2024-02-19T18:55:44.264 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:55:45.891 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:45 smithi174 ceph-mon[41061]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:45.891 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:45 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:45.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:45 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:45.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:45 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:45.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:45 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:55:45.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:45 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:45.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:45 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:45.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:45 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:45.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:45 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:45.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:45 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:45.949 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:45 smithi114 ceph-mon[29441]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:45.950 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:45 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:45.950 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:45 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:45.950 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:45 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:45.950 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:45 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:55:45.950 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:45 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:45.950 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:45 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:45.950 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:45 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:45.950 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:45 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:45.950 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:45 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:45.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.954+0000 7f004b9a0700 1 -- 172.21.15.114:0/3317556509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00441067a0 msgr2=0x7f0044106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:45.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.954+0000 7f004b9a0700 1 --2- 172.21.15.114:0/3317556509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00441067a0 0x7f0044106b80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0040009910 tx=0x7f0040009c20 comp rx=0 tx=0).stop 2024-02-19T18:55:45.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.955+0000 7f004b9a0700 1 -- 172.21.15.114:0/3317556509 shutdown_connections 2024-02-19T18:55:45.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.955+0000 7f004b9a0700 1 --2- 172.21.15.114:0/3317556509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00441067a0 0x7f0044106b80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:45.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.955+0000 7f004b9a0700 1 -- 172.21.15.114:0/3317556509 >> 172.21.15.114:0/3317556509 conn(0x7f00440fc290 msgr2=0x7f00440fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:45.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.955+0000 7f004b9a0700 1 -- 172.21.15.114:0/3317556509 shutdown_connections 2024-02-19T18:55:45.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.955+0000 7f004b9a0700 1 -- 172.21.15.114:0/3317556509 wait complete. 2024-02-19T18:55:45.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.955+0000 7f004b9a0700 1 Processor -- start 2024-02-19T18:55:45.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.955+0000 7f004b9a0700 1 -- start start 2024-02-19T18:55:45.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.956+0000 7f004b9a0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00441067a0 0x7f0044195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:45.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.956+0000 7f004b9a0700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00441963b0 0x7f004419a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:45.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.956+0000 7f004b9a0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00441969d0 con 0x7f00441067a0 2024-02-19T18:55:45.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.956+0000 7f004b9a0700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0044196b40 con 0x7f00441963b0 2024-02-19T18:55:45.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.956+0000 7f004973c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00441067a0 0x7f0044195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:45.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.956+0000 7f004973c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00441067a0 0x7f0044195e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43972/0 (socket says 172.21.15.114:43972) 2024-02-19T18:55:45.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.956+0000 7f004973c700 1 -- 172.21.15.114:0/2898120922 learned_addr learned my addr 172.21.15.114:0/2898120922 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:55:45.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.956+0000 7f004973c700 1 -- 172.21.15.114:0/2898120922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00441963b0 msgr2=0x7f004419a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:45.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.956+0000 7f004973c700 1 --2- 172.21.15.114:0/2898120922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00441963b0 0x7f004419a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:45.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.956+0000 7f004973c700 1 -- 172.21.15.114:0/2898120922 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00400095f0 con 0x7f00441067a0 2024-02-19T18:55:45.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.956+0000 7f004973c700 1 --2- 172.21.15.114:0/2898120922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00441067a0 0x7f0044195e70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0040005b40 tx=0x7f004000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:45.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.956+0000 7f003a7fc700 1 -- 172.21.15.114:0/2898120922 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f004001d070 con 0x7f00441067a0 2024-02-19T18:55:45.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.957+0000 7f004b9a0700 1 -- 172.21.15.114:0/2898120922 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f004419ae20 con 0x7f00441067a0 2024-02-19T18:55:45.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.957+0000 7f003a7fc700 1 -- 172.21.15.114:0/2898120922 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0040005140 con 0x7f00441067a0 2024-02-19T18:55:45.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.957+0000 7f003a7fc700 1 -- 172.21.15.114:0/2898120922 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0040017440 con 0x7f00441067a0 2024-02-19T18:55:45.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.957+0000 7f004b9a0700 1 -- 172.21.15.114:0/2898120922 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f004419b370 con 0x7f00441067a0 2024-02-19T18:55:45.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.957+0000 7f004b9a0700 1 -- 172.21.15.114:0/2898120922 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f004419b6a0 con 0x7f00441067a0 2024-02-19T18:55:45.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.959+0000 7f003a7fc700 1 -- 172.21.15.114:0/2898120922 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f004000f6f0 con 0x7f00441067a0 2024-02-19T18:55:45.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.959+0000 7f003a7fc700 1 --2- 172.21.15.114:0/2898120922 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f003006c560 0x7f003006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:45.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.959+0000 7f003a7fc700 1 -- 172.21.15.114:0/2898120922 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f004008bfa0 con 0x7f00441067a0 2024-02-19T18:55:45.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.959+0000 7f0048f3b700 1 --2- 172.21.15.114:0/2898120922 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f003006c560 0x7f003006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:45.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.959+0000 7f0048f3b700 1 --2- 172.21.15.114:0/2898120922 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f003006c560 0x7f003006ea20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f00340098e0 tx=0x7f0034006dd0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:45.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:45.963+0000 7f003a7fc700 1 -- 172.21.15.114:0/2898120922 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f00400578e0 con 0x7f00441067a0 2024-02-19T18:55:46.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:46.171+0000 7f004b9a0700 1 -- 172.21.15.114:0/2898120922 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f00440623c0 con 0x7f00441067a0 2024-02-19T18:55:46.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:46.171+0000 7f003a7fc700 1 -- 172.21.15.114:0/2898120922 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7f0040026020 con 0x7f00441067a0 2024-02-19T18:55:46.189 INFO:teuthology.orchestra.run.smithi114.stdout:# minimal ceph.conf for be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T18:55:46.189 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-02-19T18:55:46.189 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T18:55:46.189 INFO:teuthology.orchestra.run.smithi114.stdout: mon_host = [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-02-19T18:55:46.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:46.173+0000 7f004b9a0700 1 -- 172.21.15.114:0/2898120922 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f003006c560 msgr2=0x7f003006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:46.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:46.173+0000 7f004b9a0700 1 --2- 172.21.15.114:0/2898120922 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f003006c560 0x7f003006ea20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f00340098e0 tx=0x7f0034006dd0 comp rx=0 tx=0).stop 2024-02-19T18:55:46.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:46.173+0000 7f004b9a0700 1 -- 172.21.15.114:0/2898120922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00441067a0 msgr2=0x7f0044195e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:46.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:46.173+0000 7f004b9a0700 1 --2- 172.21.15.114:0/2898120922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00441067a0 0x7f0044195e70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0040005b40 tx=0x7f004000bda0 comp rx=0 tx=0).stop 2024-02-19T18:55:46.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:46.173+0000 7f004b9a0700 1 -- 172.21.15.114:0/2898120922 shutdown_connections 2024-02-19T18:55:46.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:46.173+0000 7f004b9a0700 1 --2- 172.21.15.114:0/2898120922 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f003006c560 0x7f003006ea20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:46.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:46.173+0000 7f004b9a0700 1 --2- 172.21.15.114:0/2898120922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00441067a0 0x7f0044195e70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:46.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:46.173+0000 7f004b9a0700 1 --2- 172.21.15.114:0/2898120922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00441963b0 0x7f004419a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:46.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:46.173+0000 7f004b9a0700 1 -- 172.21.15.114:0/2898120922 >> 172.21.15.114:0/2898120922 conn(0x7f00440fc290 msgr2=0x7f00441022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:46.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:46.174+0000 7f004b9a0700 1 -- 172.21.15.114:0/2898120922 shutdown_connections 2024-02-19T18:55:46.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:46.174+0000 7f004b9a0700 1 -- 172.21.15.114:0/2898120922 wait complete. 2024-02-19T18:55:47.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:46 smithi114 ceph-mon[29441]: Updating smithi114:/etc/ceph/ceph.conf 2024-02-19T18:55:47.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:46 smithi114 ceph-mon[29441]: Updating smithi174:/etc/ceph/ceph.conf 2024-02-19T18:55:47.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:46 smithi114 ceph-mon[29441]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:55:47.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:46 smithi114 ceph-mon[29441]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:55:47.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:46 smithi114 ceph-mon[29441]: Reconfiguring mon.smithi114 (unknown last config time)... 2024-02-19T18:55:47.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:46 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T18:55:47.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:46 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T18:55:47.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:46 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:47.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:46 smithi114 ceph-mon[29441]: Reconfiguring daemon mon.smithi114 on smithi114 2024-02-19T18:55:47.033 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:46 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/2898120922' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:47.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:46 smithi174 ceph-mon[41061]: Updating smithi114:/etc/ceph/ceph.conf 2024-02-19T18:55:47.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:46 smithi174 ceph-mon[41061]: Updating smithi174:/etc/ceph/ceph.conf 2024-02-19T18:55:47.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:46 smithi174 ceph-mon[41061]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:55:47.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:46 smithi174 ceph-mon[41061]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:55:47.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:46 smithi174 ceph-mon[41061]: Reconfiguring mon.smithi114 (unknown last config time)... 2024-02-19T18:55:47.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:46 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T18:55:47.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:46 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T18:55:47.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:46 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:47.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:46 smithi174 ceph-mon[41061]: Reconfiguring daemon mon.smithi114 on smithi114 2024-02-19T18:55:47.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:46 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/2898120922' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:47.693 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-19T18:55:47.693 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T18:55:47.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.conf 2024-02-19T18:55:47.723 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T18:55:47.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T18:55:47.790 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-02-19T18:55:47.790 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.conf 2024-02-19T18:55:47.824 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-02-19T18:55:47.825 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T18:55:47.894 INFO:tasks.cephadm:Deploying OSDs... 2024-02-19T18:55:47.895 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T18:55:47.895 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-02-19T18:55:47.910 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-19T18:55:47.910 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-02-19T18:55:47.935 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:47 smithi114 ceph-mon[29441]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:47.935 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:47 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:47.935 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:47 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:47.935 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:47 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi114.pcvtrk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T18:55:47.935 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:47 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T18:55:47.935 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:47 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:47.935 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:47 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:55:47.949 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-19T18:55:47.949 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:55:47.949 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-02-19T18:55:47.950 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:55:47.950 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:55:47.950 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-02-19 18:54:54.182173361 +0000 2024-02-19T18:55:47.950 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-02-19 18:47:23.030964210 +0000 2024-02-19T18:55:47.950 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-02-19 18:47:23.030964210 +0000 2024-02-19T18:55:47.950 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-02-19 18:47:23.030964210 +0000 2024-02-19T18:55:47.950 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-19T18:55:48.015 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-02-19T18:55:48.015 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-02-19T18:55:48.015 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000240063 s, 2.1 MB/s 2024-02-19T18:55:48.016 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-19T18:55:48.042 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:47 smithi174 ceph-mon[41061]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:48.042 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:47 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:48.043 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:47 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:48.043 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:47 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi114.pcvtrk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T18:55:48.043 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:47 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T18:55:48.043 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:47 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:48.043 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:47 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:55:48.072 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-02-19T18:55:48.127 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-19T18:55:48.127 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:55:48.128 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-19T18:55:48.128 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:55:48.128 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:55:48.128 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-02-19 18:54:54.183173340 +0000 2024-02-19T18:55:48.128 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-02-19 18:47:23.381956720 +0000 2024-02-19T18:55:48.128 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-02-19 18:47:23.381956720 +0000 2024-02-19T18:55:48.128 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-02-19 18:47:23.381956720 +0000 2024-02-19T18:55:48.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-19T18:55:48.193 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-02-19T18:55:48.193 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-02-19T18:55:48.194 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000228609 s, 2.2 MB/s 2024-02-19T18:55:48.195 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-19T18:55:48.250 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-02-19T18:55:48.304 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-19T18:55:48.304 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:55:48.304 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-02-19T18:55:48.304 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:55:48.304 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:55:48.304 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-02-19 18:54:54.183173340 +0000 2024-02-19T18:55:48.304 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-02-19 18:47:23.727949337 +0000 2024-02-19T18:55:48.305 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-02-19 18:47:23.727949337 +0000 2024-02-19T18:55:48.305 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-02-19 18:47:23.727949337 +0000 2024-02-19T18:55:48.305 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-19T18:55:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-02-19T18:55:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-02-19T18:55:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000197386 s, 2.6 MB/s 2024-02-19T18:55:48.371 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-19T18:55:48.427 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-02-19T18:55:48.481 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-19T18:55:48.482 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:55:48.482 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-19T18:55:48.482 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:55:48.482 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:55:48.482 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-02-19 18:54:54.183173340 +0000 2024-02-19T18:55:48.482 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-02-19 18:47:24.068942061 +0000 2024-02-19T18:55:48.482 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-02-19 18:47:24.068942061 +0000 2024-02-19T18:55:48.482 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-02-19 18:47:24.068942061 +0000 2024-02-19T18:55:48.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-19T18:55:48.547 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-02-19T18:55:48.547 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-02-19T18:55:48.547 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000222208 s, 2.3 MB/s 2024-02-19T18:55:48.548 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-19T18:55:48.604 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-02-19T18:55:48.604 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-02-19T18:55:48.619 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-19T18:55:48.620 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_1 2024-02-19T18:55:48.675 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-19T18:55:48.675 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:55:48.675 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-19T18:55:48.675 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:55:48.675 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:55:48.675 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-02-19 18:55:34.955715691 +0000 2024-02-19T18:55:48.675 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-02-19 18:47:23.029165034 +0000 2024-02-19T18:55:48.675 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-02-19 18:47:23.029165034 +0000 2024-02-19T18:55:48.675 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-02-19 18:47:23.029165034 +0000 2024-02-19T18:55:48.676 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-19T18:55:48.741 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-02-19T18:55:48.741 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-02-19T18:55:48.741 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000212079 s, 2.4 MB/s 2024-02-19T18:55:48.742 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-19T18:55:48.798 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_2 2024-02-19T18:55:48.852 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-19T18:55:48.852 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:55:48.852 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-19T18:55:48.852 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:55:48.852 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:55:48.852 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-02-19 18:55:34.955715691 +0000 2024-02-19T18:55:48.852 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-02-19 18:47:23.390160398 +0000 2024-02-19T18:55:48.852 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-02-19 18:47:23.390160398 +0000 2024-02-19T18:55:48.853 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-02-19 18:47:23.390160398 +0000 2024-02-19T18:55:48.853 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-19T18:55:48.918 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-02-19T18:55:48.918 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-02-19T18:55:48.918 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000235184 s, 2.2 MB/s 2024-02-19T18:55:48.920 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-19T18:55:48.975 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_3 2024-02-19T18:55:49.029 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-19T18:55:49.029 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:55:49.029 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-02-19T18:55:49.029 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:55:49.029 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:55:49.029 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-02-19 18:55:34.956715678 +0000 2024-02-19T18:55:49.029 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-02-19 18:47:23.749155788 +0000 2024-02-19T18:55:49.029 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-02-19 18:47:23.749155788 +0000 2024-02-19T18:55:49.030 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-02-19 18:47:23.749155788 +0000 2024-02-19T18:55:49.030 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-19T18:55:49.093 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:48 smithi174 ceph-mon[41061]: Reconfiguring mgr.smithi114.pcvtrk (unknown last config time)... 2024-02-19T18:55:49.093 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:48 smithi174 ceph-mon[41061]: Reconfiguring daemon mgr.smithi114.pcvtrk on smithi114 2024-02-19T18:55:49.095 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-02-19T18:55:49.095 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-02-19T18:55:49.095 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000201525 s, 2.5 MB/s 2024-02-19T18:55:49.096 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-19T18:55:49.152 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_4 2024-02-19T18:55:49.206 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-19T18:55:49.206 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:55:49.206 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-02-19T18:55:49.206 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:55:49.206 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:55:49.206 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-02-19 18:55:34.956715678 +0000 2024-02-19T18:55:49.206 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-02-19 18:47:24.088151435 +0000 2024-02-19T18:55:49.206 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-02-19 18:47:24.088151435 +0000 2024-02-19T18:55:49.206 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-02-19 18:47:24.088151435 +0000 2024-02-19T18:55:49.207 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-19T18:55:49.271 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-02-19T18:55:49.271 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-02-19T18:55:49.271 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000229322 s, 2.2 MB/s 2024-02-19T18:55:49.272 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-19T18:55:49.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:48 smithi114 ceph-mon[29441]: Reconfiguring mgr.smithi114.pcvtrk (unknown last config time)... 2024-02-19T18:55:49.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:48 smithi114 ceph-mon[29441]: Reconfiguring daemon mgr.smithi114.pcvtrk on smithi114 2024-02-19T18:55:49.328 INFO:tasks.cephadm:Deploying osd.0 on smithi114 with /dev/vg_nvme/lv_4... 2024-02-19T18:55:49.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-19T18:55:50.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:49 smithi114 ceph-mon[29441]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:50.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:49 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:50.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:49 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:50.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:49 smithi114 ceph-mon[29441]: Reconfiguring ceph-exporter.smithi114 (monmap changed)... 2024-02-19T18:55:50.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:49 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi114", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T18:55:50.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:49 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:50.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:49 smithi114 ceph-mon[29441]: Reconfiguring daemon ceph-exporter.smithi114 on smithi114 2024-02-19T18:55:50.378 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:55:50.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:49 smithi174 ceph-mon[41061]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:50.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:49 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:50.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:49 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:50.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:49 smithi174 ceph-mon[41061]: Reconfiguring ceph-exporter.smithi114 (monmap changed)... 2024-02-19T18:55:50.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:49 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi114", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T18:55:50.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:49 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:50.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:49 smithi174 ceph-mon[41061]: Reconfiguring daemon ceph-exporter.smithi114 on smithi114 2024-02-19T18:55:51.604 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:51 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:51.604 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:51 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:51.604 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:51 smithi114 ceph-mon[29441]: Reconfiguring crash.smithi114 (monmap changed)... 2024-02-19T18:55:51.604 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:51 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi114", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:55:51.604 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:51 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:51.604 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:51 smithi114 ceph-mon[29441]: Reconfiguring daemon crash.smithi114 on smithi114 2024-02-19T18:55:51.605 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:51 smithi114 ceph-mon[29441]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:51.641 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:51 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:51.641 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:51 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:51.641 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:51 smithi174 ceph-mon[41061]: Reconfiguring crash.smithi114 (monmap changed)... 2024-02-19T18:55:51.641 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:51 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi114", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:55:51.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:51 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:51.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:51 smithi174 ceph-mon[41061]: Reconfiguring daemon crash.smithi114 on smithi114 2024-02-19T18:55:51.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:51 smithi174 ceph-mon[41061]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:53.192 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-19T18:55:53.192 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-19T18:55:53.192 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-02-19T18:55:53.192 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-02-19T18:55:53.192 INFO:teuthology.orchestra.run.smithi114.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0134046 s, 782 MB/s 2024-02-19T18:55:53.192 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-02-19T18:55:53.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:53 smithi114 ceph-mon[29441]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:53.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:53 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:53.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:53 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:53.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:53 smithi114 ceph-mon[29441]: Reconfiguring alertmanager.smithi114 (dependencies changed)... 2024-02-19T18:55:53.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:53 smithi114 ceph-mon[29441]: Reconfiguring daemon alertmanager.smithi114 on smithi114 2024-02-19T18:55:54.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:53 smithi174 ceph-mon[41061]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:54.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:53 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:54.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:53 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:55:54.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:53 smithi174 ceph-mon[41061]: Reconfiguring alertmanager.smithi114 (dependencies changed)... 2024-02-19T18:55:54.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:53 smithi174 ceph-mon[41061]: Reconfiguring daemon alertmanager.smithi114 on smithi114 2024-02-19T18:55:54.580 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi114:vg_nvme/lv_4 2024-02-19T18:55:55.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:55 smithi114 ceph-mon[29441]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:55.891 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:55 smithi174 ceph-mon[41061]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:56.372 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:55:58.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:57 smithi114 ceph-mon[29441]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:58.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:57 smithi174 ceph-mon[41061]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:55:58.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.573+0000 7f61ba55f700 1 -- 172.21.15.114:0/1005671029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b4074dc0 msgr2=0x7f61b4073220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:58.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.573+0000 7f61ba55f700 1 --2- 172.21.15.114:0/1005671029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b4074dc0 0x7f61b4073220 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009910 tx=0x7f61a4009c20 comp rx=0 tx=0).stop 2024-02-19T18:55:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.573+0000 7f61ba55f700 1 -- 172.21.15.114:0/1005671029 shutdown_connections 2024-02-19T18:55:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.573+0000 7f61ba55f700 1 --2- 172.21.15.114:0/1005671029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61b40737f0 0x7f61b4073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.573+0000 7f61ba55f700 1 --2- 172.21.15.114:0/1005671029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b4074dc0 0x7f61b4073220 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.573+0000 7f61ba55f700 1 -- 172.21.15.114:0/1005671029 >> 172.21.15.114:0/1005671029 conn(0x7f61b40fc140 msgr2=0x7f61b40fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.573+0000 7f61ba55f700 1 -- 172.21.15.114:0/1005671029 shutdown_connections 2024-02-19T18:55:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.573+0000 7f61ba55f700 1 -- 172.21.15.114:0/1005671029 wait complete. 2024-02-19T18:55:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61ba55f700 1 Processor -- start 2024-02-19T18:55:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61ba55f700 1 -- start start 2024-02-19T18:55:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61ba55f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b40737f0 0x7f61b4198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61ba55f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61b4074dc0 0x7f61b4198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61ba55f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b41991c0 con 0x7f61b40737f0 2024-02-19T18:55:58.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61ba55f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b4199300 con 0x7f61b4074dc0 2024-02-19T18:55:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61b3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b40737f0 0x7f61b4198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61b3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b40737f0 0x7f61b4198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47210/0 (socket says 172.21.15.114:47210) 2024-02-19T18:55:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61b3fff700 1 -- 172.21.15.114:0/2387419835 learned_addr learned my addr 172.21.15.114:0/2387419835 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:55:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61abfff700 1 --2- 172.21.15.114:0/2387419835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61b4074dc0 0x7f61b4198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61b3fff700 1 -- 172.21.15.114:0/2387419835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61b4074dc0 msgr2=0x7f61b4198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61b3fff700 1 --2- 172.21.15.114:0/2387419835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61b4074dc0 0x7f61b4198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61b3fff700 1 -- 172.21.15.114:0/2387419835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a40095f0 con 0x7f61b40737f0 2024-02-19T18:55:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61abfff700 1 --2- 172.21.15.114:0/2387419835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61b4074dc0 0x7f61b4198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T18:55:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61b3fff700 1 --2- 172.21.15.114:0/2387419835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b40737f0 0x7f61b4198660 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f61a400bda0 tx=0x7f61a400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61b1ffb700 1 -- 172.21.15.114:0/2387419835 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61a401d070 con 0x7f61b40737f0 2024-02-19T18:55:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61b1ffb700 1 -- 172.21.15.114:0/2387419835 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f61a4004c50 con 0x7f61b40737f0 2024-02-19T18:55:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.574+0000 7f61ba55f700 1 -- 172.21.15.114:0/2387419835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b419dd50 con 0x7f61b40737f0 2024-02-19T18:55:58.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.575+0000 7f61b1ffb700 1 -- 172.21.15.114:0/2387419835 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61a400f690 con 0x7f61b40737f0 2024-02-19T18:55:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.575+0000 7f61ba55f700 1 -- 172.21.15.114:0/2387419835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b419e260 con 0x7f61b40737f0 2024-02-19T18:55:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.576+0000 7f61ba55f700 1 -- 172.21.15.114:0/2387419835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b410ba40 con 0x7f61b40737f0 2024-02-19T18:55:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.576+0000 7f61b1ffb700 1 -- 172.21.15.114:0/2387419835 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f61a4022a20 con 0x7f61b40737f0 2024-02-19T18:55:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.576+0000 7f61b1ffb700 1 --2- 172.21.15.114:0/2387419835 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f61a006c460 0x7f61a006e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.577+0000 7f61b1ffb700 1 -- 172.21.15.114:0/2387419835 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f61a408bef0 con 0x7f61b40737f0 2024-02-19T18:55:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.577+0000 7f61abfff700 1 --2- 172.21.15.114:0/2387419835 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f61a006c460 0x7f61a006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:58.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.577+0000 7f61abfff700 1 --2- 172.21.15.114:0/2387419835 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f61a006c460 0x7f61a006e920 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f61b4074af0 tx=0x7f619c008040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:58.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.581+0000 7f61b1ffb700 1 -- 172.21.15.114:0/2387419835 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f61a405b240 con 0x7f61b40737f0 2024-02-19T18:55:58.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:55:58.791+0000 7f61ba55f700 1 -- 172.21.15.114:0/2387419835 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f61b41080c0 con 0x7f61a006c460 2024-02-19T18:55:58.996 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:58 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:55:58.996 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:58 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:55:58.996 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:58 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:59.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:58 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:55:59.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:58 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:55:59.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:58 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:00.188 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:59 smithi114 ceph-mon[29441]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:00.188 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:55:59 smithi114 ceph-mon[29441]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:56:00.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:59 smithi174 ceph-mon[41061]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:00.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:55:59 smithi174 ceph-mon[41061]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:56:02.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:01 smithi174 ceph-mon[41061]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:02.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:01 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:02.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:01 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:02.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:01 smithi174 ceph-mon[41061]: Reconfiguring grafana.smithi114 (dependencies changed)... 2024-02-19T18:56:02.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:01 smithi174 ceph-mon[41061]: Reconfiguring daemon grafana.smithi114 on smithi114 2024-02-19T18:56:02.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:01 smithi114 ceph-mon[29441]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:02.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:01 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:02.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:01 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:02.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:01 smithi114 ceph-mon[29441]: Reconfiguring grafana.smithi114 (dependencies changed)... 2024-02-19T18:56:02.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:01 smithi114 ceph-mon[29441]: Reconfiguring daemon grafana.smithi114 on smithi114 2024-02-19T18:56:03.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:02 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/2730114737' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93a2d632-7629-4a15-b026-93b40fba59d7"}]: dispatch 2024-02-19T18:56:03.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:02 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/2730114737' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93a2d632-7629-4a15-b026-93b40fba59d7"}]': finished 2024-02-19T18:56:03.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:02 smithi114 ceph-mon[29441]: osdmap e6: 1 total, 0 up, 1 in 2024-02-19T18:56:03.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:02 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:56:03.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:02 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1816441958' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:56:03.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:02 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:56:03.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:02 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/2730114737' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93a2d632-7629-4a15-b026-93b40fba59d7"}]: dispatch 2024-02-19T18:56:03.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:02 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/2730114737' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93a2d632-7629-4a15-b026-93b40fba59d7"}]': finished 2024-02-19T18:56:03.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:02 smithi174 ceph-mon[41061]: osdmap e6: 1 total, 0 up, 1 in 2024-02-19T18:56:03.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:02 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:56:03.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:02 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1816441958' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:56:03.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:02 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:56:04.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:03 smithi114 ceph-mon[29441]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:04.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:03 smithi174 ceph-mon[41061]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:05.940 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:05 smithi114 ceph-mon[29441]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:06.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:05 smithi174 ceph-mon[41061]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:08.212 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:07 smithi114 ceph-mon[29441]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:08.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:07 smithi174 ceph-mon[41061]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:10.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:09 smithi114 ceph-mon[29441]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:10.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:09 smithi174 ceph-mon[41061]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:12.214 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:11 smithi114 ceph-mon[29441]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:12.214 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:11 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:12.214 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:11 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:12.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:11 smithi174 ceph-mon[41061]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:12.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:11 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:12.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:11 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:13.163 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:12 smithi114 ceph-mon[29441]: Reconfiguring prometheus.smithi114 (dependencies changed)... 2024-02-19T18:56:13.163 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:12 smithi114 ceph-mon[29441]: Reconfiguring daemon prometheus.smithi114 on smithi114 2024-02-19T18:56:13.163 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:12 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-19T18:56:13.163 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:12 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:13.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:12 smithi174 ceph-mon[41061]: Reconfiguring prometheus.smithi114 (dependencies changed)... 2024-02-19T18:56:13.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:12 smithi174 ceph-mon[41061]: Reconfiguring daemon prometheus.smithi114 on smithi114 2024-02-19T18:56:13.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:12 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-19T18:56:13.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:12 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:14.136 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:13 smithi114 ceph-mon[29441]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:14.136 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:13 smithi114 ceph-mon[29441]: Deploying daemon osd.0 on smithi114 2024-02-19T18:56:14.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:13 smithi174 ceph-mon[41061]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:14.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:13 smithi174 ceph-mon[41061]: Deploying daemon osd.0 on smithi114 2024-02-19T18:56:15.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:14 smithi114 ceph-mon[29441]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:15.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:14 smithi174 ceph-mon[41061]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:17.887 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:17 smithi174 ceph-mon[41061]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:17.887 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:17 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:17.887 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:17 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:17.887 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:17 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi174", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T18:56:17.887 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:17 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:17.887 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:17 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:56:18.008 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:17 smithi114 ceph-mon[29441]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:18.008 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:17 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:18.008 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:17 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:18.008 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:17 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi174", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T18:56:18.008 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:17 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:18.008 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:17 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:56:19.180 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:18 smithi174 ceph-mon[41061]: Reconfiguring ceph-exporter.smithi174 (monmap changed)... 2024-02-19T18:56:19.180 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:18 smithi174 ceph-mon[41061]: Reconfiguring daemon ceph-exporter.smithi174 on smithi174 2024-02-19T18:56:19.181 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:18 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:19.181 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:18 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:19.181 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:18 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi174", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:56:19.181 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:18 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:19.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:18 smithi114 ceph-mon[29441]: Reconfiguring ceph-exporter.smithi174 (monmap changed)... 2024-02-19T18:56:19.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:18 smithi114 ceph-mon[29441]: Reconfiguring daemon ceph-exporter.smithi174 on smithi174 2024-02-19T18:56:19.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:18 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:19.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:18 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:19.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:18 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi174", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:56:19.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:18 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:20.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:19 smithi174 ceph-mon[41061]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:20.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:19 smithi174 ceph-mon[41061]: Reconfiguring crash.smithi174 (monmap changed)... 2024-02-19T18:56:20.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:19 smithi174 ceph-mon[41061]: Reconfiguring daemon crash.smithi174 on smithi174 2024-02-19T18:56:20.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:19 smithi114 ceph-mon[29441]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:20.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:19 smithi114 ceph-mon[29441]: Reconfiguring crash.smithi174 (monmap changed)... 2024-02-19T18:56:20.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:19 smithi114 ceph-mon[29441]: Reconfiguring daemon crash.smithi174 on smithi174 2024-02-19T18:56:21.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:21 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:21.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:21 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:21.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:21 smithi114 ceph-mon[29441]: Reconfiguring mgr.smithi174.lydvnj (monmap changed)... 2024-02-19T18:56:21.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:21 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi174.lydvnj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T18:56:21.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:21 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T18:56:21.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:21 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:21.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:21 smithi114 ceph-mon[29441]: Reconfiguring daemon mgr.smithi174.lydvnj on smithi174 2024-02-19T18:56:21.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:21 smithi114 ceph-mon[29441]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:21.594 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:21 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:21.594 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:21 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:21.595 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:21 smithi174 ceph-mon[41061]: Reconfiguring mgr.smithi174.lydvnj (monmap changed)... 2024-02-19T18:56:21.595 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:21 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi174.lydvnj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T18:56:21.595 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:21 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T18:56:21.595 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:21 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:21.595 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:21 smithi174 ceph-mon[41061]: Reconfiguring daemon mgr.smithi174.lydvnj on smithi174 2024-02-19T18:56:21.595 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:21 smithi174 ceph-mon[41061]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:22.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:22 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:22.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:22 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:22.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:22 smithi114 ceph-mon[29441]: Reconfiguring mon.smithi174 (monmap changed)... 2024-02-19T18:56:22.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:22 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T18:56:22.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:22 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T18:56:22.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:22 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:22.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:22 smithi114 ceph-mon[29441]: Reconfiguring daemon mon.smithi174 on smithi174 2024-02-19T18:56:22.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:22 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:22.874 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:22 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:23.034 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:22 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:23.034 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:22 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:23.034 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:22 smithi174 ceph-mon[41061]: Reconfiguring mon.smithi174 (monmap changed)... 2024-02-19T18:56:23.034 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:22 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T18:56:23.034 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:22 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T18:56:23.034 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:22 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:23.034 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:22 smithi174 ceph-mon[41061]: Reconfiguring daemon mon.smithi174 on smithi174 2024-02-19T18:56:23.034 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:22 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:23.034 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:22 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:24.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:23 smithi114 ceph-mon[29441]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:24.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:24.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:24.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T18:56:24.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T18:56:24.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:24.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T18:56:24.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T18:56:24.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:24.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T18:56:24.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T18:56:24.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:24.035 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:56:24.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:23 smithi174 ceph-mon[41061]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:24.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:24.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:24.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T18:56:24.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T18:56:24.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:24.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T18:56:24.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T18:56:24.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:24.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T18:56:24.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T18:56:24.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:24.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:56:25.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:24 smithi114 ceph-mon[29441]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T18:56:25.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:24 smithi114 ceph-mon[29441]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T18:56:25.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:24 smithi114 ceph-mon[29441]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T18:56:25.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:24 smithi114 ceph-mon[29441]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T18:56:25.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:24 smithi114 ceph-mon[29441]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T18:56:25.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:24 smithi114 ceph-mon[29441]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T18:56:25.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:24 smithi174 ceph-mon[41061]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T18:56:25.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:24 smithi174 ceph-mon[41061]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T18:56:25.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:24 smithi174 ceph-mon[41061]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T18:56:25.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:24 smithi174 ceph-mon[41061]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T18:56:25.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:24 smithi174 ceph-mon[41061]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T18:56:25.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:24 smithi174 ceph-mon[41061]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T18:56:25.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:25.413+0000 7f61b1ffb700 1 -- 172.21.15.114:0/2387419835 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f61b41080c0 con 0x7f61a006c460 2024-02-19T18:56:25.431 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 0 on host 'smithi114' 2024-02-19T18:56:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:25.415+0000 7f61ba55f700 1 -- 172.21.15.114:0/2387419835 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f61a006c460 msgr2=0x7f61a006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:25.415+0000 7f61ba55f700 1 --2- 172.21.15.114:0/2387419835 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f61a006c460 0x7f61a006e920 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f61b4074af0 tx=0x7f619c008040 comp rx=0 tx=0).stop 2024-02-19T18:56:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:25.415+0000 7f61ba55f700 1 -- 172.21.15.114:0/2387419835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b40737f0 msgr2=0x7f61b4198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:25.416+0000 7f61ba55f700 1 --2- 172.21.15.114:0/2387419835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b40737f0 0x7f61b4198660 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f61a400bda0 tx=0x7f61a400be80 comp rx=0 tx=0).stop 2024-02-19T18:56:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:25.416+0000 7f61ba55f700 1 -- 172.21.15.114:0/2387419835 shutdown_connections 2024-02-19T18:56:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:25.416+0000 7f61ba55f700 1 --2- 172.21.15.114:0/2387419835 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f61a006c460 0x7f61a006e920 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:25.416+0000 7f61ba55f700 1 --2- 172.21.15.114:0/2387419835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b40737f0 0x7f61b4198660 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:25.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:25.416+0000 7f61ba55f700 1 --2- 172.21.15.114:0/2387419835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61b4074dc0 0x7f61b4198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:25.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:25.416+0000 7f61ba55f700 1 -- 172.21.15.114:0/2387419835 >> 172.21.15.114:0/2387419835 conn(0x7f61b40fc140 msgr2=0x7f61b41069a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:25.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:25.416+0000 7f61ba55f700 1 -- 172.21.15.114:0/2387419835 shutdown_connections 2024-02-19T18:56:25.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:25.416+0000 7f61ba55f700 1 -- 172.21.15.114:0/2387419835 wait complete. 2024-02-19T18:56:26.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:25 smithi114 ceph-mon[29441]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:26.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:25 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:26.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:25 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:26.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:25 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:26.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:25 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:26.385 DEBUG:teuthology.orchestra.run.smithi114:osd.0> sudo journalctl -f -n 0 -u ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.0.service 2024-02-19T18:56:26.387 INFO:tasks.cephadm:Deploying osd.1 on smithi114 with /dev/vg_nvme/lv_3... 2024-02-19T18:56:26.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-19T18:56:26.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:25 smithi174 ceph-mon[41061]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:26.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:25 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:26.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:25 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:26.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:25 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:26.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:25 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:27.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:26 smithi114 ceph-mon[29441]: from='osd.0 [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-19T18:56:27.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:26 smithi174 ceph-mon[41061]: from='osd.0 [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-19T18:56:28.115 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:56:28.238 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:27 smithi114 ceph-mon[29441]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:28.238 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:27 smithi114 ceph-mon[29441]: from='osd.0 [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-19T18:56:28.238 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:27 smithi114 ceph-mon[29441]: osdmap e7: 1 total, 0 up, 1 in 2024-02-19T18:56:28.238 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:27 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:56:28.238 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:27 smithi114 ceph-mon[29441]: from='osd.0 [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-02-19T18:56:28.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 18:56:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[48069]: 2024-02-19T18:56:27.919+0000 7fef6d321700 -1 osd.0 0 waiting for initial osdmap 2024-02-19T18:56:28.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 18:56:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[48069]: 2024-02-19T18:56:27.923+0000 7fef66110700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T18:56:28.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:27 smithi174 ceph-mon[41061]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:28.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:27 smithi174 ceph-mon[41061]: from='osd.0 [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-19T18:56:28.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:27 smithi174 ceph-mon[41061]: osdmap e7: 1 total, 0 up, 1 in 2024-02-19T18:56:28.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:27 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:56:28.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:27 smithi174 ceph-mon[41061]: from='osd.0 [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-02-19T18:56:29.176 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:28 smithi114 ceph-mon[29441]: from='osd.0 [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-02-19T18:56:29.177 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:28 smithi114 ceph-mon[29441]: osdmap e8: 1 total, 0 up, 1 in 2024-02-19T18:56:29.177 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:28 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:56:29.177 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:28 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:56:29.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:28 smithi174 ceph-mon[41061]: from='osd.0 [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-02-19T18:56:29.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:28 smithi174 ceph-mon[41061]: osdmap e8: 1 total, 0 up, 1 in 2024-02-19T18:56:29.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:28 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:56:29.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:28 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:56:30.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:29 smithi174 ceph-mon[41061]: purged_snaps scrub starts 2024-02-19T18:56:30.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:29 smithi174 ceph-mon[41061]: purged_snaps scrub ok 2024-02-19T18:56:30.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:29 smithi174 ceph-mon[41061]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:30.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:29 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:56:30.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:29 smithi174 ceph-mon[41061]: osd.0 [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688] boot 2024-02-19T18:56:30.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:29 smithi174 ceph-mon[41061]: osdmap e9: 1 total, 1 up, 1 in 2024-02-19T18:56:30.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:29 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:56:30.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:29 smithi114 ceph-mon[29441]: purged_snaps scrub starts 2024-02-19T18:56:30.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:29 smithi114 ceph-mon[29441]: purged_snaps scrub ok 2024-02-19T18:56:30.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:29 smithi114 ceph-mon[29441]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:56:30.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:29 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:56:30.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:29 smithi114 ceph-mon[29441]: osd.0 [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688] boot 2024-02-19T18:56:30.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:29 smithi114 ceph-mon[29441]: osdmap e9: 1 total, 1 up, 1 in 2024-02-19T18:56:30.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:29 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:56:31.252 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:30 smithi114 ceph-mon[29441]: osdmap e10: 1 total, 1 up, 1 in 2024-02-19T18:56:31.252 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:30 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:31.252 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:30 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:31.379 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-19T18:56:31.380 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-19T18:56:31.380 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-02-19T18:56:31.380 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-02-19T18:56:31.380 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011322 s, 926 MB/s 2024-02-19T18:56:31.380 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-02-19T18:56:31.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:30 smithi174 ceph-mon[41061]: osdmap e10: 1 total, 1 up, 1 in 2024-02-19T18:56:31.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:30 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:31.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:30 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:32.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:31 smithi114 ceph-mon[29441]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:32.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:31 smithi174 ceph-mon[41061]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:32.579 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi114:vg_nvme/lv_3 2024-02-19T18:56:33.126 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:56:33.207 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:32 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:56:33.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:32 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:56:34.121 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:33 smithi114 ceph-mon[29441]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:34.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:33 smithi174 ceph-mon[41061]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.387+0000 7f2e5e323700 1 -- 172.21.15.114:0/1213885335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e58100bc0 msgr2=0x7f2e58100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.387+0000 7f2e5e323700 1 --2- 172.21.15.114:0/1213885335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e58100bc0 0x7f2e58100fe0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009910 tx=0x7f2e48009c20 comp rx=0 tx=0).stop 2024-02-19T18:56:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.387+0000 7f2e5e323700 1 -- 172.21.15.114:0/1213885335 shutdown_connections 2024-02-19T18:56:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.387+0000 7f2e5e323700 1 --2- 172.21.15.114:0/1213885335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e58101dc0 0x7f2e58102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.387+0000 7f2e5e323700 1 --2- 172.21.15.114:0/1213885335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e58100bc0 0x7f2e58100fe0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.387+0000 7f2e5e323700 1 -- 172.21.15.114:0/1213885335 >> 172.21.15.114:0/1213885335 conn(0x7f2e580fc140 msgr2=0x7f2e580fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.387+0000 7f2e5e323700 1 -- 172.21.15.114:0/1213885335 shutdown_connections 2024-02-19T18:56:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.387+0000 7f2e5e323700 1 -- 172.21.15.114:0/1213885335 wait complete. 2024-02-19T18:56:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.387+0000 7f2e5e323700 1 Processor -- start 2024-02-19T18:56:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.387+0000 7f2e5e323700 1 -- start start 2024-02-19T18:56:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.388+0000 7f2e5e323700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e58100bc0 0x7f2e58196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.388+0000 7f2e5e323700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e58101dc0 0x7f2e581968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.388+0000 7f2e5e323700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e58196ed0 con 0x7f2e58101dc0 2024-02-19T18:56:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.388+0000 7f2e5e323700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e58197010 con 0x7f2e58100bc0 2024-02-19T18:56:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.388+0000 7f2e57fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e58100bc0 0x7f2e58196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.388+0000 7f2e577fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e58101dc0 0x7f2e581968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.388+0000 7f2e57fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e58100bc0 0x7f2e58196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.114:47476/0 (socket says 172.21.15.114:47476) 2024-02-19T18:56:35.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.388+0000 7f2e57fff700 1 -- 172.21.15.114:0/272448921 learned_addr learned my addr 172.21.15.114:0/272448921 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:56:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.388+0000 7f2e577fe700 1 -- 172.21.15.114:0/272448921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e58100bc0 msgr2=0x7f2e58196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.388+0000 7f2e577fe700 1 --2- 172.21.15.114:0/272448921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e58100bc0 0x7f2e58196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.388+0000 7f2e577fe700 1 -- 172.21.15.114:0/272448921 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e480095f0 con 0x7f2e58101dc0 2024-02-19T18:56:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.388+0000 7f2e577fe700 1 --2- 172.21.15.114:0/272448921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e58101dc0 0x7f2e581968b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c00c8f0 tx=0x7f2e4c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.388+0000 7f2e557fa700 1 -- 172.21.15.114:0/272448921 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e4c00e850 con 0x7f2e58101dc0 2024-02-19T18:56:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.388+0000 7f2e5e323700 1 -- 172.21.15.114:0/272448921 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e5819bac0 con 0x7f2e58101dc0 2024-02-19T18:56:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.388+0000 7f2e557fa700 1 -- 172.21.15.114:0/272448921 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2e4c00ee60 con 0x7f2e58101dc0 2024-02-19T18:56:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.388+0000 7f2e557fa700 1 -- 172.21.15.114:0/272448921 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e4c018630 con 0x7f2e58101dc0 2024-02-19T18:56:35.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.388+0000 7f2e5e323700 1 -- 172.21.15.114:0/272448921 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e5819c010 con 0x7f2e58101dc0 2024-02-19T18:56:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.389+0000 7f2e5e323700 1 -- 172.21.15.114:0/272448921 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e58109720 con 0x7f2e58101dc0 2024-02-19T18:56:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.390+0000 7f2e557fa700 1 -- 172.21.15.114:0/272448921 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f2e4c00f8a0 con 0x7f2e58101dc0 2024-02-19T18:56:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.390+0000 7f2e557fa700 1 --2- 172.21.15.114:0/272448921 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2e4006c460 0x7f2e4006e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.390+0000 7f2e557fa700 1 -- 172.21.15.114:0/272448921 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7f2e4c08a770 con 0x7f2e58101dc0 2024-02-19T18:56:35.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.390+0000 7f2e57fff700 1 --2- 172.21.15.114:0/272448921 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2e4006c460 0x7f2e4006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:35.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.391+0000 7f2e57fff700 1 --2- 172.21.15.114:0/272448921 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2e4006c460 0x7f2e4006e920 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2e48006010 tx=0x7f2e4800b540 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:35.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.395+0000 7f2e557fa700 1 -- 172.21.15.114:0/272448921 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2e4c059860 con 0x7f2e58101dc0 2024-02-19T18:56:35.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:35.596+0000 7f2e5e323700 1 -- 172.21.15.114:0/272448921 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e58106710 con 0x7f2e4006c460 2024-02-19T18:56:36.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:35 smithi114 ceph-mon[29441]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:36.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:35 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:56:36.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:35 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:56:36.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:35 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:36.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:35 smithi174 ceph-mon[41061]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:36.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:35 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:56:36.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:35 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:56:36.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:35 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:37.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:36 smithi174 ceph-mon[41061]: from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:56:37.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:36 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:37.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:36 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:37.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:36 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:56:37.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:36 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:37.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:36 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:37.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:36 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:56:37.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:36 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:37.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:36 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:56:37.228 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:36 smithi114 ceph-mon[29441]: from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:56:37.228 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:36 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:37.229 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:36 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:37.229 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:36 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:56:37.229 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:36 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:37.229 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:36 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:37.229 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:36 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:56:37.229 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:36 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:37.229 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:36 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:56:38.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:37 smithi114 ceph-mon[29441]: Detected new or changed devices on smithi114 2024-02-19T18:56:38.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:37 smithi114 ceph-mon[29441]: Adjusting osd_memory_target on smithi114 to 11840M 2024-02-19T18:56:38.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:37 smithi114 ceph-mon[29441]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:38.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:37 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:38.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:37 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:38.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:37 smithi174 ceph-mon[41061]: Detected new or changed devices on smithi114 2024-02-19T18:56:38.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:37 smithi174 ceph-mon[41061]: Adjusting osd_memory_target on smithi114 to 11840M 2024-02-19T18:56:38.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:37 smithi174 ceph-mon[41061]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:38.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:38.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:39.015 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:38 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1192509739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8334d634-9c49-4614-88e0-ca4545fa761b"}]: dispatch 2024-02-19T18:56:39.016 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:38 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1192509739' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8334d634-9c49-4614-88e0-ca4545fa761b"}]': finished 2024-02-19T18:56:39.016 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:38 smithi114 ceph-mon[29441]: osdmap e11: 2 total, 1 up, 2 in 2024-02-19T18:56:39.016 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:38 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:56:39.016 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:38 smithi114 ceph-mon[29441]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:39.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:38 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1192509739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8334d634-9c49-4614-88e0-ca4545fa761b"}]: dispatch 2024-02-19T18:56:39.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:38 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1192509739' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8334d634-9c49-4614-88e0-ca4545fa761b"}]': finished 2024-02-19T18:56:39.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:38 smithi174 ceph-mon[41061]: osdmap e11: 2 total, 1 up, 2 in 2024-02-19T18:56:39.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:38 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:56:39.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:38 smithi174 ceph-mon[41061]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:40.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:39 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/2211727555' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:56:40.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:39 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/2211727555' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:56:41.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:40 smithi114 ceph-mon[29441]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:41.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:40 smithi174 ceph-mon[41061]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:42.891 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:42 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:42.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:42 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:42.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:42 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:42.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:42 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:56:42.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:42 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:42 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:42 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:42 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:42 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:56:42.965 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:42 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:44.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:43 smithi114 ceph-mon[29441]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:44.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:43 smithi174 ceph-mon[41061]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:46.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:45 smithi174 ceph-mon[41061]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:46.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:45 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-19T18:56:46.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:45 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:46.271 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:45 smithi114 ceph-mon[29441]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:46.271 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:45 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-19T18:56:46.271 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:45 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:47.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:46 smithi114 ceph-mon[29441]: Deploying daemon osd.1 on smithi114 2024-02-19T18:56:47.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:46 smithi174 ceph-mon[41061]: Deploying daemon osd.1 on smithi114 2024-02-19T18:56:48.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:47 smithi114 ceph-mon[29441]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:48.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:47 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:56:48.293 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:47 smithi174 ceph-mon[41061]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:48.293 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:47 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:56:50.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:49 smithi114 ceph-mon[29441]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:50.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:49 smithi174 ceph-mon[41061]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:52.111 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:51 smithi114 ceph-mon[29441]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:52.111 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:51 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:52.111 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:51 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:52.111 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:51 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:56:52.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:51 smithi174 ceph-mon[41061]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:52.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:51 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:52.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:51 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:52.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:51 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:56:54.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:53 smithi114 ceph-mon[29441]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:54.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:53 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:54.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:53 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:54.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:53 smithi174 ceph-mon[41061]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:54.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:53 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:54.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:53 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:55.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:55.399+0000 7f2e557fa700 1 -- 172.21.15.114:0/272448921 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2e58106710 con 0x7f2e4006c460 2024-02-19T18:56:55.417 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 1 on host 'smithi114' 2024-02-19T18:56:55.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:55.401+0000 7f2e5e323700 1 -- 172.21.15.114:0/272448921 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2e4006c460 msgr2=0x7f2e4006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:55.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:55.401+0000 7f2e5e323700 1 --2- 172.21.15.114:0/272448921 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2e4006c460 0x7f2e4006e920 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2e48006010 tx=0x7f2e4800b540 comp rx=0 tx=0).stop 2024-02-19T18:56:55.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:55.401+0000 7f2e5e323700 1 -- 172.21.15.114:0/272448921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e58101dc0 msgr2=0x7f2e581968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:55.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:55.401+0000 7f2e5e323700 1 --2- 172.21.15.114:0/272448921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e58101dc0 0x7f2e581968b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c00c8f0 tx=0x7f2e4c00cc00 comp rx=0 tx=0).stop 2024-02-19T18:56:55.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:55.401+0000 7f2e5e323700 1 -- 172.21.15.114:0/272448921 shutdown_connections 2024-02-19T18:56:55.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:55.401+0000 7f2e5e323700 1 --2- 172.21.15.114:0/272448921 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2e4006c460 0x7f2e4006e920 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:55.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:55.401+0000 7f2e5e323700 1 --2- 172.21.15.114:0/272448921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e58100bc0 0x7f2e58196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:55.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:55.401+0000 7f2e5e323700 1 --2- 172.21.15.114:0/272448921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e58101dc0 0x7f2e581968b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:55.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:55.401+0000 7f2e5e323700 1 -- 172.21.15.114:0/272448921 >> 172.21.15.114:0/272448921 conn(0x7f2e580fc140 msgr2=0x7f2e58104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:55.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:55.401+0000 7f2e5e323700 1 -- 172.21.15.114:0/272448921 shutdown_connections 2024-02-19T18:56:55.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:56:55.401+0000 7f2e5e323700 1 -- 172.21.15.114:0/272448921 wait complete. 2024-02-19T18:56:56.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:55 smithi114 ceph-mon[29441]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:56.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:55 smithi114 ceph-mon[29441]: from='osd.1 [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-19T18:56:56.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:55 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:56.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:55 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:56.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:55 smithi174 ceph-mon[41061]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:56.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:55 smithi174 ceph-mon[41061]: from='osd.1 [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-19T18:56:56.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:55 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:56.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:55 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:56.710 DEBUG:teuthology.orchestra.run.smithi114:osd.1> sudo journalctl -f -n 0 -u ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.1.service 2024-02-19T18:56:56.713 INFO:tasks.cephadm:Deploying osd.2 on smithi114 with /dev/vg_nvme/lv_2... 2024-02-19T18:56:56.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-19T18:56:57.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:56 smithi114 ceph-mon[29441]: from='osd.1 [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-19T18:56:57.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:56 smithi114 ceph-mon[29441]: osdmap e12: 2 total, 1 up, 2 in 2024-02-19T18:56:57.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:56 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:56:57.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:56 smithi114 ceph-mon[29441]: from='osd.1 [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-02-19T18:56:57.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 18:56:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[55538]: 2024-02-19T18:56:56.938+0000 7fcca2075700 -1 osd.1 0 waiting for initial osdmap 2024-02-19T18:56:57.287 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 18:56:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[55538]: 2024-02-19T18:56:56.944+0000 7fcc9de6a700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T18:56:57.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:56 smithi174 ceph-mon[41061]: from='osd.1 [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-19T18:56:57.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:56 smithi174 ceph-mon[41061]: osdmap e12: 2 total, 1 up, 2 in 2024-02-19T18:56:57.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:56 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:56:57.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:56 smithi174 ceph-mon[41061]: from='osd.1 [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-02-19T18:56:57.523 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:56:58.281 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:57 smithi114 ceph-mon[29441]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:58.281 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:57 smithi114 ceph-mon[29441]: from='osd.1 [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-02-19T18:56:58.281 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:57 smithi114 ceph-mon[29441]: osdmap e13: 2 total, 1 up, 2 in 2024-02-19T18:56:58.281 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:57 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:56:58.281 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:57 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:56:58.281 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:57 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:56:58.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:57 smithi174 ceph-mon[41061]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:58.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:57 smithi174 ceph-mon[41061]: from='osd.1 [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-02-19T18:56:58.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:57 smithi174 ceph-mon[41061]: osdmap e13: 2 total, 1 up, 2 in 2024-02-19T18:56:58.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:57 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:56:58.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:57 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:56:58.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:57 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:56:59.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:58 smithi114 ceph-mon[29441]: purged_snaps scrub starts 2024-02-19T18:56:59.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:58 smithi114 ceph-mon[29441]: purged_snaps scrub ok 2024-02-19T18:56:59.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:58 smithi114 ceph-mon[29441]: osd.1 [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125] boot 2024-02-19T18:56:59.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:58 smithi114 ceph-mon[29441]: osdmap e14: 2 total, 2 up, 2 in 2024-02-19T18:56:59.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:58 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:56:59.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:58 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:59.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:58 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:59.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:58 smithi114 ceph-mon[29441]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:56:59.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:58 smithi174 ceph-mon[41061]: purged_snaps scrub starts 2024-02-19T18:56:59.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:58 smithi174 ceph-mon[41061]: purged_snaps scrub ok 2024-02-19T18:56:59.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:58 smithi174 ceph-mon[41061]: osd.1 [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125] boot 2024-02-19T18:56:59.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:58 smithi174 ceph-mon[41061]: osdmap e14: 2 total, 2 up, 2 in 2024-02-19T18:56:59.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:58 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:56:59.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:58 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:59.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:58 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:56:59.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:58 smithi174 ceph-mon[41061]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:57:00.129 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:56:59 smithi114 ceph-mon[29441]: osdmap e15: 2 total, 2 up, 2 in 2024-02-19T18:57:00.317 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-19T18:57:00.317 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-19T18:57:00.317 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-02-19T18:57:00.317 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-02-19T18:57:00.317 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113398 s, 925 MB/s 2024-02-19T18:57:00.317 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-02-19T18:57:00.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:56:59 smithi174 ceph-mon[41061]: osdmap e15: 2 total, 2 up, 2 in 2024-02-19T18:57:01.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:00 smithi114 ceph-mon[29441]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:01.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:00 smithi174 ceph-mon[41061]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:01.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi114:vg_nvme/lv_2 2024-02-19T18:57:02.323 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:57:03.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:02 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:03.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:02 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:04.243 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:03 smithi114 ceph-mon[29441]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:04.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:03 smithi174 ceph-mon[41061]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:04.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.662+0000 7fb452d17700 1 -- 172.21.15.114:0/4263882578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb44c100c00 msgr2=0x7fb44c101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:04.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.662+0000 7fb452d17700 1 --2- 172.21.15.114:0/4263882578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb44c100c00 0x7fb44c101020 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb44400b0a0 tx=0x7fb44400b3b0 comp rx=0 tx=0).stop 2024-02-19T18:57:04.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.662+0000 7fb452d17700 1 -- 172.21.15.114:0/4263882578 shutdown_connections 2024-02-19T18:57:04.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.662+0000 7fb452d17700 1 --2- 172.21.15.114:0/4263882578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb44c101e00 0x7fb44c102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:04.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.662+0000 7fb452d17700 1 --2- 172.21.15.114:0/4263882578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb44c100c00 0x7fb44c101020 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:04.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.662+0000 7fb452d17700 1 -- 172.21.15.114:0/4263882578 >> 172.21.15.114:0/4263882578 conn(0x7fb44c0fc180 msgr2=0x7fb44c0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:04.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.662+0000 7fb452d17700 1 -- 172.21.15.114:0/4263882578 shutdown_connections 2024-02-19T18:57:04.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.662+0000 7fb452d17700 1 -- 172.21.15.114:0/4263882578 wait complete. 2024-02-19T18:57:04.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.663+0000 7fb452d17700 1 Processor -- start 2024-02-19T18:57:04.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.663+0000 7fb452d17700 1 -- start start 2024-02-19T18:57:04.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.663+0000 7fb452d17700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb44c100c00 0x7fb44c196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:04.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.663+0000 7fb452d17700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb44c101e00 0x7fb44c1968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:04.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.663+0000 7fb452d17700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c196ef0 con 0x7fb44c101e00 2024-02-19T18:57:04.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.663+0000 7fb452d17700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c197030 con 0x7fb44c100c00 2024-02-19T18:57:04.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.663+0000 7fb44bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb44c101e00 0x7fb44c1968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:04.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.663+0000 7fb44bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb44c101e00 0x7fb44c1968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59746/0 (socket says 172.21.15.114:59746) 2024-02-19T18:57:04.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.663+0000 7fb44bfff700 1 -- 172.21.15.114:0/1851649836 learned_addr learned my addr 172.21.15.114:0/1851649836 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:57:04.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.663+0000 7fb450ab3700 1 --2- 172.21.15.114:0/1851649836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb44c100c00 0x7fb44c196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:04.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.663+0000 7fb44bfff700 1 -- 172.21.15.114:0/1851649836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb44c100c00 msgr2=0x7fb44c196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:04.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.663+0000 7fb44bfff700 1 --2- 172.21.15.114:0/1851649836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb44c100c00 0x7fb44c196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:04.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.663+0000 7fb44bfff700 1 -- 172.21.15.114:0/1851649836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb444009d00 con 0x7fb44c101e00 2024-02-19T18:57:04.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.664+0000 7fb44bfff700 1 --2- 172.21.15.114:0/1851649836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb44c101e00 0x7fb44c1968d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb44000d8d0 tx=0x7fb44000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:04.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.664+0000 7fb449ffb700 1 -- 172.21.15.114:0/1851649836 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4400098a0 con 0x7fb44c101e00 2024-02-19T18:57:04.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.664+0000 7fb449ffb700 1 -- 172.21.15.114:0/1851649836 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb440010460 con 0x7fb44c101e00 2024-02-19T18:57:04.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.664+0000 7fb449ffb700 1 -- 172.21.15.114:0/1851649836 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb44000f5f0 con 0x7fb44c101e00 2024-02-19T18:57:04.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.664+0000 7fb452d17700 1 -- 172.21.15.114:0/1851649836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb44c19bae0 con 0x7fb44c101e00 2024-02-19T18:57:04.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.664+0000 7fb452d17700 1 -- 172.21.15.114:0/1851649836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb44c19c0b0 con 0x7fb44c101e00 2024-02-19T18:57:04.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.665+0000 7fb452d17700 1 -- 172.21.15.114:0/1851649836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb44c109760 con 0x7fb44c101e00 2024-02-19T18:57:04.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.665+0000 7fb449ffb700 1 -- 172.21.15.114:0/1851649836 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7fb440009a00 con 0x7fb44c101e00 2024-02-19T18:57:04.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.666+0000 7fb449ffb700 1 --2- 172.21.15.114:0/1851649836 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb43406c460 0x7fb43406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:04.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.666+0000 7fb449ffb700 1 -- 172.21.15.114:0/1851649836 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7fb44008a9d0 con 0x7fb44c101e00 2024-02-19T18:57:04.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.666+0000 7fb450ab3700 1 --2- 172.21.15.114:0/1851649836 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb43406c460 0x7fb43406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:04.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.666+0000 7fb450ab3700 1 --2- 172.21.15.114:0/1851649836 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb43406c460 0x7fb43406e920 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb44400b830 tx=0x7fb4440060c0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:04.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.671+0000 7fb449ffb700 1 -- 172.21.15.114:0/1851649836 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb440059910 con 0x7fb44c101e00 2024-02-19T18:57:04.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:04.893+0000 7fb452d17700 1 -- 172.21.15.114:0/1851649836 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb44c106750 con 0x7fb43406c460 2024-02-19T18:57:05.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:05 smithi114 ceph-mon[29441]: Detected new or changed devices on smithi114 2024-02-19T18:57:05.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:05 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:05.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:05 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:05.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:05 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:05.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:05 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:05.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:05 smithi114 ceph-mon[29441]: Adjusting osd_memory_target on smithi114 to 5920M 2024-02-19T18:57:05.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:05 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:05.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:05 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:05.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:05 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:05.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:05 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:05.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:05 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:57:05.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:05 smithi114 ceph-mon[29441]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:05.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:05 smithi114 ceph-mon[29441]: from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:57:05.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:05 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:57:05.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:05 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:57:05.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:05 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:05.391 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:05 smithi174 ceph-mon[41061]: Detected new or changed devices on smithi114 2024-02-19T18:57:05.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:05 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:05.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:05 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:05.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:05 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:05.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:05 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:05.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:05 smithi174 ceph-mon[41061]: Adjusting osd_memory_target on smithi114 to 5920M 2024-02-19T18:57:05.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:05 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:05.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:05 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:05.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:05 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:05.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:05 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:05.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:05 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:57:05.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:05 smithi174 ceph-mon[41061]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:05.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:05 smithi174 ceph-mon[41061]: from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:57:05.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:05 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:57:05.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:05 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:57:05.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:05 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:07.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:07 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:07.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:07 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:07.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:07 smithi174 ceph-mon[41061]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:07.515 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:07 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:07.515 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:07 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:07.515 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:07 smithi114 ceph-mon[29441]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:09.111 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:08 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1457039193' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d27485a-1fb1-432d-a5a4-f46c8031a6ff"}]: dispatch 2024-02-19T18:57:09.111 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:08 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1457039193' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2d27485a-1fb1-432d-a5a4-f46c8031a6ff"}]': finished 2024-02-19T18:57:09.112 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:08 smithi114 ceph-mon[29441]: osdmap e16: 3 total, 2 up, 3 in 2024-02-19T18:57:09.112 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:08 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:57:09.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:08 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1457039193' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d27485a-1fb1-432d-a5a4-f46c8031a6ff"}]: dispatch 2024-02-19T18:57:09.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:08 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1457039193' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2d27485a-1fb1-432d-a5a4-f46c8031a6ff"}]': finished 2024-02-19T18:57:09.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:08 smithi174 ceph-mon[41061]: osdmap e16: 3 total, 2 up, 3 in 2024-02-19T18:57:09.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:08 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:57:10.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:09 smithi114 ceph-mon[29441]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:10.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:09 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/195449605' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:57:10.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:09 smithi174 ceph-mon[41061]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:10.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:09 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/195449605' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:57:11.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:11 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:11.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:11 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:11.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:11 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:11.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:11 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:11.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:11 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:11.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:11 smithi114 ceph-mon[29441]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:11.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:11 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:11.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:11 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:11.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:11 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:11.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:11 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:11.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:11 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:11.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:11 smithi174 ceph-mon[41061]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:14.146 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:13 smithi114 ceph-mon[29441]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:14.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:13 smithi174 ceph-mon[41061]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:16.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:15 smithi174 ceph-mon[41061]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:16.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:15 smithi114 ceph-mon[29441]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:17.204 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-19T18:57:17.204 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:17.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:16 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-19T18:57:17.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:16 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:18.220 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:17 smithi114 ceph-mon[29441]: Deploying daemon osd.2 on smithi114 2024-02-19T18:57:18.220 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:17 smithi114 ceph-mon[29441]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:18.220 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:17 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:18.293 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:17 smithi174 ceph-mon[41061]: Deploying daemon osd.2 on smithi114 2024-02-19T18:57:18.293 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:17 smithi174 ceph-mon[41061]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:18.293 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:17 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:20.061 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:19 smithi114 ceph-mon[29441]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:20.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:19 smithi174 ceph-mon[41061]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:22.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:21 smithi114 ceph-mon[29441]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:22.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:21 smithi174 ceph-mon[41061]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:23.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:22 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:23.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:22 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:23.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:22 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:57:23.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:22 smithi114 ceph-mon[29441]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:23.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:22 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:23.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:22 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:23.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:22 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:57:23.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:22 smithi174 ceph-mon[41061]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:25.995 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:25 smithi114 ceph-mon[29441]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:26.141 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:25 smithi174 ceph-mon[41061]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:26.167 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 2 on host 'smithi114' 2024-02-19T18:57:26.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:26.149+0000 7fb449ffb700 1 -- 172.21.15.114:0/1851649836 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb44c106750 con 0x7fb43406c460 2024-02-19T18:57:26.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:26.152+0000 7fb452d17700 1 -- 172.21.15.114:0/1851649836 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb43406c460 msgr2=0x7fb43406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:26.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:26.152+0000 7fb452d17700 1 --2- 172.21.15.114:0/1851649836 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb43406c460 0x7fb43406e920 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb44400b830 tx=0x7fb4440060c0 comp rx=0 tx=0).stop 2024-02-19T18:57:26.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:26.152+0000 7fb452d17700 1 -- 172.21.15.114:0/1851649836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb44c101e00 msgr2=0x7fb44c1968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:26.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:26.152+0000 7fb452d17700 1 --2- 172.21.15.114:0/1851649836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb44c101e00 0x7fb44c1968d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb44000d8d0 tx=0x7fb44000dbe0 comp rx=0 tx=0).stop 2024-02-19T18:57:26.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:26.152+0000 7fb452d17700 1 -- 172.21.15.114:0/1851649836 shutdown_connections 2024-02-19T18:57:26.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:26.152+0000 7fb452d17700 1 --2- 172.21.15.114:0/1851649836 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb43406c460 0x7fb43406e920 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:26.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:26.152+0000 7fb452d17700 1 --2- 172.21.15.114:0/1851649836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb44c100c00 0x7fb44c196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:26.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:26.152+0000 7fb452d17700 1 --2- 172.21.15.114:0/1851649836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb44c101e00 0x7fb44c1968d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:26.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:26.152+0000 7fb452d17700 1 -- 172.21.15.114:0/1851649836 >> 172.21.15.114:0/1851649836 conn(0x7fb44c0fc180 msgr2=0x7fb44c105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:26.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:26.152+0000 7fb452d17700 1 -- 172.21.15.114:0/1851649836 shutdown_connections 2024-02-19T18:57:26.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:57:26.152+0000 7fb452d17700 1 -- 172.21.15.114:0/1851649836 wait complete. 2024-02-19T18:57:26.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:26 smithi114 ceph-mon[29441]: from='osd.2 [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-19T18:57:26.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:26 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:26.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:26 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:26.809 DEBUG:teuthology.orchestra.run.smithi114:osd.2> sudo journalctl -f -n 0 -u ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.2.service 2024-02-19T18:57:26.811 INFO:tasks.cephadm:Deploying osd.3 on smithi174 with /dev/vg_nvme/lv_4... 2024-02-19T18:57:26.811 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-19T18:57:27.027 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi174/config 2024-02-19T18:57:27.130 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:26 smithi174 ceph-mon[41061]: from='osd.2 [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-19T18:57:27.130 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:26 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:27.130 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:26 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:27.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:27 smithi114 ceph-mon[29441]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:27.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:27 smithi114 ceph-mon[29441]: from='osd.2 [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-19T18:57:27.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:27 smithi114 ceph-mon[29441]: osdmap e17: 3 total, 2 up, 3 in 2024-02-19T18:57:27.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:27 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:57:27.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:27 smithi114 ceph-mon[29441]: from='osd.2 [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-02-19T18:57:27.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 18:57:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[63366]: 2024-02-19T18:57:27.697+0000 7f13e5366700 -1 osd.2 0 waiting for initial osdmap 2024-02-19T18:57:27.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 18:57:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[63366]: 2024-02-19T18:57:27.701+0000 7f13e0159700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T18:57:28.069 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:27 smithi174 ceph-mon[41061]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:28.069 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:27 smithi174 ceph-mon[41061]: from='osd.2 [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-19T18:57:28.069 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:27 smithi174 ceph-mon[41061]: osdmap e17: 3 total, 2 up, 3 in 2024-02-19T18:57:28.069 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:27 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:57:28.069 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:27 smithi174 ceph-mon[41061]: from='osd.2 [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-02-19T18:57:28.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:28 smithi114 ceph-mon[29441]: from='osd.2 [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-02-19T18:57:28.787 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:28 smithi114 ceph-mon[29441]: osdmap e18: 3 total, 2 up, 3 in 2024-02-19T18:57:28.787 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:28 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:57:28.787 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:28 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:57:28.830 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-19T18:57:28.831 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-19T18:57:28.831 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-02-19T18:57:28.831 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-02-19T18:57:28.831 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113208 s, 926 MB/s 2024-02-19T18:57:28.831 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-02-19T18:57:28.979 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:28 smithi174 ceph-mon[41061]: from='osd.2 [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-02-19T18:57:28.979 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:28 smithi174 ceph-mon[41061]: osdmap e18: 3 total, 2 up, 3 in 2024-02-19T18:57:28.979 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:28 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:57:28.979 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:28 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:57:29.463 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi174:vg_nvme/lv_4 2024-02-19T18:57:29.679 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi174/config 2024-02-19T18:57:29.893 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:29 smithi114 ceph-mon[29441]: purged_snaps scrub starts 2024-02-19T18:57:29.893 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:29 smithi114 ceph-mon[29441]: purged_snaps scrub ok 2024-02-19T18:57:29.893 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:29 smithi114 ceph-mon[29441]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:29.893 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:29 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:57:29.893 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:29 smithi114 ceph-mon[29441]: osd.2 [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196] boot 2024-02-19T18:57:29.893 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:29 smithi114 ceph-mon[29441]: osdmap e19: 3 total, 3 up, 3 in 2024-02-19T18:57:29.893 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:29 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:57:29.894 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:29 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:29.894 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:29 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:30.028 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:29 smithi174 ceph-mon[41061]: purged_snaps scrub starts 2024-02-19T18:57:30.028 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:29 smithi174 ceph-mon[41061]: purged_snaps scrub ok 2024-02-19T18:57:30.028 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:29 smithi174 ceph-mon[41061]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-19T18:57:30.028 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:29 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:57:30.028 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:29 smithi174 ceph-mon[41061]: osd.2 [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196] boot 2024-02-19T18:57:30.028 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:29 smithi174 ceph-mon[41061]: osdmap e19: 3 total, 3 up, 3 in 2024-02-19T18:57:30.028 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:29 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:57:30.029 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:29 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:30.029 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:29 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:31.104 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.084+0000 7f2bfd4ec700 1 -- 172.21.15.174:0/3051122545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf8100cc0 msgr2=0x7f2bf8101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:31.104 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.084+0000 7f2bfd4ec700 1 --2- 172.21.15.174:0/3051122545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf8100cc0 0x7f2bf8101120 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2bec009910 tx=0x7f2bec009c20 comp rx=0 tx=0).stop 2024-02-19T18:57:31.104 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.084+0000 7f2bfd4ec700 1 -- 172.21.15.174:0/3051122545 shutdown_connections 2024-02-19T18:57:31.104 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.084+0000 7f2bfd4ec700 1 --2- 172.21.15.174:0/3051122545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf8100cc0 0x7f2bf8101120 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:31.104 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.084+0000 7f2bfd4ec700 1 --2- 172.21.15.174:0/3051122545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bf80ffac0 0x7f2bf80ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:31.104 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.084+0000 7f2bfd4ec700 1 -- 172.21.15.174:0/3051122545 >> 172.21.15.174:0/3051122545 conn(0x7f2bf80fb060 msgr2=0x7f2bf80fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:31.104 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.084+0000 7f2bfd4ec700 1 -- 172.21.15.174:0/3051122545 shutdown_connections 2024-02-19T18:57:31.105 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.084+0000 7f2bfd4ec700 1 -- 172.21.15.174:0/3051122545 wait complete. 2024-02-19T18:57:31.105 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.085+0000 7f2bfd4ec700 1 Processor -- start 2024-02-19T18:57:31.105 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.085+0000 7f2bfd4ec700 1 -- start start 2024-02-19T18:57:31.105 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.085+0000 7f2bfd4ec700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf80ffac0 0x7f2bf8069910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:31.105 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.085+0000 7f2bfd4ec700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bf8100cc0 0x7f2bf819b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:31.105 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.085+0000 7f2bfd4ec700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf819b670 con 0x7f2bf8100cc0 2024-02-19T18:57:31.105 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.085+0000 7f2bfd4ec700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf819b7b0 con 0x7f2bf80ffac0 2024-02-19T18:57:31.105 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.085+0000 7f2bf6ffd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf80ffac0 0x7f2bf8069910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:31.105 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.085+0000 7f2bf67fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bf8100cc0 0x7f2bf819b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:31.105 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.085+0000 7f2bf6ffd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf80ffac0 0x7f2bf8069910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:55774/0 (socket says 172.21.15.174:55774) 2024-02-19T18:57:31.105 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.085+0000 7f2bf67fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bf8100cc0 0x7f2bf819b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.174:54346/0 (socket says 172.21.15.174:54346) 2024-02-19T18:57:31.105 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.085+0000 7f2bf6ffd700 1 -- 172.21.15.174:0/820376360 learned_addr learned my addr 172.21.15.174:0/820376360 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:57:31.106 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.086+0000 7f2bf6ffd700 1 -- 172.21.15.174:0/820376360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bf8100cc0 msgr2=0x7f2bf819b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:31.106 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.086+0000 7f2bf6ffd700 1 --2- 172.21.15.174:0/820376360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bf8100cc0 0x7f2bf819b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:31.106 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.086+0000 7f2bf6ffd700 1 -- 172.21.15.174:0/820376360 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bec0095f0 con 0x7f2bf80ffac0 2024-02-19T18:57:31.106 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.086+0000 7f2bf6ffd700 1 --2- 172.21.15.174:0/820376360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf80ffac0 0x7f2bf8069910 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2be800d930 tx=0x7f2be800dc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:31.106 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.086+0000 7f2bdffff700 1 -- 172.21.15.174:0/820376360 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2be800b920 con 0x7f2bf80ffac0 2024-02-19T18:57:31.106 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.086+0000 7f2bfd4ec700 1 -- 172.21.15.174:0/820376360 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf819ba90 con 0x7f2bf80ffac0 2024-02-19T18:57:31.106 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.086+0000 7f2bdffff700 1 -- 172.21.15.174:0/820376360 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2be8010460 con 0x7f2bf80ffac0 2024-02-19T18:57:31.106 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.086+0000 7f2bdffff700 1 -- 172.21.15.174:0/820376360 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2be800f590 con 0x7f2bf80ffac0 2024-02-19T18:57:31.106 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.086+0000 7f2bfd4ec700 1 -- 172.21.15.174:0/820376360 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf819bfb0 con 0x7f2bf80ffac0 2024-02-19T18:57:31.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.087+0000 7f2bfd4ec700 1 -- 172.21.15.174:0/820376360 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf8108620 con 0x7f2bf80ffac0 2024-02-19T18:57:31.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.087+0000 7f2bdffff700 1 -- 172.21.15.174:0/820376360 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f2be800f850 con 0x7f2bf80ffac0 2024-02-19T18:57:31.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.088+0000 7f2bdffff700 1 --2- 172.21.15.174:0/820376360 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2be006c460 0x7f2be006e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:31.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.088+0000 7f2bdffff700 1 -- 172.21.15.174:0/820376360 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3221+0+0 (secure 0 0 0) 0x7f2be808aae0 con 0x7f2bf80ffac0 2024-02-19T18:57:31.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.088+0000 7f2bf67fc700 1 --2- 172.21.15.174:0/820376360 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2be006c460 0x7f2be006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:31.107 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.088+0000 7f2bf67fc700 1 --2- 172.21.15.174:0/820376360 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2be006c460 0x7f2be006e920 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2bec00b5c0 tx=0x7f2bec005ee0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:31.110 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.092+0000 7f2bdffff700 1 -- 172.21.15.174:0/820376360 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2be80596f0 con 0x7f2bf80ffac0 2024-02-19T18:57:31.228 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:30 smithi114 ceph-mon[29441]: osdmap e20: 3 total, 3 up, 3 in 2024-02-19T18:57:31.305 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:31.288+0000 7f2bfd4ec700 1 -- 172.21.15.174:0/820376360 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2bf8105610 con 0x7f2be006c460 2024-02-19T18:57:31.306 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:30 smithi174 ceph-mon[41061]: osdmap e20: 3 total, 3 up, 3 in 2024-02-19T18:57:32.079 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:31 smithi174 ceph-mon[41061]: pgmap v82: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:57:32.079 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:31 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-19T18:57:32.079 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:31 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-19T18:57:32.079 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:31 smithi174 ceph-mon[41061]: osdmap e21: 3 total, 3 up, 3 in 2024-02-19T18:57:32.079 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:31 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-19T18:57:32.079 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:31 smithi174 ceph-mon[41061]: from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:57:32.079 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:31 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:57:32.079 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:31 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:57:32.079 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:31 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:32.212 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:31 smithi114 ceph-mon[29441]: pgmap v82: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:57:32.212 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:31 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-19T18:57:32.212 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:31 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-19T18:57:32.213 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:31 smithi114 ceph-mon[29441]: osdmap e21: 3 total, 3 up, 3 in 2024-02-19T18:57:32.213 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:31 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-19T18:57:32.213 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:31 smithi114 ceph-mon[29441]: from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:57:32.213 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:31 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:57:32.213 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:31 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:57:32.213 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:31 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:33.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 18:57:32 smithi114 sudo[68053]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-19T18:57:33.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 18:57:32 smithi114 sudo[68053]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:57:33.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 18:57:32 smithi114 sudo[68053]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:57:33.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 18:57:32 smithi114 sudo[68053]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:57:33.286 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 18:57:33 smithi114 sudo[68104]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-19T18:57:33.286 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 18:57:33 smithi114 sudo[68104]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:57:33.286 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 18:57:33 smithi114 sudo[68104]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:57:33.286 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 18:57:33 smithi114 sudo[68104]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:57:33.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-19T18:57:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:33 smithi114 ceph-mon[29441]: osdmap e22: 3 total, 3 up, 3 in 2024-02-19T18:57:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:33 smithi114 ceph-mon[29441]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:57:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:33 smithi114 ceph-mon[29441]: Detected new or changed devices on smithi114 2024-02-19T18:57:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:33 smithi114 ceph-mon[29441]: Adjusting osd_memory_target on smithi114 to 3946M 2024-02-19T18:57:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:57:33.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:33 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-19T18:57:33.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:33 smithi174 ceph-mon[41061]: osdmap e22: 3 total, 3 up, 3 in 2024-02-19T18:57:33.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:33 smithi174 ceph-mon[41061]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:57:33.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:33 smithi174 ceph-mon[41061]: Detected new or changed devices on smithi114 2024-02-19T18:57:33.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:33 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:33.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:33 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:33.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:33 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:33.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:33 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:33.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:33 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:33.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:33 smithi174 ceph-mon[41061]: Adjusting osd_memory_target on smithi114 to 3946M 2024-02-19T18:57:33.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:33 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:33.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:33 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:33.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:33 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:33.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:33 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:33.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:33 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:33.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:33 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:57:33.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 18:57:33 smithi114 sudo[68178]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-19T18:57:33.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 18:57:33 smithi114 sudo[68178]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:57:33.629 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 18:57:33 smithi114 sudo[68178]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:57:33.629 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 18:57:33 smithi114 sudo[68178]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:57:33.890 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 18:57:33 smithi114 sudo[68202]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-19T18:57:33.890 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 18:57:33 smithi114 sudo[68202]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:57:33.890 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 18:57:33 smithi114 sudo[68202]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:57:33.890 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 18:57:33 smithi114 sudo[68202]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:57:33.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 18:57:33 smithi114 sudo[68199]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-19T18:57:33.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 18:57:33 smithi114 sudo[68199]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:57:33.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 18:57:33 smithi114 sudo[68199]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:57:33.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 18:57:33 smithi114 sudo[68199]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:57:34.118 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:34.100+0000 7f2bdffff700 1 -- 172.21.15.174:0/820376360 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2be8056030 con 0x7f2bf80ffac0 2024-02-19T18:57:34.149 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:34 smithi114 ceph-mon[29441]: osdmap e23: 3 total, 3 up, 3 in 2024-02-19T18:57:34.149 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:34 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:34.149 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:34 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/2971137065' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdbbfdf3-8e5c-472c-9894-97741a049a31"}]: dispatch 2024-02-19T18:57:34.149 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:34 smithi114 ceph-mon[29441]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdbbfdf3-8e5c-472c-9894-97741a049a31"}]: dispatch 2024-02-19T18:57:34.149 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:34 smithi114 ceph-mon[29441]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fdbbfdf3-8e5c-472c-9894-97741a049a31"}]': finished 2024-02-19T18:57:34.149 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:34 smithi114 ceph-mon[29441]: osdmap e24: 4 total, 3 up, 4 in 2024-02-19T18:57:34.150 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:34 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:57:34.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:34 smithi174 ceph-mon[41061]: osdmap e23: 3 total, 3 up, 3 in 2024-02-19T18:57:34.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:34 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:34.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:34 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/2971137065' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdbbfdf3-8e5c-472c-9894-97741a049a31"}]: dispatch 2024-02-19T18:57:34.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:34 smithi174 ceph-mon[41061]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdbbfdf3-8e5c-472c-9894-97741a049a31"}]: dispatch 2024-02-19T18:57:34.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:34 smithi174 ceph-mon[41061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fdbbfdf3-8e5c-472c-9894-97741a049a31"}]': finished 2024-02-19T18:57:34.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:34 smithi174 ceph-mon[41061]: osdmap e24: 4 total, 3 up, 4 in 2024-02-19T18:57:34.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:34 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:57:34.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 18:57:34 smithi114 sudo[68305]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-19T18:57:34.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 18:57:34 smithi114 sudo[68305]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:57:34.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 18:57:34 smithi114 sudo[68305]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:57:34.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 18:57:34 smithi114 sudo[68305]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:57:34.659 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:34 smithi114 sudo[68327]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-19T18:57:34.659 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:34 smithi114 sudo[68327]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:57:34.659 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:34 smithi114 sudo[68327]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:57:34.659 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:34 smithi114 sudo[68327]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:57:34.957 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:34 smithi114 sudo[68330]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-19T18:57:34.957 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:34 smithi114 sudo[68330]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:57:34.958 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:34 smithi114 sudo[68330]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:57:34.958 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:34 smithi114 sudo[68330]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:57:35.162 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:34 smithi174 sudo[45096]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-19T18:57:35.163 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:34 smithi174 sudo[45096]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:57:35.163 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:34 smithi174 sudo[45096]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:57:35.163 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:34 smithi174 sudo[45096]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:57:35.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:35 smithi114 ceph-mon[29441]: mgrmap e19: smithi114.pcvtrk(active, since 2m), standbys: smithi174.lydvnj 2024-02-19T18:57:35.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:35 smithi114 ceph-mon[29441]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-19T18:57:35.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:35 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/97937051' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:57:35.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:35 smithi114 ceph-mon[29441]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:57:35.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:35 smithi114 ceph-mon[29441]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-19T18:57:35.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:35 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T18:57:35.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:35 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:57:35.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:35 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T18:57:35.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:35 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:57:35.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:35 smithi174 sudo[45100]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-19T18:57:35.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:35 smithi174 sudo[45100]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:57:35.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:35 smithi174 sudo[45100]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:57:35.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:35 smithi174 sudo[45100]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:57:35.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:35 smithi174 ceph-mon[41061]: mgrmap e19: smithi114.pcvtrk(active, since 2m), standbys: smithi174.lydvnj 2024-02-19T18:57:35.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:35 smithi174 ceph-mon[41061]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-19T18:57:35.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:35 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/97937051' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:57:35.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:35 smithi174 ceph-mon[41061]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:57:35.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:35 smithi174 ceph-mon[41061]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-19T18:57:35.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:35 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T18:57:35.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:35 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:57:35.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:35 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T18:57:35.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:35 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T18:57:36.380 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:36 smithi114 ceph-mon[29441]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-19T18:57:36.380 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:36 smithi114 ceph-mon[29441]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-19T18:57:36.381 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:36 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:36.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:36 smithi174 ceph-mon[41061]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-19T18:57:36.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:36 smithi174 ceph-mon[41061]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-19T18:57:36.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:36 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:38.223 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:37 smithi174 ceph-mon[41061]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:57:38.223 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:38.223 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:38.223 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:38.223 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:38.223 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:37 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:38.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:37 smithi114 ceph-mon[29441]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:57:38.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:37 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:38.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:37 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:38.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:37 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:38.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:37 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:38.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:37 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:40.049 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:39 smithi174 ceph-mon[41061]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:57:40.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:39 smithi114 ceph-mon[29441]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:57:41.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:40 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-19T18:57:41.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:40 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:41.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:40 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-19T18:57:41.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:40 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:42.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:41 smithi114 ceph-mon[29441]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-19T18:57:42.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:41 smithi114 ceph-mon[29441]: Deploying daemon osd.3 on smithi174 2024-02-19T18:57:42.308 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:41 smithi174 ceph-mon[41061]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-19T18:57:42.308 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:41 smithi174 ceph-mon[41061]: Deploying daemon osd.3 on smithi174 2024-02-19T18:57:44.253 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:43 smithi174 ceph-mon[41061]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-19T18:57:44.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:43 smithi114 ceph-mon[29441]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-19T18:57:46.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:45 smithi174 ceph-mon[41061]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-19T18:57:46.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:45 smithi114 ceph-mon[29441]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-19T18:57:47.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:47 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:57:47.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:47 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:47.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:47 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:47.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:47 smithi114 ceph-mon[29441]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-19T18:57:47.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:47 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:57:47.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:47 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:47.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:47 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:47.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:47 smithi174 ceph-mon[41061]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-19T18:57:48.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:48 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:48.317 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:48 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:49.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:49 smithi174 ceph-mon[41061]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-19T18:57:49.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:49 smithi114 ceph-mon[29441]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-19T18:57:49.870 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 3 on host 'smithi174' 2024-02-19T18:57:49.870 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:49.850+0000 7f2bdffff700 1 -- 172.21.15.174:0/820376360 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2bf8105610 con 0x7f2be006c460 2024-02-19T18:57:49.871 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:49.853+0000 7f2bfd4ec700 1 -- 172.21.15.174:0/820376360 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2be006c460 msgr2=0x7f2be006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:49.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:49.853+0000 7f2bfd4ec700 1 --2- 172.21.15.174:0/820376360 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2be006c460 0x7f2be006e920 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2bec00b5c0 tx=0x7f2bec005ee0 comp rx=0 tx=0).stop 2024-02-19T18:57:49.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:49.853+0000 7f2bfd4ec700 1 -- 172.21.15.174:0/820376360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf80ffac0 msgr2=0x7f2bf8069910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:49.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:49.853+0000 7f2bfd4ec700 1 --2- 172.21.15.174:0/820376360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf80ffac0 0x7f2bf8069910 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2be800d930 tx=0x7f2be800dc40 comp rx=0 tx=0).stop 2024-02-19T18:57:49.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:49.853+0000 7f2bfd4ec700 1 -- 172.21.15.174:0/820376360 shutdown_connections 2024-02-19T18:57:49.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:49.853+0000 7f2bfd4ec700 1 --2- 172.21.15.174:0/820376360 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2be006c460 0x7f2be006e920 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:49.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:49.853+0000 7f2bfd4ec700 1 --2- 172.21.15.174:0/820376360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf80ffac0 0x7f2bf8069910 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:49.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:49.853+0000 7f2bfd4ec700 1 --2- 172.21.15.174:0/820376360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bf8100cc0 0x7f2bf819b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:49.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:49.853+0000 7f2bfd4ec700 1 -- 172.21.15.174:0/820376360 >> 172.21.15.174:0/820376360 conn(0x7f2bf80fb060 msgr2=0x7f2bf8103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:49.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:49.853+0000 7f2bfd4ec700 1 -- 172.21.15.174:0/820376360 shutdown_connections 2024-02-19T18:57:49.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:49.853+0000 7f2bfd4ec700 1 -- 172.21.15.174:0/820376360 wait complete. 2024-02-19T18:57:50.450 DEBUG:teuthology.orchestra.run.smithi174:osd.3> sudo journalctl -f -n 0 -u ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.3.service 2024-02-19T18:57:50.452 INFO:tasks.cephadm:Deploying osd.4 on smithi174 with /dev/vg_nvme/lv_3... 2024-02-19T18:57:50.452 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-19T18:57:51.041 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:50 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:51.042 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:50 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:51.042 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:50 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:51.042 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:50 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:51.042 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:50 smithi174 ceph-mon[41061]: from='osd.3 [v2:172.21.15.174:6800/2070964484,v1:172.21.15.174:6801/2070964484]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T18:57:51.042 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:50 smithi174 ceph-mon[41061]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T18:57:51.192 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:50 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:51.192 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:50 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:51.192 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:50 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:51.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:50 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:51.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:50 smithi114 ceph-mon[29441]: from='osd.3 [v2:172.21.15.174:6800/2070964484,v1:172.21.15.174:6801/2070964484]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T18:57:51.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:50 smithi114 ceph-mon[29441]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T18:57:51.645 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi174/config 2024-02-19T18:57:52.098 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:51 smithi174 ceph-mon[41061]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-19T18:57:52.098 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:51 smithi174 ceph-mon[41061]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-19T18:57:52.098 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:51 smithi174 ceph-mon[41061]: osdmap e25: 4 total, 3 up, 4 in 2024-02-19T18:57:52.098 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:51 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:57:52.098 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:51 smithi174 ceph-mon[41061]: from='osd.3 [v2:172.21.15.174:6800/2070964484,v1:172.21.15.174:6801/2070964484]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T18:57:52.098 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:51 smithi174 ceph-mon[41061]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T18:57:52.099 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 18:57:51 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3[47098]: 2024-02-19T18:57:51.921+0000 7fc454351700 -1 osd.3 0 waiting for initial osdmap 2024-02-19T18:57:52.099 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 18:57:51 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3[47098]: 2024-02-19T18:57:51.926+0000 7fc44c93f700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T18:57:52.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:51 smithi114 ceph-mon[29441]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-19T18:57:52.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:51 smithi114 ceph-mon[29441]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-19T18:57:52.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:51 smithi114 ceph-mon[29441]: osdmap e25: 4 total, 3 up, 4 in 2024-02-19T18:57:52.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:51 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:57:52.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:51 smithi114 ceph-mon[29441]: from='osd.3 [v2:172.21.15.174:6800/2070964484,v1:172.21.15.174:6801/2070964484]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T18:57:52.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:51 smithi114 ceph-mon[29441]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T18:57:53.130 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:52 smithi174 ceph-mon[41061]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-02-19T18:57:53.130 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:52 smithi174 ceph-mon[41061]: osdmap e26: 4 total, 3 up, 4 in 2024-02-19T18:57:53.130 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:52 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:57:53.130 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:52 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:57:53.130 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:52 smithi174 ceph-mon[41061]: osd.3 [v2:172.21.15.174:6800/2070964484,v1:172.21.15.174:6801/2070964484] boot 2024-02-19T18:57:53.130 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:52 smithi174 ceph-mon[41061]: osdmap e27: 4 total, 4 up, 4 in 2024-02-19T18:57:53.130 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:52 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:57:53.188 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:52 smithi114 ceph-mon[29441]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-02-19T18:57:53.189 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:52 smithi114 ceph-mon[29441]: osdmap e26: 4 total, 3 up, 4 in 2024-02-19T18:57:53.189 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:52 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:57:53.189 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:52 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:57:53.189 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:52 smithi114 ceph-mon[29441]: osd.3 [v2:172.21.15.174:6800/2070964484,v1:172.21.15.174:6801/2070964484] boot 2024-02-19T18:57:53.189 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:52 smithi114 ceph-mon[29441]: osdmap e27: 4 total, 4 up, 4 in 2024-02-19T18:57:53.189 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:52 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:57:53.925 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-19T18:57:53.925 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-19T18:57:53.925 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-02-19T18:57:53.926 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-02-19T18:57:53.926 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0154339 s, 679 MB/s 2024-02-19T18:57:53.926 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-02-19T18:57:53.948 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:53 smithi174 ceph-mon[41061]: purged_snaps scrub starts 2024-02-19T18:57:53.948 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:53 smithi174 ceph-mon[41061]: purged_snaps scrub ok 2024-02-19T18:57:53.949 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:53 smithi174 ceph-mon[41061]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:57:53.949 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:53 smithi174 ceph-mon[41061]: osdmap e28: 4 total, 4 up, 4 in 2024-02-19T18:57:54.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:53 smithi114 ceph-mon[29441]: purged_snaps scrub starts 2024-02-19T18:57:54.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:53 smithi114 ceph-mon[29441]: purged_snaps scrub ok 2024-02-19T18:57:54.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:53 smithi114 ceph-mon[29441]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-19T18:57:54.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:53 smithi114 ceph-mon[29441]: osdmap e28: 4 total, 4 up, 4 in 2024-02-19T18:57:54.946 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi174:vg_nvme/lv_3 2024-02-19T18:57:55.148 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi174/config 2024-02-19T18:57:55.270 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:54 smithi174 ceph-mon[41061]: osdmap e29: 4 total, 4 up, 4 in 2024-02-19T18:57:55.270 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:54 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:55.270 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:54 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:55.270 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:54 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:55.270 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:54 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:55.270 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:54 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:55.270 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:54 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:55.270 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:54 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:55.270 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:54 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:57:55.271 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:54 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:55.271 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:54 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:55.271 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:54 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:55.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:54 smithi114 ceph-mon[29441]: osdmap e29: 4 total, 4 up, 4 in 2024-02-19T18:57:55.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:54 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:55.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:54 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:55.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:54 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:55.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:54 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:55.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:54 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:55.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:54 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:55.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:54 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:55.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:54 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:57:55.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:54 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:55.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:54 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:55.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:54 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:57:56.231 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:55 smithi114 ceph-mon[29441]: Detected new or changed devices on smithi174 2024-02-19T18:57:56.231 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:55 smithi114 ceph-mon[29441]: Adjusting osd_memory_target on smithi174 to 14912M 2024-02-19T18:57:56.232 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:55 smithi114 ceph-mon[29441]: pgmap v103: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 155 KiB/s, 0 objects/s recovering 2024-02-19T18:57:56.232 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:55 smithi114 ceph-mon[29441]: osdmap e30: 4 total, 4 up, 4 in 2024-02-19T18:57:56.276 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:55 smithi174 ceph-mon[41061]: Detected new or changed devices on smithi174 2024-02-19T18:57:56.277 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:55 smithi174 ceph-mon[41061]: Adjusting osd_memory_target on smithi174 to 14912M 2024-02-19T18:57:56.277 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:55 smithi174 ceph-mon[41061]: pgmap v103: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 155 KiB/s, 0 objects/s recovering 2024-02-19T18:57:56.277 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:55 smithi174 ceph-mon[41061]: osdmap e30: 4 total, 4 up, 4 in 2024-02-19T18:57:56.611 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.592+0000 7fa0b489e700 1 -- 172.21.15.174:0/407317222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b00ffac0 msgr2=0x7fa0b00ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:56.611 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.592+0000 7fa0b489e700 1 --2- 172.21.15.174:0/407317222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b00ffac0 0x7fa0b00ffee0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009910 tx=0x7fa0a0009c20 comp rx=0 tx=0).stop 2024-02-19T18:57:56.611 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.592+0000 7fa0b489e700 1 -- 172.21.15.174:0/407317222 shutdown_connections 2024-02-19T18:57:56.611 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.592+0000 7fa0b489e700 1 --2- 172.21.15.174:0/407317222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0b0100cc0 0x7fa0b0101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:56.611 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.592+0000 7fa0b489e700 1 --2- 172.21.15.174:0/407317222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b00ffac0 0x7fa0b00ffee0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:56.611 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.592+0000 7fa0b489e700 1 -- 172.21.15.174:0/407317222 >> 172.21.15.174:0/407317222 conn(0x7fa0b00fb060 msgr2=0x7fa0b00fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:56.611 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.592+0000 7fa0b489e700 1 -- 172.21.15.174:0/407317222 shutdown_connections 2024-02-19T18:57:56.611 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.592+0000 7fa0b489e700 1 -- 172.21.15.174:0/407317222 wait complete. 2024-02-19T18:57:56.611 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.592+0000 7fa0b489e700 1 Processor -- start 2024-02-19T18:57:56.613 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.593+0000 7fa0b489e700 1 -- start start 2024-02-19T18:57:56.613 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.593+0000 7fa0b489e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b00ffac0 0x7fa0b0195290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:56.613 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.593+0000 7fa0b489e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0b0100cc0 0x7fa0b01957d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:56.614 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.593+0000 7fa0b489e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b0195df0 con 0x7fa0b0100cc0 2024-02-19T18:57:56.614 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.593+0000 7fa0b489e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b0195f30 con 0x7fa0b00ffac0 2024-02-19T18:57:56.614 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.593+0000 7fa0ae59c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b00ffac0 0x7fa0b0195290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:56.614 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.593+0000 7fa0ae59c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b00ffac0 0x7fa0b0195290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:50550/0 (socket says 172.21.15.174:50550) 2024-02-19T18:57:56.614 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.593+0000 7fa0ae59c700 1 -- 172.21.15.174:0/1069618462 learned_addr learned my addr 172.21.15.174:0/1069618462 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:57:56.614 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.593+0000 7fa0a5d9b700 1 --2- 172.21.15.174:0/1069618462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0b0100cc0 0x7fa0b01957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:56.614 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.593+0000 7fa0ae59c700 1 -- 172.21.15.174:0/1069618462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0b0100cc0 msgr2=0x7fa0b01957d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:56.614 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.593+0000 7fa0ae59c700 1 --2- 172.21.15.174:0/1069618462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0b0100cc0 0x7fa0b01957d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:56.614 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.593+0000 7fa0ae59c700 1 -- 172.21.15.174:0/1069618462 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a00095f0 con 0x7fa0b00ffac0 2024-02-19T18:57:56.614 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.593+0000 7fa0a5d9b700 1 --2- 172.21.15.174:0/1069618462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0b0100cc0 0x7fa0b01957d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T18:57:56.614 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.593+0000 7fa0ae59c700 1 --2- 172.21.15.174:0/1069618462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b00ffac0 0x7fa0b0195290 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000bda0 tx=0x7fa0a000be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:56.614 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.594+0000 7fa0a7fff700 1 -- 172.21.15.174:0/1069618462 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0a001f070 con 0x7fa0b00ffac0 2024-02-19T18:57:56.614 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.594+0000 7fa0b489e700 1 -- 172.21.15.174:0/1069618462 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b0068db0 con 0x7fa0b00ffac0 2024-02-19T18:57:56.615 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.594+0000 7fa0a7fff700 1 -- 172.21.15.174:0/1069618462 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa0a0019610 con 0x7fa0b00ffac0 2024-02-19T18:57:56.615 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.594+0000 7fa0a7fff700 1 -- 172.21.15.174:0/1069618462 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0a000f670 con 0x7fa0b00ffac0 2024-02-19T18:57:56.615 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.594+0000 7fa0b489e700 1 -- 172.21.15.174:0/1069618462 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b00692a0 con 0x7fa0b00ffac0 2024-02-19T18:57:56.615 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.595+0000 7fa0b489e700 1 -- 172.21.15.174:0/1069618462 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b0108620 con 0x7fa0b00ffac0 2024-02-19T18:57:56.615 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.595+0000 7fa0a7fff700 1 -- 172.21.15.174:0/1069618462 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fa0a0019780 con 0x7fa0b00ffac0 2024-02-19T18:57:56.616 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.595+0000 7fa0a7fff700 1 --2- 172.21.15.174:0/1069618462 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fa09c06c4e0 0x7fa09c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:56.616 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.595+0000 7fa0a7fff700 1 -- 172.21.15.174:0/1069618462 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3774+0+0 (secure 0 0 0) 0x7fa0a008e800 con 0x7fa0b00ffac0 2024-02-19T18:57:56.616 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.596+0000 7fa0a5d9b700 1 --2- 172.21.15.174:0/1069618462 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fa09c06c4e0 0x7fa09c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:56.616 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.596+0000 7fa0a5d9b700 1 --2- 172.21.15.174:0/1069618462 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fa09c06c4e0 0x7fa09c06e9a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa098005950 tx=0x7fa0980058e0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:56.618 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.600+0000 7fa0a7fff700 1 -- 172.21.15.174:0/1069618462 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa0a005d110 con 0x7fa0b00ffac0 2024-02-19T18:57:56.816 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:57:56.798+0000 7fa0b489e700 1 -- 172.21.15.174:0/1069618462 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0b0105610 con 0x7fa09c06c4e0 2024-02-19T18:57:57.284 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:56 smithi174 ceph-mon[41061]: osdmap e31: 4 total, 4 up, 4 in 2024-02-19T18:57:57.284 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:56 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:57:57.284 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:56 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:57:57.284 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:56 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:57.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:56 smithi114 ceph-mon[29441]: osdmap e31: 4 total, 4 up, 4 in 2024-02-19T18:57:57.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:56 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:57:57.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:56 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:57:57.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:56 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:58.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:57 smithi114 ceph-mon[29441]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 155 KiB/s, 0 objects/s recovering 2024-02-19T18:57:58.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:57 smithi114 ceph-mon[29441]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:57:58.291 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:57 smithi174 ceph-mon[41061]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 155 KiB/s, 0 objects/s recovering 2024-02-19T18:57:58.291 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:57 smithi174 ceph-mon[41061]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:58:00.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:59 smithi174 ceph-mon[41061]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-02-19T18:58:00.147 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:59 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/987917206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebf7daeb-8e62-4a0d-ab1c-612cd6a8bf07"}]: dispatch 2024-02-19T18:58:00.148 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:59 smithi174 ceph-mon[41061]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebf7daeb-8e62-4a0d-ab1c-612cd6a8bf07"}]: dispatch 2024-02-19T18:58:00.148 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:59 smithi174 ceph-mon[41061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ebf7daeb-8e62-4a0d-ab1c-612cd6a8bf07"}]': finished 2024-02-19T18:58:00.148 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:59 smithi174 ceph-mon[41061]: osdmap e32: 5 total, 4 up, 5 in 2024-02-19T18:58:00.148 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:57:59 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:58:00.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:59 smithi114 ceph-mon[29441]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-02-19T18:58:00.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:59 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/987917206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebf7daeb-8e62-4a0d-ab1c-612cd6a8bf07"}]: dispatch 2024-02-19T18:58:00.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:59 smithi114 ceph-mon[29441]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebf7daeb-8e62-4a0d-ab1c-612cd6a8bf07"}]: dispatch 2024-02-19T18:58:00.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:59 smithi114 ceph-mon[29441]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ebf7daeb-8e62-4a0d-ab1c-612cd6a8bf07"}]': finished 2024-02-19T18:58:00.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:59 smithi114 ceph-mon[29441]: osdmap e32: 5 total, 4 up, 5 in 2024-02-19T18:58:00.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:57:59 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:58:01.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:00 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/3015765410' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:58:01.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:00 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/3015765410' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:58:02.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:01 smithi114 ceph-mon[29441]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:02.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:01 smithi174 ceph-mon[41061]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:03.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:02 smithi114 ceph-mon[29441]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:03.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:02 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:03.302 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:02 smithi174 ceph-mon[41061]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:03.302 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:02 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:06.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:05 smithi114 ceph-mon[29441]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:06.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:05 smithi174 ceph-mon[41061]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:07.007 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:06 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-19T18:58:07.007 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:06 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:07.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:06 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-19T18:58:07.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:06 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:08.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:07 smithi114 ceph-mon[29441]: Deploying daemon osd.4 on smithi174 2024-02-19T18:58:08.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:07 smithi114 ceph-mon[29441]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:08.058 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:07 smithi174 ceph-mon[41061]: Deploying daemon osd.4 on smithi174 2024-02-19T18:58:08.058 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:07 smithi174 ceph-mon[41061]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:10.089 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:09 smithi174 ceph-mon[41061]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:10.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:09 smithi114 ceph-mon[29441]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:12.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:11 smithi174 ceph-mon[41061]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:12.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:11 smithi114 ceph-mon[29441]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:13.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:12 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:13.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:12 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:13.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:12 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:13.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:12 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:13.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:12 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:13.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:12 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:14.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:13 smithi114 ceph-mon[29441]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:14.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:13 smithi174 ceph-mon[41061]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:15.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:14 smithi174 ceph-mon[41061]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:15.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:14 smithi114 ceph-mon[29441]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:15.630 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:15.612+0000 7fa0a7fff700 1 -- 172.21.15.174:0/1069618462 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa0b0105610 con 0x7fa09c06c4e0 2024-02-19T18:58:15.630 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 4 on host 'smithi174' 2024-02-19T18:58:15.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:15.615+0000 7fa0b489e700 1 -- 172.21.15.174:0/1069618462 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fa09c06c4e0 msgr2=0x7fa09c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:15.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:15.615+0000 7fa0b489e700 1 --2- 172.21.15.174:0/1069618462 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fa09c06c4e0 0x7fa09c06e9a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa098005950 tx=0x7fa0980058e0 comp rx=0 tx=0).stop 2024-02-19T18:58:15.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:15.615+0000 7fa0b489e700 1 -- 172.21.15.174:0/1069618462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b00ffac0 msgr2=0x7fa0b0195290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:15.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:15.615+0000 7fa0b489e700 1 --2- 172.21.15.174:0/1069618462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b00ffac0 0x7fa0b0195290 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000bda0 tx=0x7fa0a000be80 comp rx=0 tx=0).stop 2024-02-19T18:58:15.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:15.615+0000 7fa0b489e700 1 -- 172.21.15.174:0/1069618462 shutdown_connections 2024-02-19T18:58:15.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:15.615+0000 7fa0b489e700 1 --2- 172.21.15.174:0/1069618462 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fa09c06c4e0 0x7fa09c06e9a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:15.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:15.615+0000 7fa0b489e700 1 --2- 172.21.15.174:0/1069618462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0b00ffac0 0x7fa0b0195290 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:15.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:15.615+0000 7fa0b489e700 1 --2- 172.21.15.174:0/1069618462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0b0100cc0 0x7fa0b01957d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:15.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:15.615+0000 7fa0b489e700 1 -- 172.21.15.174:0/1069618462 >> 172.21.15.174:0/1069618462 conn(0x7fa0b00fb060 msgr2=0x7fa0b0103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:15.633 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:15.615+0000 7fa0b489e700 1 -- 172.21.15.174:0/1069618462 shutdown_connections 2024-02-19T18:58:15.634 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:15.615+0000 7fa0b489e700 1 -- 172.21.15.174:0/1069618462 wait complete. 2024-02-19T18:58:16.594 DEBUG:teuthology.orchestra.run.smithi174:osd.4> sudo journalctl -f -n 0 -u ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.4.service 2024-02-19T18:58:16.596 INFO:tasks.cephadm:Deploying osd.5 on smithi174 with /dev/vg_nvme/lv_2... 2024-02-19T18:58:16.596 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-19T18:58:16.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:16.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:16.787 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:16 smithi114 ceph-mon[29441]: from='osd.4 [v2:172.21.15.174:6808/2301992532,v1:172.21.15.174:6809/2301992532]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T18:58:16.787 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:16 smithi114 ceph-mon[29441]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T18:58:16.857 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:16 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:16.857 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:16 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:16.857 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:16 smithi174 ceph-mon[41061]: from='osd.4 [v2:172.21.15.174:6808/2301992532,v1:172.21.15.174:6809/2301992532]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T18:58:16.857 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:16 smithi174 ceph-mon[41061]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T18:58:17.175 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi174/config 2024-02-19T18:58:17.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:17 smithi174 ceph-mon[41061]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-19T18:58:17.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:17 smithi174 ceph-mon[41061]: osdmap e33: 5 total, 4 up, 5 in 2024-02-19T18:58:17.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:17 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:58:17.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:17 smithi174 ceph-mon[41061]: from='osd.4 [v2:172.21.15.174:6808/2301992532,v1:172.21.15.174:6809/2301992532]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T18:58:17.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:17 smithi174 ceph-mon[41061]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T18:58:17.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:17 smithi174 ceph-mon[41061]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:17.893 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 18:58:17 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4[53547]: 2024-02-19T18:58:17.579+0000 7f9815a21700 -1 osd.4 0 waiting for initial osdmap 2024-02-19T18:58:17.893 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 18:58:17 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4[53547]: 2024-02-19T18:58:17.585+0000 7f980e00f700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T18:58:18.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:17 smithi114 ceph-mon[29441]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-19T18:58:18.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:17 smithi114 ceph-mon[29441]: osdmap e33: 5 total, 4 up, 5 in 2024-02-19T18:58:18.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:17 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:58:18.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:17 smithi114 ceph-mon[29441]: from='osd.4 [v2:172.21.15.174:6808/2301992532,v1:172.21.15.174:6809/2301992532]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T18:58:18.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:17 smithi114 ceph-mon[29441]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T18:58:18.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:17 smithi114 ceph-mon[29441]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-19T18:58:18.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:18 smithi174 ceph-mon[41061]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-02-19T18:58:18.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:18 smithi174 ceph-mon[41061]: osdmap e34: 5 total, 4 up, 5 in 2024-02-19T18:58:18.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:18 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:58:18.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:18 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:58:18.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:18 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:19.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:18 smithi114 ceph-mon[29441]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-02-19T18:58:19.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:18 smithi114 ceph-mon[29441]: osdmap e34: 5 total, 4 up, 5 in 2024-02-19T18:58:19.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:18 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:58:19.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:18 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:58:19.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:18 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:19.080 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-19T18:58:19.080 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-19T18:58:19.080 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-02-19T18:58:19.080 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-02-19T18:58:19.080 INFO:teuthology.orchestra.run.smithi174.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126533 s, 829 MB/s 2024-02-19T18:58:19.080 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-02-19T18:58:19.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:19 smithi174 ceph-mon[41061]: purged_snaps scrub starts 2024-02-19T18:58:19.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:19 smithi174 ceph-mon[41061]: purged_snaps scrub ok 2024-02-19T18:58:19.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:19 smithi174 ceph-mon[41061]: osd.4 [v2:172.21.15.174:6808/2301992532,v1:172.21.15.174:6809/2301992532] boot 2024-02-19T18:58:19.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:19 smithi174 ceph-mon[41061]: osdmap e35: 5 total, 5 up, 5 in 2024-02-19T18:58:19.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:19 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:58:19.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:19 smithi174 ceph-mon[41061]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:19.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:19 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:19.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:19 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:19.793 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi174:vg_nvme/lv_2 2024-02-19T18:58:20.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:19 smithi114 ceph-mon[29441]: purged_snaps scrub starts 2024-02-19T18:58:20.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:19 smithi114 ceph-mon[29441]: purged_snaps scrub ok 2024-02-19T18:58:20.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:19 smithi114 ceph-mon[29441]: osd.4 [v2:172.21.15.174:6808/2301992532,v1:172.21.15.174:6809/2301992532] boot 2024-02-19T18:58:20.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:19 smithi114 ceph-mon[29441]: osdmap e35: 5 total, 5 up, 5 in 2024-02-19T18:58:20.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:19 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:58:20.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:19 smithi114 ceph-mon[29441]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:20.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:19 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:20.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:19 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:20.953 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi174/config 2024-02-19T18:58:21.061 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:20 smithi174 ceph-mon[41061]: osdmap e36: 5 total, 5 up, 5 in 2024-02-19T18:58:21.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:20 smithi114 ceph-mon[29441]: osdmap e36: 5 total, 5 up, 5 in 2024-02-19T18:58:22.111 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:21 smithi174 ceph-mon[41061]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:22.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:21 smithi114 ceph-mon[29441]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:23.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.555+0000 7f8bb9c51700 1 -- 172.21.15.174:0/1835118136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bb40ffd90 msgr2=0x7f8bb4100210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:23.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.555+0000 7f8bb9c51700 1 --2- 172.21.15.174:0/1835118136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bb40ffd90 0x7f8bb4100210 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c009910 tx=0x7f8b9c009c20 comp rx=0 tx=0).stop 2024-02-19T18:58:23.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.555+0000 7f8bb9c51700 1 -- 172.21.15.174:0/1835118136 shutdown_connections 2024-02-19T18:58:23.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.555+0000 7f8bb9c51700 1 --2- 172.21.15.174:0/1835118136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bb40ffd90 0x7f8bb4100210 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:23.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.555+0000 7f8bb9c51700 1 --2- 172.21.15.174:0/1835118136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bb40ff430 0x7f8bb40ff850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:23.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.555+0000 7f8bb9c51700 1 -- 172.21.15.174:0/1835118136 >> 172.21.15.174:0/1835118136 conn(0x7f8bb40fb030 msgr2=0x7f8bb40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:23.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.555+0000 7f8bb9c51700 1 -- 172.21.15.174:0/1835118136 shutdown_connections 2024-02-19T18:58:23.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.555+0000 7f8bb9c51700 1 -- 172.21.15.174:0/1835118136 wait complete. 2024-02-19T18:58:23.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.556+0000 7f8bb9c51700 1 Processor -- start 2024-02-19T18:58:23.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.556+0000 7f8bb9c51700 1 -- start start 2024-02-19T18:58:23.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.556+0000 7f8bb9c51700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bb40ff430 0x7f8bb4198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:23.576 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.556+0000 7f8bb9c51700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bb40ffd90 0x7f8bb4198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:23.576 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.556+0000 7f8bb9c51700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb4199060 con 0x7f8bb40ff430 2024-02-19T18:58:23.576 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.556+0000 7f8bb9c51700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb41991a0 con 0x7f8bb40ffd90 2024-02-19T18:58:23.576 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.556+0000 7f8baaffd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bb40ffd90 0x7f8bb4198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:23.576 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.556+0000 7f8baaffd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bb40ffd90 0x7f8bb4198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:48766/0 (socket says 172.21.15.174:48766) 2024-02-19T18:58:23.576 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.556+0000 7f8baaffd700 1 -- 172.21.15.174:0/3522307395 learned_addr learned my addr 172.21.15.174:0/3522307395 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:58:23.576 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.556+0000 7f8bb37fe700 1 --2- 172.21.15.174:0/3522307395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bb40ff430 0x7f8bb4198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:23.576 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.556+0000 7f8baaffd700 1 -- 172.21.15.174:0/3522307395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bb40ff430 msgr2=0x7f8bb4198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:23.576 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.556+0000 7f8baaffd700 1 --2- 172.21.15.174:0/3522307395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bb40ff430 0x7f8bb4198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:23.576 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.556+0000 7f8baaffd700 1 -- 172.21.15.174:0/3522307395 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9c0095f0 con 0x7f8bb40ffd90 2024-02-19T18:58:23.576 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.557+0000 7f8baaffd700 1 --2- 172.21.15.174:0/3522307395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bb40ffd90 0x7f8bb4198a40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c00bd70 tx=0x7f8b9c00be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:23.576 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.557+0000 7f8bb17fa700 1 -- 172.21.15.174:0/3522307395 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b9c01d070 con 0x7f8bb40ffd90 2024-02-19T18:58:23.577 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.557+0000 7f8bb17fa700 1 -- 172.21.15.174:0/3522307395 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8b9c022470 con 0x7f8bb40ffd90 2024-02-19T18:58:23.577 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.557+0000 7f8bb17fa700 1 -- 172.21.15.174:0/3522307395 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b9c00f670 con 0x7f8bb40ffd90 2024-02-19T18:58:23.577 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.557+0000 7f8bb9c51700 1 -- 172.21.15.174:0/3522307395 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb419dbf0 con 0x7f8bb40ffd90 2024-02-19T18:58:23.577 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.557+0000 7f8bb9c51700 1 -- 172.21.15.174:0/3522307395 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb419e0e0 con 0x7f8bb40ffd90 2024-02-19T18:58:23.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.558+0000 7f8bb9c51700 1 -- 172.21.15.174:0/3522307395 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb410b900 con 0x7f8bb40ffd90 2024-02-19T18:58:23.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.558+0000 7f8bb17fa700 1 -- 172.21.15.174:0/3522307395 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f8b9c022a50 con 0x7f8bb40ffd90 2024-02-19T18:58:23.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.559+0000 7f8bb17fa700 1 --2- 172.21.15.174:0/3522307395 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8ba006c4e0 0x7f8ba006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:23.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.559+0000 7f8bb17fa700 1 -- 172.21.15.174:0/3522307395 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4206+0+0 (secure 0 0 0) 0x7f8b9c08c900 con 0x7f8bb40ffd90 2024-02-19T18:58:23.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.559+0000 7f8bb37fe700 1 --2- 172.21.15.174:0/3522307395 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8ba006c4e0 0x7f8ba006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:23.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.559+0000 7f8bb37fe700 1 --2- 172.21.15.174:0/3522307395 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8ba006c4e0 0x7f8ba006e9a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4005fd0 tx=0x7f8ba400a560 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:23.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.563+0000 7f8bb17fa700 1 -- 172.21.15.174:0/3522307395 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8b9c05b190 con 0x7f8bb40ffd90 2024-02-19T18:58:23.779 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:23.761+0000 7f8bb9c51700 1 -- 172.21.15.174:0/3522307395 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f8bb4109180 con 0x7f8ba006c4e0 2024-02-19T18:58:24.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:23 smithi174 ceph-mon[41061]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:24.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:24.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:24.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:58:24.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:58:24.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:24.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:24.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:24.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:24.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:24.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:24.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:24.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:24.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:58:24.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:58:24.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:24.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:23 smithi114 ceph-mon[29441]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:24.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:24.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:24.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:58:24.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:58:24.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:24.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:24.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:24.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:24.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:24.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:24.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:24.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:24.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:58:24.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:58:24.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:25.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:24 smithi114 ceph-mon[29441]: Detected new or changed devices on smithi174 2024-02-19T18:58:25.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:24 smithi114 ceph-mon[29441]: Adjusting osd_memory_target on smithi174 to 7456M 2024-02-19T18:58:25.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:24 smithi114 ceph-mon[29441]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:58:25.337 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:24 smithi174 ceph-mon[41061]: Detected new or changed devices on smithi174 2024-02-19T18:58:25.337 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:24 smithi174 ceph-mon[41061]: Adjusting osd_memory_target on smithi174 to 7456M 2024-02-19T18:58:25.337 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:24 smithi174 ceph-mon[41061]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:58:26.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:25 smithi114 ceph-mon[29441]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:26.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:25 smithi174 ceph-mon[41061]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:27.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:26 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/1313528685' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a8599a6-778b-4367-bfb1-be184ee9d468"}]: dispatch 2024-02-19T18:58:27.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:26 smithi174 ceph-mon[41061]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a8599a6-778b-4367-bfb1-be184ee9d468"}]: dispatch 2024-02-19T18:58:27.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:26 smithi174 ceph-mon[41061]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a8599a6-778b-4367-bfb1-be184ee9d468"}]': finished 2024-02-19T18:58:27.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:26 smithi174 ceph-mon[41061]: osdmap e37: 6 total, 5 up, 6 in 2024-02-19T18:58:27.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:26 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:58:27.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:26 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/1313528685' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a8599a6-778b-4367-bfb1-be184ee9d468"}]: dispatch 2024-02-19T18:58:27.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:26 smithi114 ceph-mon[29441]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a8599a6-778b-4367-bfb1-be184ee9d468"}]: dispatch 2024-02-19T18:58:27.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:26 smithi114 ceph-mon[29441]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a8599a6-778b-4367-bfb1-be184ee9d468"}]': finished 2024-02-19T18:58:27.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:26 smithi114 ceph-mon[29441]: osdmap e37: 6 total, 5 up, 6 in 2024-02-19T18:58:27.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:26 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:58:28.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:27 smithi114 ceph-mon[29441]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:28.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:27 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/4113919317' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:58:28.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:27 smithi174 ceph-mon[41061]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:28.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:27 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/4113919317' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:58:30.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:29 smithi114 ceph-mon[29441]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:30.302 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:29 smithi174 ceph-mon[41061]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:32.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:31 smithi174 ceph-mon[41061]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:32.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:31 smithi114 ceph-mon[29441]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:33.102 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:32 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:33.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:32 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:34.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:33 smithi114 ceph-mon[29441]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:34.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-19T18:58:34.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:34.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:33 smithi174 ceph-mon[41061]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:34.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:33 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-19T18:58:34.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:33 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:35.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:34 smithi114 ceph-mon[29441]: Deploying daemon osd.5 on smithi174 2024-02-19T18:58:35.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:34 smithi174 ceph-mon[41061]: Deploying daemon osd.5 on smithi174 2024-02-19T18:58:36.109 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:35 smithi174 ceph-mon[41061]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:36.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:35 smithi114 ceph-mon[29441]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:37.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:36 smithi114 ceph-mon[29441]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:37.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:36 smithi174 ceph-mon[41061]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:40.214 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:39 smithi174 ceph-mon[41061]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:40.214 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:39 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:40.214 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:39 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:40.214 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:39 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:40.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:39 smithi114 ceph-mon[29441]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:40.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:39 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:40.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:39 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:40.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:39 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:42.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:41 smithi114 ceph-mon[29441]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:42.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:41 smithi174 ceph-mon[41061]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:43.198 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 5 on host 'smithi174' 2024-02-19T18:58:43.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:43.179+0000 7f8bb17fa700 1 -- 172.21.15.174:0/3522307395 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8bb4109180 con 0x7f8ba006c4e0 2024-02-19T18:58:43.199 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:43.181+0000 7f8bb9c51700 1 -- 172.21.15.174:0/3522307395 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8ba006c4e0 msgr2=0x7f8ba006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:43.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:43.181+0000 7f8bb9c51700 1 --2- 172.21.15.174:0/3522307395 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8ba006c4e0 0x7f8ba006e9a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4005fd0 tx=0x7f8ba400a560 comp rx=0 tx=0).stop 2024-02-19T18:58:43.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:43.181+0000 7f8bb9c51700 1 -- 172.21.15.174:0/3522307395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bb40ffd90 msgr2=0x7f8bb4198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:43.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:43.181+0000 7f8bb9c51700 1 --2- 172.21.15.174:0/3522307395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bb40ffd90 0x7f8bb4198a40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c00bd70 tx=0x7f8b9c00be50 comp rx=0 tx=0).stop 2024-02-19T18:58:43.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:43.181+0000 7f8bb9c51700 1 -- 172.21.15.174:0/3522307395 shutdown_connections 2024-02-19T18:58:43.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:43.181+0000 7f8bb9c51700 1 --2- 172.21.15.174:0/3522307395 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8ba006c4e0 0x7f8ba006e9a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:43.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:43.181+0000 7f8bb9c51700 1 --2- 172.21.15.174:0/3522307395 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8bb40ff430 0x7f8bb4198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:43.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:43.181+0000 7f8bb9c51700 1 --2- 172.21.15.174:0/3522307395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bb40ffd90 0x7f8bb4198a40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:43.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:43.181+0000 7f8bb9c51700 1 -- 172.21.15.174:0/3522307395 >> 172.21.15.174:0/3522307395 conn(0x7f8bb40fb030 msgr2=0x7f8bb4107a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:43.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:43.181+0000 7f8bb9c51700 1 -- 172.21.15.174:0/3522307395 shutdown_connections 2024-02-19T18:58:43.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:43.181+0000 7f8bb9c51700 1 -- 172.21.15.174:0/3522307395 wait complete. 2024-02-19T18:58:44.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:43 smithi174 ceph-mon[41061]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:44.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:43 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:44.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:43 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:44.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:43 smithi174 ceph-mon[41061]: from='osd.5 [v2:172.21.15.174:6816/1280361687,v1:172.21.15.174:6817/1280361687]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T18:58:44.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:43 smithi174 ceph-mon[41061]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T18:58:44.206 DEBUG:teuthology.orchestra.run.smithi174:osd.5> sudo journalctl -f -n 0 -u ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.5.service 2024-02-19T18:58:44.209 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-02-19T18:58:44.209 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph osd stat -f json 2024-02-19T18:58:44.240 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:43 smithi114 ceph-mon[29441]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:44.240 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:43 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:44.240 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:43 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:44.240 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:43 smithi114 ceph-mon[29441]: from='osd.5 [v2:172.21.15.174:6816/1280361687,v1:172.21.15.174:6817/1280361687]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T18:58:44.240 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:43 smithi114 ceph-mon[29441]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T18:58:44.431 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:58:45.468 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:45 smithi174 ceph-mon[41061]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-19T18:58:45.469 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:45 smithi174 ceph-mon[41061]: osdmap e38: 6 total, 5 up, 6 in 2024-02-19T18:58:45.469 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:45 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:58:45.469 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:45 smithi174 ceph-mon[41061]: from='osd.5 [v2:172.21.15.174:6816/1280361687,v1:172.21.15.174:6817/1280361687]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T18:58:45.469 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:45 smithi174 ceph-mon[41061]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T18:58:45.469 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:45 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:45.469 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:45 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:45.469 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:45 smithi174 ceph-mon[41061]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:45.469 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 18:58:45 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5[60160]: 2024-02-19T18:58:45.139+0000 7f468bddd700 -1 osd.5 0 waiting for initial osdmap 2024-02-19T18:58:45.469 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 18:58:45 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5[60160]: 2024-02-19T18:58:45.149+0000 7f4684bcc700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T18:58:45.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:45 smithi114 ceph-mon[29441]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-19T18:58:45.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:45 smithi114 ceph-mon[29441]: osdmap e38: 6 total, 5 up, 6 in 2024-02-19T18:58:45.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:45 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:58:45.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:45 smithi114 ceph-mon[29441]: from='osd.5 [v2:172.21.15.174:6816/1280361687,v1:172.21.15.174:6817/1280361687]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T18:58:45.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:45 smithi114 ceph-mon[29441]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T18:58:45.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:45 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:45.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:45 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:45.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:45 smithi114 ceph-mon[29441]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:46.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.113+0000 7f234a0d8700 1 -- 172.21.15.114:0/3291865213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23440ffad0 msgr2=0x7f23440ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:46.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.113+0000 7f234a0d8700 1 --2- 172.21.15.114:0/3291865213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23440ffad0 0x7f23440ffef0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2334009910 tx=0x7f2334009c20 comp rx=0 tx=0).stop 2024-02-19T18:58:46.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.113+0000 7f234a0d8700 1 -- 172.21.15.114:0/3291865213 shutdown_connections 2024-02-19T18:58:46.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.113+0000 7f234a0d8700 1 --2- 172.21.15.114:0/3291865213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2344100cd0 0x7f2344101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.113+0000 7f234a0d8700 1 --2- 172.21.15.114:0/3291865213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23440ffad0 0x7f23440ffef0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.113+0000 7f234a0d8700 1 -- 172.21.15.114:0/3291865213 >> 172.21.15.114:0/3291865213 conn(0x7f23440fb050 msgr2=0x7f23440fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:46.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.114+0000 7f234a0d8700 1 -- 172.21.15.114:0/3291865213 shutdown_connections 2024-02-19T18:58:46.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.114+0000 7f234a0d8700 1 -- 172.21.15.114:0/3291865213 wait complete. 2024-02-19T18:58:46.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.114+0000 7f234a0d8700 1 Processor -- start 2024-02-19T18:58:46.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.114+0000 7f234a0d8700 1 -- start start 2024-02-19T18:58:46.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.114+0000 7f234a0d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23440ffad0 0x7f2344103a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:46.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.114+0000 7f234a0d8700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2344100cd0 0x7f23441020b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:46.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.114+0000 7f234a0d8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23441025f0 con 0x7f23440ffad0 2024-02-19T18:58:46.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.114+0000 7f234a0d8700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2344102730 con 0x7f2344100cd0 2024-02-19T18:58:46.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.115+0000 7f23437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23440ffad0 0x7f2344103a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:46.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.115+0000 7f23437fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23440ffad0 0x7f2344103a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39242/0 (socket says 172.21.15.114:39242) 2024-02-19T18:58:46.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.115+0000 7f23437fe700 1 -- 172.21.15.114:0/768487235 learned_addr learned my addr 172.21.15.114:0/768487235 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:58:46.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.115+0000 7f2342ffd700 1 --2- 172.21.15.114:0/768487235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2344100cd0 0x7f23441020b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:46.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.115+0000 7f23437fe700 1 -- 172.21.15.114:0/768487235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2344100cd0 msgr2=0x7f23441020b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:46.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.115+0000 7f23437fe700 1 --2- 172.21.15.114:0/768487235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2344100cd0 0x7f23441020b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.115+0000 7f23437fe700 1 -- 172.21.15.114:0/768487235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23340095f0 con 0x7f23440ffad0 2024-02-19T18:58:46.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.115+0000 7f23437fe700 1 --2- 172.21.15.114:0/768487235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23440ffad0 0x7f2344103a60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f233400bda0 tx=0x7f233400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:46.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.115+0000 7f2340ff9700 1 -- 172.21.15.114:0/768487235 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f233401d070 con 0x7f23440ffad0 2024-02-19T18:58:46.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.115+0000 7f234a0d8700 1 -- 172.21.15.114:0/768487235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23441029b0 con 0x7f23440ffad0 2024-02-19T18:58:46.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.115+0000 7f2340ff9700 1 -- 172.21.15.114:0/768487235 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2334022470 con 0x7f23440ffad0 2024-02-19T18:58:46.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.115+0000 7f2340ff9700 1 -- 172.21.15.114:0/768487235 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f233400f670 con 0x7f23440ffad0 2024-02-19T18:58:46.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.115+0000 7f234a0d8700 1 -- 172.21.15.114:0/768487235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2344102ea0 con 0x7f23440ffad0 2024-02-19T18:58:46.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.116+0000 7f234a0d8700 1 -- 172.21.15.114:0/768487235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2344108630 con 0x7f23440ffad0 2024-02-19T18:58:46.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.117+0000 7f2340ff9700 1 -- 172.21.15.114:0/768487235 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2334004af0 con 0x7f23440ffad0 2024-02-19T18:58:46.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.117+0000 7f2340ff9700 1 --2- 172.21.15.114:0/768487235 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f232c06c4e0 0x7f232c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:46.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.117+0000 7f2342ffd700 1 --2- 172.21.15.114:0/768487235 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f232c06c4e0 0x7f232c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:46.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.117+0000 7f2340ff9700 1 -- 172.21.15.114:0/768487235 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4358+0+0 (secure 0 0 0) 0x7f233408ca40 con 0x7f23440ffad0 2024-02-19T18:58:46.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.118+0000 7f2342ffd700 1 --2- 172.21.15.114:0/768487235 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f232c06c4e0 0x7f232c06e9a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f233800ac50 tx=0x7f233800a380 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:46.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.122+0000 7f2340ff9700 1 -- 172.21.15.114:0/768487235 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f233405b1b0 con 0x7f23440ffad0 2024-02-19T18:58:46.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:46 smithi114 ceph-mon[29441]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-02-19T18:58:46.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:46 smithi114 ceph-mon[29441]: osdmap e39: 6 total, 5 up, 6 in 2024-02-19T18:58:46.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:46 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:58:46.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:46 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:58:46.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.313+0000 7f234a0d8700 1 -- 172.21.15.114:0/768487235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f234419e850 con 0x7f23440ffad0 2024-02-19T18:58:46.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.314+0000 7f2340ff9700 1 -- 172.21.15.114:0/768487235 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f233405ad40 con 0x7f23440ffad0 2024-02-19T18:58:46.332 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:58:46.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.315+0000 7f234a0d8700 1 -- 172.21.15.114:0/768487235 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f232c06c4e0 msgr2=0x7f232c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:46.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.316+0000 7f234a0d8700 1 --2- 172.21.15.114:0/768487235 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f232c06c4e0 0x7f232c06e9a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f233800ac50 tx=0x7f233800a380 comp rx=0 tx=0).stop 2024-02-19T18:58:46.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.316+0000 7f234a0d8700 1 -- 172.21.15.114:0/768487235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23440ffad0 msgr2=0x7f2344103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:46.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.316+0000 7f234a0d8700 1 --2- 172.21.15.114:0/768487235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23440ffad0 0x7f2344103a60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f233400bda0 tx=0x7f233400bdd0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.316+0000 7f234a0d8700 1 -- 172.21.15.114:0/768487235 shutdown_connections 2024-02-19T18:58:46.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.316+0000 7f234a0d8700 1 --2- 172.21.15.114:0/768487235 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f232c06c4e0 0x7f232c06e9a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.316+0000 7f234a0d8700 1 --2- 172.21.15.114:0/768487235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23440ffad0 0x7f2344103a60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.316+0000 7f234a0d8700 1 --2- 172.21.15.114:0/768487235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2344100cd0 0x7f23441020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.316+0000 7f234a0d8700 1 -- 172.21.15.114:0/768487235 >> 172.21.15.114:0/768487235 conn(0x7f23440fb050 msgr2=0x7f2344068db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:46.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.316+0000 7f234a0d8700 1 -- 172.21.15.114:0/768487235 shutdown_connections 2024-02-19T18:58:46.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:46.316+0000 7f234a0d8700 1 -- 172.21.15.114:0/768487235 wait complete. 2024-02-19T18:58:46.442 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:46 smithi174 ceph-mon[41061]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-02-19T18:58:46.442 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:46 smithi174 ceph-mon[41061]: osdmap e39: 6 total, 5 up, 6 in 2024-02-19T18:58:46.443 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:46 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:58:46.443 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:46 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:58:47.009 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1708369126,"num_in_osds":6,"osd_in_since":1708369106,"num_remapped_pgs":0} 2024-02-19T18:58:47.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-19T18:58:47.265 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:58:47.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:47 smithi114 ceph-mon[29441]: purged_snaps scrub starts 2024-02-19T18:58:47.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:47 smithi114 ceph-mon[29441]: purged_snaps scrub ok 2024-02-19T18:58:47.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:47 smithi114 ceph-mon[29441]: osd.5 [v2:172.21.15.174:6816/1280361687,v1:172.21.15.174:6817/1280361687] boot 2024-02-19T18:58:47.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:47 smithi114 ceph-mon[29441]: osdmap e40: 6 total, 6 up, 6 in 2024-02-19T18:58:47.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:47 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:58:47.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:47 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/768487235' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-19T18:58:47.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:47 smithi114 ceph-mon[29441]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:47.636 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:47 smithi174 ceph-mon[41061]: purged_snaps scrub starts 2024-02-19T18:58:47.636 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:47 smithi174 ceph-mon[41061]: purged_snaps scrub ok 2024-02-19T18:58:47.636 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:47 smithi174 ceph-mon[41061]: osd.5 [v2:172.21.15.174:6816/1280361687,v1:172.21.15.174:6817/1280361687] boot 2024-02-19T18:58:47.637 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:47 smithi174 ceph-mon[41061]: osdmap e40: 6 total, 6 up, 6 in 2024-02-19T18:58:47.637 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:47 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:58:47.637 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:47 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/768487235' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-19T18:58:47.637 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:47 smithi174 ceph-mon[41061]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-19T18:58:48.379 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:48 smithi114 ceph-mon[29441]: osdmap e41: 6 total, 6 up, 6 in 2024-02-19T18:58:48.380 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:48 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:48.380 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:48 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:48.380 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:48 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:48.380 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:48 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:58:48.380 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:48 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:58:48.380 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:48 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:58:48.380 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:48 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:48.380 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:48 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:48.380 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:48 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:48.380 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:48 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:48.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:48 smithi174 ceph-mon[41061]: osdmap e41: 6 total, 6 up, 6 in 2024-02-19T18:58:48.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:48 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:48.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:48 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:48.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:48 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:48.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:48 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:58:48.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:48 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:58:48.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:48 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:58:48.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:48 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:48.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:48 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:48.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:48 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:48.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:48 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.906+0000 7ff436982700 1 -- 172.21.15.114:0/1410438217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff430069020 msgr2=0x7ff430105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.906+0000 7ff436982700 1 --2- 172.21.15.114:0/1410438217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff430069020 0x7ff430105790 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff424009910 tx=0x7ff424009c20 comp rx=0 tx=0).stop 2024-02-19T18:58:48.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.906+0000 7ff436982700 1 -- 172.21.15.114:0/1410438217 shutdown_connections 2024-02-19T18:58:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.906+0000 7ff436982700 1 --2- 172.21.15.114:0/1410438217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff430069020 0x7ff430105790 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.906+0000 7ff436982700 1 --2- 172.21.15.114:0/1410438217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff430068670 0x7ff430068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.906+0000 7ff436982700 1 -- 172.21.15.114:0/1410438217 >> 172.21.15.114:0/1410438217 conn(0x7ff4300755a0 msgr2=0x7ff4300759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.906+0000 7ff436982700 1 -- 172.21.15.114:0/1410438217 shutdown_connections 2024-02-19T18:58:48.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.906+0000 7ff436982700 1 -- 172.21.15.114:0/1410438217 wait complete. 2024-02-19T18:58:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.907+0000 7ff436982700 1 Processor -- start 2024-02-19T18:58:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.907+0000 7ff436982700 1 -- start start 2024-02-19T18:58:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.907+0000 7ff436982700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff430068670 0x7ff4300ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.907+0000 7ff436982700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff430069020 0x7ff4301001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.907+0000 7ff436982700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff430100720 con 0x7ff430068670 2024-02-19T18:58:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.907+0000 7ff436982700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff430100860 con 0x7ff430069020 2024-02-19T18:58:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.907+0000 7ff42ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff430068670 0x7ff4300ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:48.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.907+0000 7ff42ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff430068670 0x7ff4300ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39260/0 (socket says 172.21.15.114:39260) 2024-02-19T18:58:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.907+0000 7ff42ffff700 1 -- 172.21.15.114:0/1748048870 learned_addr learned my addr 172.21.15.114:0/1748048870 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:58:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.907+0000 7ff42ffff700 1 -- 172.21.15.114:0/1748048870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff430069020 msgr2=0x7ff4301001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.907+0000 7ff42ffff700 1 --2- 172.21.15.114:0/1748048870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff430069020 0x7ff4301001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.908+0000 7ff42ffff700 1 -- 172.21.15.114:0/1748048870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4240095f0 con 0x7ff430068670 2024-02-19T18:58:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.908+0000 7ff42ffff700 1 --2- 172.21.15.114:0/1748048870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff430068670 0x7ff4300ffca0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff42000c8a0 tx=0x7ff42000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.908+0000 7ff42d7fa700 1 -- 172.21.15.114:0/1748048870 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff42000cea0 con 0x7ff430068670 2024-02-19T18:58:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.908+0000 7ff42d7fa700 1 -- 172.21.15.114:0/1748048870 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff420004830 con 0x7ff430068670 2024-02-19T18:58:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.908+0000 7ff436982700 1 -- 172.21.15.114:0/1748048870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4301a28d0 con 0x7ff430068670 2024-02-19T18:58:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.908+0000 7ff42d7fa700 1 -- 172.21.15.114:0/1748048870 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4200055b0 con 0x7ff430068670 2024-02-19T18:58:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.908+0000 7ff436982700 1 -- 172.21.15.114:0/1748048870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4301a2ce0 con 0x7ff430068670 2024-02-19T18:58:48.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.909+0000 7ff436982700 1 -- 172.21.15.114:0/1748048870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4301090a0 con 0x7ff430068670 2024-02-19T18:58:48.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.909+0000 7ff42d7fa700 1 -- 172.21.15.114:0/1748048870 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ff420007ad0 con 0x7ff430068670 2024-02-19T18:58:48.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.910+0000 7ff42d7fa700 1 --2- 172.21.15.114:0/1748048870 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff41806c4e0 0x7ff41806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:48.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.910+0000 7ff42d7fa700 1 -- 172.21.15.114:0/1748048870 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff42008b170 con 0x7ff430068670 2024-02-19T18:58:48.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.910+0000 7ff42f7fe700 1 --2- 172.21.15.114:0/1748048870 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff41806c4e0 0x7ff41806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:48.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.910+0000 7ff42f7fe700 1 --2- 172.21.15.114:0/1748048870 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff41806c4e0 0x7ff41806e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff42400b5c0 tx=0x7ff424005960 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:48.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:48.914+0000 7ff42d7fa700 1 -- 172.21.15.114:0/1748048870 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff420055d10 con 0x7ff430068670 2024-02-19T18:58:49.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:49.104+0000 7ff436982700 1 -- 172.21.15.114:0/1748048870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff43004ea90 con 0x7ff430068670 2024-02-19T18:58:49.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:49.105+0000 7ff42d7fa700 1 -- 172.21.15.114:0/1748048870 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11370 (secure 0 0 0) 0x7ff420059330 con 0x7ff430068670 2024-02-19T18:58:49.123 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:58:49.123 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":41,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","created":"2024-02-19T18:51:33.283071+0000","modified":"2024-02-19T18:58:47.142333+0000","last_up_change":"2024-02-19T18:58:46.137044+0000","last_in_change":"2024-02-19T18:58:26.384214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T18:57:30.943324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"93a2d632-7629-4a15-b026-93b40fba59d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2204928688},{"type":"v1","addr":"172.21.15.114:6803","nonce":2204928688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2204928688},{"type":"v1","addr":"172.21.15.114:6805","nonce":2204928688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2204928688},{"type":"v1","addr":"172.21.15.114:6809","nonce":2204928688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2204928688},{"type":"v1","addr":"172.21.15.114:6807","nonce":2204928688}]},"public_addr":"172.21.15.114:6803/2204928688","cluster_addr":"172.21.15.114:6805/2204928688","heartbeat_back_addr":"172.21.15.114:6809/2204928688","heartbeat_front_addr":"172.21.15.114:6807/2204928688","state":["exists","up"]},{"osd":1,"uuid":"8334d634-9c49-4614-88e0-ca4545fa761b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":828515125},{"type":"v1","addr":"172.21.15.114:6811","nonce":828515125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":828515125},{"type":"v1","addr":"172.21.15.114:6813","nonce":828515125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":828515125},{"type":"v1","addr":"172.21.15.114:6817","nonce":828515125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":828515125},{"type":"v1","addr":"172.21.15.114:6815","nonce":828515125}]},"public_addr":"172.21.15.114:6811/828515125","cluster_addr":"172.21.15.114:6813/828515125","heartbeat_back_addr":"172.21.15.114:6817/828515125","heartbeat_front_addr":"172.21.15.114:6815/828515125","state":["exists","up"]},{"osd":2,"uuid":"2d27485a-1fb1-432d-a5a4-f46c8031a6ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":680148196},{"type":"v1","addr":"172.21.15.114:6819","nonce":680148196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":680148196},{"type":"v1","addr":"172.21.15.114:6821","nonce":680148196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":680148196},{"type":"v1","addr":"172.21.15.114:6825","nonce":680148196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":680148196},{"type":"v1","addr":"172.21.15.114:6823","nonce":680148196}]},"public_addr":"172.21.15.114:6819/680148196","cluster_addr":"172.21.15.114:6821/680148196","heartbeat_back_addr":"172.21.15.114:6825/680148196","heartbeat_front_addr":"172.21.15.114:6823/680148196","state":["exists","up"]},{"osd":3,"uuid":"fdbbfdf3-8e5c-472c-9894-97741a049a31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":2070964484},{"type":"v1","addr":"172.21.15.174:6801","nonce":2070964484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":2070964484},{"type":"v1","addr":"172.21.15.174:6803","nonce":2070964484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":2070964484},{"type":"v1","addr":"172.21.15.174:6807","nonce":2070964484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":2070964484},{"type":"v1","addr":"172.21.15.174:6805","nonce":2070964484}]},"public_addr":"172.21.15.174:6801/2070964484","cluster_addr":"172.21.15.174:6803/2070964484","heartbeat_back_addr":"172.21.15.174:6807/2070964484","heartbeat_front_addr":"172.21.15.174:6805/2070964484","state":["exists","up"]},{"osd":4,"uuid":"ebf7daeb-8e62-4a0d-ab1c-612cd6a8bf07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":2301992532},{"type":"v1","addr":"172.21.15.174:6809","nonce":2301992532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":2301992532},{"type":"v1","addr":"172.21.15.174:6811","nonce":2301992532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":2301992532},{"type":"v1","addr":"172.21.15.174:6815","nonce":2301992532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":2301992532},{"type":"v1","addr":"172.21.15.174:6813","nonce":2301992532}]},"public_addr":"172.21.15.174:6809/2301992532","cluster_addr":"172.21.15.174:6811/2301992532","heartbeat_back_addr":"172.21.15.174:6815/2301992532","heartbeat_front_addr":"172.21.15.174:6813/2301992532","state":["exists","up"]},{"osd":5,"uuid":"9a8599a6-778b-4367-bfb1-be184ee9d468","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1280361687},{"type":"v1","addr":"172.21.15.174:6817","nonce":1280361687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1280361687},{"type":"v1","addr":"172.21.15.174:6819","nonce":1280361687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1280361687},{"type":"v1","addr":"172.21.15.174:6823","nonce":1280361687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1280361687},{"type":"v1","addr":"172.21.15.174:6821","nonce":1280361687}]},"public_addr":"172.21.15.174:6817/1280361687","cluster_addr":"172.21.15.174:6819/1280361687","heartbeat_back_addr":"172.21.15.174:6823/1280361687","heartbeat_front_addr":"172.21.15.174:6821/1280361687","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:56:27.015050+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:56:56.239571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:57:26.893375+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:57:50.935953+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:16.875224+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:44.453528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/1382855124":"2024-02-20T18:55:02.605941+0000","172.21.15.114:0/3946301125":"2024-02-20T18:55:02.605941+0000","172.21.15.114:6801/1719431365":"2024-02-20T18:55:02.605941+0000","172.21.15.114:0/110036562":"2024-02-20T18:55:02.605941+0000","172.21.15.114:6801/971741305":"2024-02-20T18:52:38.087813+0000","172.21.15.114:6800/1719431365":"2024-02-20T18:55:02.605941+0000","172.21.15.114:0/857878097":"2024-02-20T18:52:38.087813+0000","172.21.15.114:6800/744561179":"2024-02-20T18:52:05.548945+0000","172.21.15.114:6800/971741305":"2024-02-20T18:52:38.087813+0000","172.21.15.114:6801/744561179":"2024-02-20T18:52:05.548945+0000","172.21.15.114:0/1056805783":"2024-02-20T18:52:05.548945+0000","172.21.15.114:0/945825118":"2024-02-20T18:52:38.087813+0000","172.21.15.114:0/169338133":"2024-02-20T18:52:05.548945+0000","172.21.15.114:0/1245656641":"2024-02-20T18:52:05.548945+0000","172.21.15.114:0/2533620288":"2024-02-20T18:52:38.087813+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T18:58:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:49.107+0000 7ff436982700 1 -- 172.21.15.114:0/1748048870 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff41806c4e0 msgr2=0x7ff41806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:49.107+0000 7ff436982700 1 --2- 172.21.15.114:0/1748048870 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff41806c4e0 0x7ff41806e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff42400b5c0 tx=0x7ff424005960 comp rx=0 tx=0).stop 2024-02-19T18:58:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:49.107+0000 7ff436982700 1 -- 172.21.15.114:0/1748048870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff430068670 msgr2=0x7ff4300ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:49.107+0000 7ff436982700 1 --2- 172.21.15.114:0/1748048870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff430068670 0x7ff4300ffca0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff42000c8a0 tx=0x7ff42000cc60 comp rx=0 tx=0).stop 2024-02-19T18:58:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:49.107+0000 7ff436982700 1 -- 172.21.15.114:0/1748048870 shutdown_connections 2024-02-19T18:58:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:49.107+0000 7ff436982700 1 --2- 172.21.15.114:0/1748048870 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff41806c4e0 0x7ff41806e9a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:49.107+0000 7ff436982700 1 --2- 172.21.15.114:0/1748048870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff430068670 0x7ff4300ffca0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:49.107+0000 7ff436982700 1 --2- 172.21.15.114:0/1748048870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff430069020 0x7ff4301001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:49.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:49.107+0000 7ff436982700 1 -- 172.21.15.114:0/1748048870 >> 172.21.15.114:0/1748048870 conn(0x7ff4300755a0 msgr2=0x7ff430105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:49.107+0000 7ff436982700 1 -- 172.21.15.114:0/1748048870 shutdown_connections 2024-02-19T18:58:49.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:49.107+0000 7ff436982700 1 -- 172.21.15.114:0/1748048870 wait complete. 2024-02-19T18:58:49.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:49 smithi114 ceph-mon[29441]: Detected new or changed devices on smithi174 2024-02-19T18:58:49.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:49 smithi114 ceph-mon[29441]: Adjusting osd_memory_target on smithi174 to 4970M 2024-02-19T18:58:49.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:49 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:49.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:49 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:49.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:49 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:49.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:49 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:49.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:49 smithi114 ceph-mon[29441]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:58:49.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:49 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1748048870' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T18:58:49.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:49 smithi174 ceph-mon[41061]: Detected new or changed devices on smithi174 2024-02-19T18:58:49.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:49 smithi174 ceph-mon[41061]: Adjusting osd_memory_target on smithi174 to 4970M 2024-02-19T18:58:49.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:49 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:49.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:49 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:49.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:49 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:49.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:49 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:58:49.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:49 smithi174 ceph-mon[41061]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:58:49.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:49 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1748048870' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T18:58:49.775 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T18:57:30.943324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T18:58:49.775 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-19T18:58:49.983 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:58:51.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.531+0000 7ffaee088700 1 -- 172.21.15.114:0/2293289057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffae80ff510 msgr2=0x7ffae80ff8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:51.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.531+0000 7ffaee088700 1 --2- 172.21.15.114:0/2293289057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffae80ff510 0x7ffae80ff8f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ffad8009910 tx=0x7ffad8009c20 comp rx=0 tx=0).stop 2024-02-19T18:58:51.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.531+0000 7ffaee088700 1 -- 172.21.15.114:0/2293289057 shutdown_connections 2024-02-19T18:58:51.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.531+0000 7ffaee088700 1 --2- 172.21.15.114:0/2293289057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffae80ffec0 0x7ffae81057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:51.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.531+0000 7ffaee088700 1 --2- 172.21.15.114:0/2293289057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffae80ff510 0x7ffae80ff8f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:51.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.531+0000 7ffaee088700 1 -- 172.21.15.114:0/2293289057 >> 172.21.15.114:0/2293289057 conn(0x7ffae8074e00 msgr2=0x7ffae8075210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:51.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.531+0000 7ffaee088700 1 -- 172.21.15.114:0/2293289057 shutdown_connections 2024-02-19T18:58:51.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.531+0000 7ffaee088700 1 -- 172.21.15.114:0/2293289057 wait complete. 2024-02-19T18:58:51.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.532+0000 7ffaee088700 1 Processor -- start 2024-02-19T18:58:51.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.532+0000 7ffaee088700 1 -- start start 2024-02-19T18:58:51.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.532+0000 7ffaee088700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffae80ff510 0x7ffae819c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:51.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.532+0000 7ffaee088700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffae80ffec0 0x7ffae819cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:51.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.532+0000 7ffaee088700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae819d1b0 con 0x7ffae80ffec0 2024-02-19T18:58:51.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.532+0000 7ffaee088700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae8196660 con 0x7ffae80ff510 2024-02-19T18:58:51.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.532+0000 7ffadeffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffae80ffec0 0x7ffae819cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:51.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.532+0000 7ffadeffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffae80ffec0 0x7ffae819cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39276/0 (socket says 172.21.15.114:39276) 2024-02-19T18:58:51.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.532+0000 7ffadeffd700 1 -- 172.21.15.114:0/1623665177 learned_addr learned my addr 172.21.15.114:0/1623665177 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:58:51.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.533+0000 7ffae77fe700 1 --2- 172.21.15.114:0/1623665177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffae80ff510 0x7ffae819c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:51.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.533+0000 7ffadeffd700 1 -- 172.21.15.114:0/1623665177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffae80ff510 msgr2=0x7ffae819c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:51.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.533+0000 7ffadeffd700 1 --2- 172.21.15.114:0/1623665177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffae80ff510 0x7ffae819c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:51.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.533+0000 7ffadeffd700 1 -- 172.21.15.114:0/1623665177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad80095f0 con 0x7ffae80ffec0 2024-02-19T18:58:51.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.533+0000 7ffae77fe700 1 --2- 172.21.15.114:0/1623665177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffae80ff510 0x7ffae819c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T18:58:51.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.533+0000 7ffadeffd700 1 --2- 172.21.15.114:0/1623665177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffae80ffec0 0x7ffae819cb20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffad000b480 tx=0x7ffad000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:51.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.533+0000 7ffae57fa700 1 -- 172.21.15.114:0/1623665177 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffad00118c0 con 0x7ffae80ffec0 2024-02-19T18:58:51.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.533+0000 7ffae57fa700 1 -- 172.21.15.114:0/1623665177 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ffad0010380 con 0x7ffae80ffec0 2024-02-19T18:58:51.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.533+0000 7ffaee088700 1 -- 172.21.15.114:0/1623665177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae8196940 con 0x7ffae80ffec0 2024-02-19T18:58:51.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.533+0000 7ffae57fa700 1 -- 172.21.15.114:0/1623665177 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffad000f4f0 con 0x7ffae80ffec0 2024-02-19T18:58:51.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.533+0000 7ffaee088700 1 -- 172.21.15.114:0/1623665177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae8196e90 con 0x7ffae80ffec0 2024-02-19T18:58:51.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.534+0000 7ffaee088700 1 -- 172.21.15.114:0/1623665177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae81091e0 con 0x7ffae80ffec0 2024-02-19T18:58:51.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.534+0000 7ffae57fa700 1 -- 172.21.15.114:0/1623665177 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ffad00104f0 con 0x7ffae80ffec0 2024-02-19T18:58:51.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.535+0000 7ffae57fa700 1 --2- 172.21.15.114:0/1623665177 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ffad406c4e0 0x7ffad406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:51.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.535+0000 7ffae57fa700 1 -- 172.21.15.114:0/1623665177 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ffad008b130 con 0x7ffae80ffec0 2024-02-19T18:58:51.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.535+0000 7ffae77fe700 1 --2- 172.21.15.114:0/1623665177 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ffad406c4e0 0x7ffad406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:51.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.535+0000 7ffae77fe700 1 --2- 172.21.15.114:0/1623665177 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ffad406c4e0 0x7ffad406e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffad800b5c0 tx=0x7ffad801a040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:51.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.539+0000 7ffae57fa700 1 -- 172.21.15.114:0/1623665177 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ffad00596e0 con 0x7ffae80ffec0 2024-02-19T18:58:51.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.749+0000 7ffaee088700 1 -- 172.21.15.114:0/1623665177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffae8197170 con 0x7ffae80ffec0 2024-02-19T18:58:51.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.749+0000 7ffae57fa700 1 -- 172.21.15.114:0/1623665177 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffad0059270 con 0x7ffae80ffec0 2024-02-19T18:58:51.768 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-02-19T18:58:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.751+0000 7ffaee088700 1 -- 172.21.15.114:0/1623665177 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ffad406c4e0 msgr2=0x7ffad406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.751+0000 7ffaee088700 1 --2- 172.21.15.114:0/1623665177 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ffad406c4e0 0x7ffad406e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffad800b5c0 tx=0x7ffad801a040 comp rx=0 tx=0).stop 2024-02-19T18:58:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.751+0000 7ffaee088700 1 -- 172.21.15.114:0/1623665177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffae80ffec0 msgr2=0x7ffae819cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.751+0000 7ffaee088700 1 --2- 172.21.15.114:0/1623665177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffae80ffec0 0x7ffae819cb20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffad000b480 tx=0x7ffad000b840 comp rx=0 tx=0).stop 2024-02-19T18:58:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.752+0000 7ffaee088700 1 -- 172.21.15.114:0/1623665177 shutdown_connections 2024-02-19T18:58:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.752+0000 7ffaee088700 1 --2- 172.21.15.114:0/1623665177 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ffad406c4e0 0x7ffad406e9a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.752+0000 7ffaee088700 1 --2- 172.21.15.114:0/1623665177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffae80ff510 0x7ffae819c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.752+0000 7ffaee088700 1 --2- 172.21.15.114:0/1623665177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffae80ffec0 0x7ffae819cb20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.752+0000 7ffaee088700 1 -- 172.21.15.114:0/1623665177 >> 172.21.15.114:0/1623665177 conn(0x7ffae8074e00 msgr2=0x7ffae80feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.752+0000 7ffaee088700 1 -- 172.21.15.114:0/1623665177 shutdown_connections 2024-02-19T18:58:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:51.752+0000 7ffaee088700 1 -- 172.21.15.114:0/1623665177 wait complete. 2024-02-19T18:58:52.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:51 smithi114 ceph-mon[29441]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:58:52.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:51 smithi174 ceph-mon[41061]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:58:52.448 INFO:tasks.cephadm:Setting up client nodes... 2024-02-19T18:58:52.449 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-19T18:58:52.654 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:58:53.028 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:52 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1623665177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-19T18:58:53.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:52 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1623665177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-19T18:58:54.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:53 smithi114 ceph-mon[29441]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:58:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.287+0000 7f60eaa0b700 1 -- 172.21.15.114:0/560536681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e4101dc0 msgr2=0x7f60e4102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.287+0000 7f60eaa0b700 1 --2- 172.21.15.114:0/560536681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e4101dc0 0x7f60e4102220 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f60d8009910 tx=0x7f60d8009c20 comp rx=0 tx=0).stop 2024-02-19T18:58:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.287+0000 7f60eaa0b700 1 -- 172.21.15.114:0/560536681 shutdown_connections 2024-02-19T18:58:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.287+0000 7f60eaa0b700 1 --2- 172.21.15.114:0/560536681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e4101dc0 0x7f60e4102220 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.287+0000 7f60eaa0b700 1 --2- 172.21.15.114:0/560536681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60e4100bc0 0x7f60e4100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.287+0000 7f60eaa0b700 1 -- 172.21.15.114:0/560536681 >> 172.21.15.114:0/560536681 conn(0x7f60e40fc140 msgr2=0x7f60e40fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.287+0000 7f60eaa0b700 1 -- 172.21.15.114:0/560536681 shutdown_connections 2024-02-19T18:58:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.287+0000 7f60eaa0b700 1 -- 172.21.15.114:0/560536681 wait complete. 2024-02-19T18:58:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.287+0000 7f60eaa0b700 1 Processor -- start 2024-02-19T18:58:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.287+0000 7f60eaa0b700 1 -- start start 2024-02-19T18:58:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.288+0000 7f60eaa0b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e4100bc0 0x7f60e4196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.288+0000 7f60eaa0b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60e4101dc0 0x7f60e41968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.288+0000 7f60eaa0b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e4196ed0 con 0x7f60e4100bc0 2024-02-19T18:58:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.288+0000 7f60eaa0b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e4197010 con 0x7f60e4101dc0 2024-02-19T18:58:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.288+0000 7f60e3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e4100bc0 0x7f60e4196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.288+0000 7f60e3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e4100bc0 0x7f60e4196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39292/0 (socket says 172.21.15.114:39292) 2024-02-19T18:58:54.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.288+0000 7f60e3fff700 1 -- 172.21.15.114:0/1393054741 learned_addr learned my addr 172.21.15.114:0/1393054741 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:58:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.288+0000 7f60e37fe700 1 --2- 172.21.15.114:0/1393054741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60e4101dc0 0x7f60e41968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.288+0000 7f60e3fff700 1 -- 172.21.15.114:0/1393054741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60e4101dc0 msgr2=0x7f60e41968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.288+0000 7f60e3fff700 1 --2- 172.21.15.114:0/1393054741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60e4101dc0 0x7f60e41968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.288+0000 7f60e3fff700 1 -- 172.21.15.114:0/1393054741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d80095f0 con 0x7f60e4100bc0 2024-02-19T18:58:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.288+0000 7f60e3fff700 1 --2- 172.21.15.114:0/1393054741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e4100bc0 0x7f60e4196370 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f60d400d900 tx=0x7f60d400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.288+0000 7f60e17fa700 1 -- 172.21.15.114:0/1393054741 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60d40041f0 con 0x7f60e4100bc0 2024-02-19T18:58:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.288+0000 7f60e17fa700 1 -- 172.21.15.114:0/1393054741 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f60d4004350 con 0x7f60e4100bc0 2024-02-19T18:58:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.288+0000 7f60eaa0b700 1 -- 172.21.15.114:0/1393054741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e419bac0 con 0x7f60e4100bc0 2024-02-19T18:58:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.288+0000 7f60e17fa700 1 -- 172.21.15.114:0/1393054741 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60d4003d20 con 0x7f60e4100bc0 2024-02-19T18:58:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.288+0000 7f60eaa0b700 1 -- 172.21.15.114:0/1393054741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e419c010 con 0x7f60e4100bc0 2024-02-19T18:58:54.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.289+0000 7f60eaa0b700 1 -- 172.21.15.114:0/1393054741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e4109720 con 0x7f60e4100bc0 2024-02-19T18:58:54.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.290+0000 7f60e17fa700 1 -- 172.21.15.114:0/1393054741 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f60d4009730 con 0x7f60e4100bc0 2024-02-19T18:58:54.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.290+0000 7f60e17fa700 1 --2- 172.21.15.114:0/1393054741 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f60cc06c4e0 0x7f60cc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:54.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.290+0000 7f60e17fa700 1 -- 172.21.15.114:0/1393054741 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f60d408b370 con 0x7f60e4100bc0 2024-02-19T18:58:54.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.290+0000 7f60e37fe700 1 --2- 172.21.15.114:0/1393054741 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f60cc06c4e0 0x7f60cc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:54.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.291+0000 7f60e37fe700 1 --2- 172.21.15.114:0/1393054741 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f60cc06c4e0 0x7f60cc06e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f60d8006010 tx=0x7f60d800b540 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:54.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.295+0000 7f60e17fa700 1 -- 172.21.15.114:0/1393054741 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f60d4059920 con 0x7f60e4100bc0 2024-02-19T18:58:54.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:53 smithi174 ceph-mon[41061]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:58:54.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.564+0000 7f60eaa0b700 1 -- 172.21.15.114:0/1393054741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f60e419c620 con 0x7f60e4100bc0 2024-02-19T18:58:54.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.568+0000 7f60e17fa700 1 -- 172.21.15.114:0/1393054741 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f60d4003e80 con 0x7f60e4100bc0 2024-02-19T18:58:54.586 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-02-19T18:58:54.587 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDupNNlnKTCIRAA6I03gFJTRmPE71jpbWLQhA== 2024-02-19T18:58:54.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.570+0000 7f60eaa0b700 1 -- 172.21.15.114:0/1393054741 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f60cc06c4e0 msgr2=0x7f60cc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:54.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.570+0000 7f60eaa0b700 1 --2- 172.21.15.114:0/1393054741 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f60cc06c4e0 0x7f60cc06e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f60d8006010 tx=0x7f60d800b540 comp rx=0 tx=0).stop 2024-02-19T18:58:54.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.570+0000 7f60eaa0b700 1 -- 172.21.15.114:0/1393054741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e4100bc0 msgr2=0x7f60e4196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:54.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.570+0000 7f60eaa0b700 1 --2- 172.21.15.114:0/1393054741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e4100bc0 0x7f60e4196370 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f60d400d900 tx=0x7f60d400dc10 comp rx=0 tx=0).stop 2024-02-19T18:58:54.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.570+0000 7f60eaa0b700 1 -- 172.21.15.114:0/1393054741 shutdown_connections 2024-02-19T18:58:54.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.570+0000 7f60eaa0b700 1 --2- 172.21.15.114:0/1393054741 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f60cc06c4e0 0x7f60cc06e9a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:54.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.570+0000 7f60eaa0b700 1 --2- 172.21.15.114:0/1393054741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e4100bc0 0x7f60e4196370 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:54.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.570+0000 7f60eaa0b700 1 --2- 172.21.15.114:0/1393054741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60e4101dc0 0x7f60e41968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:54.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.570+0000 7f60eaa0b700 1 -- 172.21.15.114:0/1393054741 >> 172.21.15.114:0/1393054741 conn(0x7f60e40fc140 msgr2=0x7f60e4104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:54.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.570+0000 7f60eaa0b700 1 -- 172.21.15.114:0/1393054741 shutdown_connections 2024-02-19T18:58:54.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:54.571+0000 7f60eaa0b700 1 -- 172.21.15.114:0/1393054741 wait complete. 2024-02-19T18:58:55.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:54 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1393054741' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T18:58:55.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:54 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1393054741' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T18:58:55.343 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T18:58:55.343 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-19T18:58:55.343 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-19T18:58:55.385 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-19T18:58:55.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:54 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1393054741' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T18:58:55.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:54 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1393054741' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T18:58:55.606 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi174/config 2024-02-19T18:58:56.025 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:55 smithi174 ceph-mon[41061]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:58:56.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:55 smithi114 ceph-mon[29441]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:58:57.178 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.158+0000 7f3dd95ec700 1 -- 172.21.15.174:0/379604400 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dd4101dd0 msgr2=0x7f3dd4102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:57.178 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.158+0000 7f3dd95ec700 1 --2- 172.21.15.174:0/379604400 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dd4101dd0 0x7f3dd4102230 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8009910 tx=0x7f3dc8009c20 comp rx=0 tx=0).stop 2024-02-19T18:58:57.178 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.158+0000 7f3dd95ec700 1 -- 172.21.15.174:0/379604400 shutdown_connections 2024-02-19T18:58:57.178 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.158+0000 7f3dd95ec700 1 --2- 172.21.15.174:0/379604400 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dd4101dd0 0x7f3dd4102230 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:57.178 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.158+0000 7f3dd95ec700 1 --2- 172.21.15.174:0/379604400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dd4100bd0 0x7f3dd4100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:57.178 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.158+0000 7f3dd95ec700 1 -- 172.21.15.174:0/379604400 >> 172.21.15.174:0/379604400 conn(0x7f3dd40fc170 msgr2=0x7f3dd40fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:57.178 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.158+0000 7f3dd95ec700 1 -- 172.21.15.174:0/379604400 shutdown_connections 2024-02-19T18:58:57.178 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.158+0000 7f3dd95ec700 1 -- 172.21.15.174:0/379604400 wait complete. 2024-02-19T18:58:57.178 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.158+0000 7f3dd95ec700 1 Processor -- start 2024-02-19T18:58:57.178 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.158+0000 7f3dd95ec700 1 -- start start 2024-02-19T18:58:57.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.158+0000 7f3dd95ec700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dd4100bd0 0x7f3dd4196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:57.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.158+0000 7f3dd95ec700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dd4101dd0 0x7f3dd4196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:57.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.158+0000 7f3dd95ec700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd4196e90 con 0x7f3dd4100bd0 2024-02-19T18:58:57.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.158+0000 7f3dd95ec700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd4196fd0 con 0x7f3dd4101dd0 2024-02-19T18:58:57.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.159+0000 7f3dd27fc700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dd4101dd0 0x7f3dd4196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:57.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.159+0000 7f3dd27fc700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dd4101dd0 0x7f3dd4196870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:52646/0 (socket says 172.21.15.174:52646) 2024-02-19T18:58:57.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.159+0000 7f3dd27fc700 1 -- 172.21.15.174:0/3937771761 learned_addr learned my addr 172.21.15.174:0/3937771761 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-02-19T18:58:57.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.159+0000 7f3dd2ffd700 1 --2- 172.21.15.174:0/3937771761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dd4100bd0 0x7f3dd4196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:57.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.159+0000 7f3dd27fc700 1 -- 172.21.15.174:0/3937771761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dd4100bd0 msgr2=0x7f3dd4196330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:57.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.159+0000 7f3dd27fc700 1 --2- 172.21.15.174:0/3937771761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dd4100bd0 0x7f3dd4196330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:57.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.159+0000 7f3dd27fc700 1 -- 172.21.15.174:0/3937771761 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc80095f0 con 0x7f3dd4101dd0 2024-02-19T18:58:57.179 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.159+0000 7f3dd2ffd700 1 --2- 172.21.15.174:0/3937771761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dd4100bd0 0x7f3dd4196330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T18:58:57.180 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.159+0000 7f3dd27fc700 1 --2- 172.21.15.174:0/3937771761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dd4101dd0 0x7f3dd4196870 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8006010 tx=0x7f3dc800be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:57.180 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.159+0000 7f3dbbfff700 1 -- 172.21.15.174:0/3937771761 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dc801d070 con 0x7f3dd4101dd0 2024-02-19T18:58:57.180 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.159+0000 7f3dbbfff700 1 -- 172.21.15.174:0/3937771761 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3dc800f460 con 0x7f3dd4101dd0 2024-02-19T18:58:57.180 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.159+0000 7f3dd95ec700 1 -- 172.21.15.174:0/3937771761 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd419ba20 con 0x7f3dd4101dd0 2024-02-19T18:58:57.180 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.159+0000 7f3dbbfff700 1 -- 172.21.15.174:0/3937771761 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dc8021640 con 0x7f3dd4101dd0 2024-02-19T18:58:57.180 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.159+0000 7f3dd95ec700 1 -- 172.21.15.174:0/3937771761 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd419bf10 con 0x7f3dd4101dd0 2024-02-19T18:58:57.181 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.160+0000 7f3dd95ec700 1 -- 172.21.15.174:0/3937771761 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd4109730 con 0x7f3dd4101dd0 2024-02-19T18:58:57.181 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.161+0000 7f3dbbfff700 1 -- 172.21.15.174:0/3937771761 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3dc800fab0 con 0x7f3dd4101dd0 2024-02-19T18:58:57.181 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.161+0000 7f3dbbfff700 1 --2- 172.21.15.174:0/3937771761 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f3dbc06c4e0 0x7f3dbc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:57.181 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.161+0000 7f3dbbfff700 1 -- 172.21.15.174:0/3937771761 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3dc808c8f0 con 0x7f3dd4101dd0 2024-02-19T18:58:57.181 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.161+0000 7f3dd2ffd700 1 --2- 172.21.15.174:0/3937771761 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f3dbc06c4e0 0x7f3dbc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:57.181 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.162+0000 7f3dd2ffd700 1 --2- 172.21.15.174:0/3937771761 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f3dbc06c4e0 0x7f3dbc06e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4101c30 tx=0x7f3dc4005f90 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:57.183 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.166+0000 7f3dbbfff700 1 -- 172.21.15.174:0/3937771761 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3dc805b040 con 0x7f3dd4101dd0 2024-02-19T18:58:57.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.425+0000 7f3dd95ec700 1 -- 172.21.15.174:0/3937771761 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f3dd419c8f0 con 0x7f3dd4101dd0 2024-02-19T18:58:57.448 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.430+0000 7f3dbbfff700 1 -- 172.21.15.174:0/3937771761 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f3dc8026050 con 0x7f3dd4101dd0 2024-02-19T18:58:57.448 INFO:teuthology.orchestra.run.smithi174.stdout:[client.1] 2024-02-19T18:58:57.448 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDxpNNlJCl9GRAAOVJSQu2kleL7msoQWc3S/w== 2024-02-19T18:58:57.450 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.432+0000 7f3dd95ec700 1 -- 172.21.15.174:0/3937771761 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f3dbc06c4e0 msgr2=0x7f3dbc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:57.451 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.432+0000 7f3dd95ec700 1 --2- 172.21.15.174:0/3937771761 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f3dbc06c4e0 0x7f3dbc06e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4101c30 tx=0x7f3dc4005f90 comp rx=0 tx=0).stop 2024-02-19T18:58:57.451 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.432+0000 7f3dd95ec700 1 -- 172.21.15.174:0/3937771761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dd4101dd0 msgr2=0x7f3dd4196870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:57.451 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.432+0000 7f3dd95ec700 1 --2- 172.21.15.174:0/3937771761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dd4101dd0 0x7f3dd4196870 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8006010 tx=0x7f3dc800be20 comp rx=0 tx=0).stop 2024-02-19T18:58:57.451 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.432+0000 7f3dd95ec700 1 -- 172.21.15.174:0/3937771761 shutdown_connections 2024-02-19T18:58:57.451 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.432+0000 7f3dd95ec700 1 --2- 172.21.15.174:0/3937771761 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f3dbc06c4e0 0x7f3dbc06e9a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:57.451 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.432+0000 7f3dd95ec700 1 --2- 172.21.15.174:0/3937771761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dd4100bd0 0x7f3dd4196330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:57.451 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.432+0000 7f3dd95ec700 1 --2- 172.21.15.174:0/3937771761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dd4101dd0 0x7f3dd4196870 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:57.451 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.432+0000 7f3dd95ec700 1 -- 172.21.15.174:0/3937771761 >> 172.21.15.174:0/3937771761 conn(0x7f3dd40fc170 msgr2=0x7f3dd4105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:57.451 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.432+0000 7f3dd95ec700 1 -- 172.21.15.174:0/3937771761 shutdown_connections 2024-02-19T18:58:57.451 INFO:teuthology.orchestra.run.smithi174.stderr:2024-02-19T18:58:57.432+0000 7f3dd95ec700 1 -- 172.21.15.174:0/3937771761 wait complete. 2024-02-19T18:58:58.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:57 smithi174 ceph-mon[41061]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:58:58.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:57 smithi174 ceph-mon[41061]: from='client.? 172.21.15.174:0/3937771761' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T18:58:58.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:57 smithi174 ceph-mon[41061]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T18:58:58.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:57 smithi174 ceph-mon[41061]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T18:58:58.088 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-02-19T18:58:58.088 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-19T18:58:58.088 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-19T18:58:58.172 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-19T18:58:58.172 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-19T18:58:58.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph mgr dump --format=json 2024-02-19T18:58:58.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:57 smithi114 ceph-mon[29441]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:58:58.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:57 smithi114 ceph-mon[29441]: from='client.? 172.21.15.174:0/3937771761' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T18:58:58.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:57 smithi114 ceph-mon[29441]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T18:58:58.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:57 smithi114 ceph-mon[29441]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T18:58:58.391 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:58:59.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.924+0000 7ff73d408700 1 -- 172.21.15.114:0/2238146082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff738105a40 msgr2=0x7ff738105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:59.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.924+0000 7ff73d408700 1 --2- 172.21.15.114:0/2238146082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff738105a40 0x7ff738105e20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff730009910 tx=0x7ff730009c20 comp rx=0 tx=0).stop 2024-02-19T18:58:59.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.924+0000 7ff73d408700 1 -- 172.21.15.114:0/2238146082 shutdown_connections 2024-02-19T18:58:59.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.924+0000 7ff73d408700 1 --2- 172.21.15.114:0/2238146082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7380ffa40 0x7ff7380ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:59.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.924+0000 7ff73d408700 1 --2- 172.21.15.114:0/2238146082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff738105a40 0x7ff738105e20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:59.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.924+0000 7ff73d408700 1 -- 172.21.15.114:0/2238146082 >> 172.21.15.114:0/2238146082 conn(0x7ff7380fb550 msgr2=0x7ff7380fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:59.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.924+0000 7ff73d408700 1 -- 172.21.15.114:0/2238146082 shutdown_connections 2024-02-19T18:58:59.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.924+0000 7ff73d408700 1 -- 172.21.15.114:0/2238146082 wait complete. 2024-02-19T18:58:59.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.924+0000 7ff73d408700 1 Processor -- start 2024-02-19T18:58:59.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.925+0000 7ff73d408700 1 -- start start 2024-02-19T18:58:59.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.925+0000 7ff73d408700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7380ffa40 0x7ff738197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:59.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.925+0000 7ff73d408700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff738105a40 0x7ff738197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:59.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.925+0000 7ff73d408700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff738198180 con 0x7ff738105a40 2024-02-19T18:58:59.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.925+0000 7ff73d408700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7381982c0 con 0x7ff7380ffa40 2024-02-19T18:58:59.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.925+0000 7ff7367fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff738105a40 0x7ff738197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:59.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.925+0000 7ff7367fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff738105a40 0x7ff738197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57544/0 (socket says 172.21.15.114:57544) 2024-02-19T18:58:59.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.925+0000 7ff7367fc700 1 -- 172.21.15.114:0/1783886214 learned_addr learned my addr 172.21.15.114:0/1783886214 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:58:59.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.925+0000 7ff736ffd700 1 --2- 172.21.15.114:0/1783886214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7380ffa40 0x7ff738197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:59.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.925+0000 7ff7367fc700 1 -- 172.21.15.114:0/1783886214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7380ffa40 msgr2=0x7ff738197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:59.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.925+0000 7ff7367fc700 1 --2- 172.21.15.114:0/1783886214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7380ffa40 0x7ff738197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:59.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.925+0000 7ff7367fc700 1 -- 172.21.15.114:0/1783886214 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7300095f0 con 0x7ff738105a40 2024-02-19T18:58:59.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.925+0000 7ff7367fc700 1 --2- 172.21.15.114:0/1783886214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff738105a40 0x7ff738197b60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff72c00ba50 tx=0x7ff72c00be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:59.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.926+0000 7ff71ffff700 1 -- 172.21.15.114:0/1783886214 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff72c00ca00 con 0x7ff738105a40 2024-02-19T18:58:59.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.926+0000 7ff73d408700 1 -- 172.21.15.114:0/1783886214 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7381a0690 con 0x7ff738105a40 2024-02-19T18:58:59.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.926+0000 7ff71ffff700 1 -- 172.21.15.114:0/1783886214 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff72c00cb60 con 0x7ff738105a40 2024-02-19T18:58:59.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.926+0000 7ff71ffff700 1 -- 172.21.15.114:0/1783886214 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff72c007580 con 0x7ff738105a40 2024-02-19T18:58:59.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.926+0000 7ff73d408700 1 -- 172.21.15.114:0/1783886214 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7381a0b50 con 0x7ff738105a40 2024-02-19T18:58:59.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.927+0000 7ff73d408700 1 -- 172.21.15.114:0/1783886214 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff738108040 con 0x7ff738105a40 2024-02-19T18:58:59.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.927+0000 7ff71ffff700 1 -- 172.21.15.114:0/1783886214 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ff72c00ccd0 con 0x7ff738105a40 2024-02-19T18:58:59.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.928+0000 7ff71ffff700 1 --2- 172.21.15.114:0/1783886214 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff72006c4e0 0x7ff72006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:59.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.928+0000 7ff71ffff700 1 -- 172.21.15.114:0/1783886214 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff72c006e40 con 0x7ff738105a40 2024-02-19T18:58:59.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.928+0000 7ff736ffd700 1 --2- 172.21.15.114:0/1783886214 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff72006c4e0 0x7ff72006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:59.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.928+0000 7ff736ffd700 1 --2- 172.21.15.114:0/1783886214 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff72006c4e0 0x7ff72006e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff73000b5c0 tx=0x7ff73001a040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:59.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:58:59.932+0000 7ff71ffff700 1 -- 172.21.15.114:0/1783886214 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff72c0592e0 con 0x7ff738105a40 2024-02-19T18:59:00.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:58:59 smithi114 ceph-mon[29441]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:00.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:00.176+0000 7ff73d408700 1 -- 172.21.15.114:0/1783886214 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ff738107a20 con 0x7ff738105a40 2024-02-19T18:59:00.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:00.180+0000 7ff71ffff700 1 -- 172.21.15.114:0/1783886214 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+173048 (secure 0 0 0) 0x7ff72c058e70 con 0x7ff738105a40 2024-02-19T18:59:00.201 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:59:00.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:00.182+0000 7ff73d408700 1 -- 172.21.15.114:0/1783886214 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff72006c4e0 msgr2=0x7ff72006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:00.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:00.182+0000 7ff73d408700 1 --2- 172.21.15.114:0/1783886214 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff72006c4e0 0x7ff72006e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff73000b5c0 tx=0x7ff73001a040 comp rx=0 tx=0).stop 2024-02-19T18:59:00.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:00.182+0000 7ff73d408700 1 -- 172.21.15.114:0/1783886214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff738105a40 msgr2=0x7ff738197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:00.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:00.182+0000 7ff73d408700 1 --2- 172.21.15.114:0/1783886214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff738105a40 0x7ff738197b60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff72c00ba50 tx=0x7ff72c00be10 comp rx=0 tx=0).stop 2024-02-19T18:59:00.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:00.182+0000 7ff73d408700 1 -- 172.21.15.114:0/1783886214 shutdown_connections 2024-02-19T18:59:00.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:00.182+0000 7ff73d408700 1 --2- 172.21.15.114:0/1783886214 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff72006c4e0 0x7ff72006e9a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:00.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:00.182+0000 7ff73d408700 1 --2- 172.21.15.114:0/1783886214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7380ffa40 0x7ff738197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:00.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:00.182+0000 7ff73d408700 1 --2- 172.21.15.114:0/1783886214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff738105a40 0x7ff738197b60 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:00.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:00.182+0000 7ff73d408700 1 -- 172.21.15.114:0/1783886214 >> 172.21.15.114:0/1783886214 conn(0x7ff7380fb550 msgr2=0x7ff7380fbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:00.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:00.183+0000 7ff73d408700 1 -- 172.21.15.114:0/1783886214 shutdown_connections 2024-02-19T18:59:00.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:00.183+0000 7ff73d408700 1 -- 172.21.15.114:0/1783886214 wait complete. 2024-02-19T18:59:00.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:58:59 smithi174 ceph-mon[41061]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:00.892 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":19,"active_gid":14257,"active_name":"smithi114.pcvtrk","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2188833308},{"type":"v1","addr":"172.21.15.114:6801","nonce":2188833308}]},"active_addr":"172.21.15.114:6801/2188833308","active_change":"2024-02-19T18:55:02.606062+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14284,"name":"smithi174.lydvnj","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.114:8443/","prometheus":"http://172.21.15.114:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":4052272325}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":4017451897}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":4180396322}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":3567716113}]}]} 2024-02-19T18:59:00.896 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-19T18:59:00.896 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-19T18:59:00.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-19T18:59:01.106 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:01.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:00 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1783886214' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-19T18:59:01.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:00 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1783886214' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-19T18:59:02.134 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:01 smithi114 ceph-mon[29441]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:02.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:01 smithi174 ceph-mon[41061]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:02.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.627+0000 7f633c7dc700 1 -- 172.21.15.114:0/1517999932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6334101d70 msgr2=0x7f6334102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:02.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.627+0000 7f633c7dc700 1 --2- 172.21.15.114:0/1517999932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6334101d70 0x7f6334102150 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6330009910 tx=0x7f6330009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:02.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.627+0000 7f633c7dc700 1 -- 172.21.15.114:0/1517999932 shutdown_connections 2024-02-19T18:59:02.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.627+0000 7f633c7dc700 1 --2- 172.21.15.114:0/1517999932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6334102690 0x7f633410ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:02.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.627+0000 7f633c7dc700 1 --2- 172.21.15.114:0/1517999932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6334101d70 0x7f6334102150 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:02.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.627+0000 7f633c7dc700 1 -- 172.21.15.114:0/1517999932 >> 172.21.15.114:0/1517999932 conn(0x7f63340fb540 msgr2=0x7f63340fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:02.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.628+0000 7f633c7dc700 1 -- 172.21.15.114:0/1517999932 shutdown_connections 2024-02-19T18:59:02.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.628+0000 7f633c7dc700 1 -- 172.21.15.114:0/1517999932 wait complete. 2024-02-19T18:59:02.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.628+0000 7f633c7dc700 1 Processor -- start 2024-02-19T18:59:02.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.628+0000 7f633c7dc700 1 -- start start 2024-02-19T18:59:02.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.628+0000 7f633c7dc700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6334101d70 0x7f63340ff9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:02.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.628+0000 7f633c7dc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6334102690 0x7f63340fff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:02.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.628+0000 7f633c7dc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6334100450 con 0x7f6334102690 2024-02-19T18:59:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.628+0000 7f633c7dc700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6334100590 con 0x7f6334101d70 2024-02-19T18:59:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.628+0000 7f6339d77700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6334102690 0x7f63340fff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.628+0000 7f6339d77700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6334102690 0x7f63340fff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57558/0 (socket says 172.21.15.114:57558) 2024-02-19T18:59:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.628+0000 7f6339d77700 1 -- 172.21.15.114:0/332527775 learned_addr learned my addr 172.21.15.114:0/332527775 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.629+0000 7f633a578700 1 --2- 172.21.15.114:0/332527775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6334101d70 0x7f63340ff9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.629+0000 7f6339d77700 1 -- 172.21.15.114:0/332527775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6334101d70 msgr2=0x7f63340ff9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.629+0000 7f6339d77700 1 --2- 172.21.15.114:0/332527775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6334101d70 0x7f63340ff9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.629+0000 7f6339d77700 1 -- 172.21.15.114:0/332527775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63300095f0 con 0x7f6334102690 2024-02-19T18:59:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.629+0000 7f633a578700 1 --2- 172.21.15.114:0/332527775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6334101d70 0x7f63340ff9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T18:59:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.629+0000 7f6339d77700 1 --2- 172.21.15.114:0/332527775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6334102690 0x7f63340fff10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f632400b480 tx=0x7f632400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.629+0000 7f632b7fe700 1 -- 172.21.15.114:0/332527775 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63240118c0 con 0x7f6334102690 2024-02-19T18:59:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.629+0000 7f632b7fe700 1 -- 172.21.15.114:0/332527775 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6324010380 con 0x7f6334102690 2024-02-19T18:59:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.629+0000 7f633c7dc700 1 -- 172.21.15.114:0/332527775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63341a2980 con 0x7f6334102690 2024-02-19T18:59:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.629+0000 7f632b7fe700 1 -- 172.21.15.114:0/332527775 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f632400f4f0 con 0x7f6334102690 2024-02-19T18:59:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.629+0000 7f633c7dc700 1 -- 172.21.15.114:0/332527775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63341a2d90 con 0x7f6334102690 2024-02-19T18:59:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.630+0000 7f633c7dc700 1 -- 172.21.15.114:0/332527775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6334108250 con 0x7f6334102690 2024-02-19T18:59:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.631+0000 7f632b7fe700 1 -- 172.21.15.114:0/332527775 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f63240104f0 con 0x7f6334102690 2024-02-19T18:59:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.631+0000 7f632b7fe700 1 --2- 172.21.15.114:0/332527775 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f632006c4e0 0x7f632006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.631+0000 7f632b7fe700 1 -- 172.21.15.114:0/332527775 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f632408b130 con 0x7f6334102690 2024-02-19T18:59:02.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.631+0000 7f633a578700 1 --2- 172.21.15.114:0/332527775 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f632006c4e0 0x7f632006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:02.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.631+0000 7f633a578700 1 --2- 172.21.15.114:0/332527775 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f632006c4e0 0x7f632006e9a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f633000b5c0 tx=0x7f633001a040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:02.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.635+0000 7f632b7fe700 1 -- 172.21.15.114:0/332527775 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6324055c50 con 0x7f6334102690 2024-02-19T18:59:02.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.828+0000 7f633c7dc700 1 -- 172.21.15.114:0/332527775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6334100750 con 0x7f6334102690 2024-02-19T18:59:02.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.828+0000 7f632b7fe700 1 -- 172.21.15.114:0/332527775 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11370 (secure 0 0 0) 0x7f6324059270 con 0x7f6334102690 2024-02-19T18:59:02.847 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:59:02.847 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":41,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","created":"2024-02-19T18:51:33.283071+0000","modified":"2024-02-19T18:58:47.142333+0000","last_up_change":"2024-02-19T18:58:46.137044+0000","last_in_change":"2024-02-19T18:58:26.384214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T18:57:30.943324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"93a2d632-7629-4a15-b026-93b40fba59d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2204928688},{"type":"v1","addr":"172.21.15.114:6803","nonce":2204928688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2204928688},{"type":"v1","addr":"172.21.15.114:6805","nonce":2204928688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2204928688},{"type":"v1","addr":"172.21.15.114:6809","nonce":2204928688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2204928688},{"type":"v1","addr":"172.21.15.114:6807","nonce":2204928688}]},"public_addr":"172.21.15.114:6803/2204928688","cluster_addr":"172.21.15.114:6805/2204928688","heartbeat_back_addr":"172.21.15.114:6809/2204928688","heartbeat_front_addr":"172.21.15.114:6807/2204928688","state":["exists","up"]},{"osd":1,"uuid":"8334d634-9c49-4614-88e0-ca4545fa761b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":828515125},{"type":"v1","addr":"172.21.15.114:6811","nonce":828515125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":828515125},{"type":"v1","addr":"172.21.15.114:6813","nonce":828515125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":828515125},{"type":"v1","addr":"172.21.15.114:6817","nonce":828515125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":828515125},{"type":"v1","addr":"172.21.15.114:6815","nonce":828515125}]},"public_addr":"172.21.15.114:6811/828515125","cluster_addr":"172.21.15.114:6813/828515125","heartbeat_back_addr":"172.21.15.114:6817/828515125","heartbeat_front_addr":"172.21.15.114:6815/828515125","state":["exists","up"]},{"osd":2,"uuid":"2d27485a-1fb1-432d-a5a4-f46c8031a6ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":680148196},{"type":"v1","addr":"172.21.15.114:6819","nonce":680148196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":680148196},{"type":"v1","addr":"172.21.15.114:6821","nonce":680148196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":680148196},{"type":"v1","addr":"172.21.15.114:6825","nonce":680148196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":680148196},{"type":"v1","addr":"172.21.15.114:6823","nonce":680148196}]},"public_addr":"172.21.15.114:6819/680148196","cluster_addr":"172.21.15.114:6821/680148196","heartbeat_back_addr":"172.21.15.114:6825/680148196","heartbeat_front_addr":"172.21.15.114:6823/680148196","state":["exists","up"]},{"osd":3,"uuid":"fdbbfdf3-8e5c-472c-9894-97741a049a31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":2070964484},{"type":"v1","addr":"172.21.15.174:6801","nonce":2070964484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":2070964484},{"type":"v1","addr":"172.21.15.174:6803","nonce":2070964484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":2070964484},{"type":"v1","addr":"172.21.15.174:6807","nonce":2070964484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":2070964484},{"type":"v1","addr":"172.21.15.174:6805","nonce":2070964484}]},"public_addr":"172.21.15.174:6801/2070964484","cluster_addr":"172.21.15.174:6803/2070964484","heartbeat_back_addr":"172.21.15.174:6807/2070964484","heartbeat_front_addr":"172.21.15.174:6805/2070964484","state":["exists","up"]},{"osd":4,"uuid":"ebf7daeb-8e62-4a0d-ab1c-612cd6a8bf07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":2301992532},{"type":"v1","addr":"172.21.15.174:6809","nonce":2301992532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":2301992532},{"type":"v1","addr":"172.21.15.174:6811","nonce":2301992532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":2301992532},{"type":"v1","addr":"172.21.15.174:6815","nonce":2301992532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":2301992532},{"type":"v1","addr":"172.21.15.174:6813","nonce":2301992532}]},"public_addr":"172.21.15.174:6809/2301992532","cluster_addr":"172.21.15.174:6811/2301992532","heartbeat_back_addr":"172.21.15.174:6815/2301992532","heartbeat_front_addr":"172.21.15.174:6813/2301992532","state":["exists","up"]},{"osd":5,"uuid":"9a8599a6-778b-4367-bfb1-be184ee9d468","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1280361687},{"type":"v1","addr":"172.21.15.174:6817","nonce":1280361687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1280361687},{"type":"v1","addr":"172.21.15.174:6819","nonce":1280361687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1280361687},{"type":"v1","addr":"172.21.15.174:6823","nonce":1280361687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1280361687},{"type":"v1","addr":"172.21.15.174:6821","nonce":1280361687}]},"public_addr":"172.21.15.174:6817/1280361687","cluster_addr":"172.21.15.174:6819/1280361687","heartbeat_back_addr":"172.21.15.174:6823/1280361687","heartbeat_front_addr":"172.21.15.174:6821/1280361687","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:56:27.015050+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:56:56.239571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:57:26.893375+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:57:50.935953+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:16.875224+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:44.453528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/1382855124":"2024-02-20T18:55:02.605941+0000","172.21.15.114:0/3946301125":"2024-02-20T18:55:02.605941+0000","172.21.15.114:6801/1719431365":"2024-02-20T18:55:02.605941+0000","172.21.15.114:0/110036562":"2024-02-20T18:55:02.605941+0000","172.21.15.114:6801/971741305":"2024-02-20T18:52:38.087813+0000","172.21.15.114:6800/1719431365":"2024-02-20T18:55:02.605941+0000","172.21.15.114:0/857878097":"2024-02-20T18:52:38.087813+0000","172.21.15.114:6800/744561179":"2024-02-20T18:52:05.548945+0000","172.21.15.114:6800/971741305":"2024-02-20T18:52:38.087813+0000","172.21.15.114:6801/744561179":"2024-02-20T18:52:05.548945+0000","172.21.15.114:0/1056805783":"2024-02-20T18:52:05.548945+0000","172.21.15.114:0/945825118":"2024-02-20T18:52:38.087813+0000","172.21.15.114:0/169338133":"2024-02-20T18:52:05.548945+0000","172.21.15.114:0/1245656641":"2024-02-20T18:52:05.548945+0000","172.21.15.114:0/2533620288":"2024-02-20T18:52:38.087813+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T18:59:02.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.830+0000 7f633c7dc700 1 -- 172.21.15.114:0/332527775 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f632006c4e0 msgr2=0x7f632006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:02.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.830+0000 7f633c7dc700 1 --2- 172.21.15.114:0/332527775 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f632006c4e0 0x7f632006e9a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f633000b5c0 tx=0x7f633001a040 comp rx=0 tx=0).stop 2024-02-19T18:59:02.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.831+0000 7f633c7dc700 1 -- 172.21.15.114:0/332527775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6334102690 msgr2=0x7f63340fff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:02.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.831+0000 7f633c7dc700 1 --2- 172.21.15.114:0/332527775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6334102690 0x7f63340fff10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f632400b480 tx=0x7f632400b840 comp rx=0 tx=0).stop 2024-02-19T18:59:02.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.831+0000 7f633c7dc700 1 -- 172.21.15.114:0/332527775 shutdown_connections 2024-02-19T18:59:02.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.831+0000 7f633c7dc700 1 --2- 172.21.15.114:0/332527775 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f632006c4e0 0x7f632006e9a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:02.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.831+0000 7f633c7dc700 1 --2- 172.21.15.114:0/332527775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6334101d70 0x7f63340ff9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:02.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.831+0000 7f633c7dc700 1 --2- 172.21.15.114:0/332527775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6334102690 0x7f63340fff10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:02.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.831+0000 7f633c7dc700 1 -- 172.21.15.114:0/332527775 >> 172.21.15.114:0/332527775 conn(0x7f63340fb540 msgr2=0x7f6334105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:02.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.831+0000 7f633c7dc700 1 -- 172.21.15.114:0/332527775 shutdown_connections 2024-02-19T18:59:02.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:02.831+0000 7f633c7dc700 1 -- 172.21.15.114:0/332527775 wait complete. 2024-02-19T18:59:03.107 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:02 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:03.107 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:02 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/332527775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T18:59:03.306 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:02 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:03.307 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:02 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/332527775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T18:59:03.479 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-19T18:59:03.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-19T18:59:03.693 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:04.108 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:03 smithi114 ceph-mon[29441]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:04.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:03 smithi174 ceph-mon[41061]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:05.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.375+0000 7fb813ff6700 1 -- 172.21.15.114:0/2296860262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb80c0ffbd0 msgr2=0x7fb80c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:05.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.375+0000 7fb813ff6700 1 --2- 172.21.15.114:0/2296860262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb80c0ffbd0 0x7fb80c100050 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb80000b0a0 tx=0x7fb80000b3b0 comp rx=0 tx=0).stop 2024-02-19T18:59:05.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.375+0000 7fb813ff6700 1 -- 172.21.15.114:0/2296860262 shutdown_connections 2024-02-19T18:59:05.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.375+0000 7fb813ff6700 1 --2- 172.21.15.114:0/2296860262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb80c0ffbd0 0x7fb80c100050 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:05.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.375+0000 7fb813ff6700 1 --2- 172.21.15.114:0/2296860262 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb80c105a40 0x7fb80c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:05.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.375+0000 7fb813ff6700 1 -- 172.21.15.114:0/2296860262 >> 172.21.15.114:0/2296860262 conn(0x7fb80c0755a0 msgr2=0x7fb80c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:05.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.375+0000 7fb813ff6700 1 -- 172.21.15.114:0/2296860262 shutdown_connections 2024-02-19T18:59:05.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.375+0000 7fb813ff6700 1 -- 172.21.15.114:0/2296860262 wait complete. 2024-02-19T18:59:05.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.376+0000 7fb813ff6700 1 Processor -- start 2024-02-19T18:59:05.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.376+0000 7fb813ff6700 1 -- start start 2024-02-19T18:59:05.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.376+0000 7fb813ff6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb80c0ffbd0 0x7fb80c101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:05.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.376+0000 7fb813ff6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb80c105a40 0x7fb80c101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:05.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.376+0000 7fb813ff6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c102090 con 0x7fb80c105a40 2024-02-19T18:59:05.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.376+0000 7fb813ff6700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c1021d0 con 0x7fb80c0ffbd0 2024-02-19T18:59:05.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.376+0000 7fb811591700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb80c105a40 0x7fb80c101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:05.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.376+0000 7fb811591700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb80c105a40 0x7fb80c101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57576/0 (socket says 172.21.15.114:57576) 2024-02-19T18:59:05.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.376+0000 7fb811591700 1 -- 172.21.15.114:0/949180749 learned_addr learned my addr 172.21.15.114:0/949180749 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:05.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.376+0000 7fb811d92700 1 --2- 172.21.15.114:0/949180749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb80c0ffbd0 0x7fb80c101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:05.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.376+0000 7fb811591700 1 -- 172.21.15.114:0/949180749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb80c0ffbd0 msgr2=0x7fb80c101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:05.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.376+0000 7fb811591700 1 --2- 172.21.15.114:0/949180749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb80c0ffbd0 0x7fb80c101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:05.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.376+0000 7fb811591700 1 -- 172.21.15.114:0/949180749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb800009d00 con 0x7fb80c105a40 2024-02-19T18:59:05.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.376+0000 7fb811d92700 1 --2- 172.21.15.114:0/949180749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb80c0ffbd0 0x7fb80c101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T18:59:05.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.377+0000 7fb811591700 1 --2- 172.21.15.114:0/949180749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb80c105a40 0x7fb80c101b50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb800000f80 tx=0x7fb80000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:05.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.377+0000 7fb80affd700 1 -- 172.21.15.114:0/949180749 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb80000e040 con 0x7fb80c105a40 2024-02-19T18:59:05.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.377+0000 7fb80affd700 1 -- 172.21.15.114:0/949180749 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb8000044f0 con 0x7fb80c105a40 2024-02-19T18:59:05.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.377+0000 7fb813ff6700 1 -- 172.21.15.114:0/949180749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80c102450 con 0x7fb80c105a40 2024-02-19T18:59:05.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.377+0000 7fb80affd700 1 -- 172.21.15.114:0/949180749 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb80001d9f0 con 0x7fb80c105a40 2024-02-19T18:59:05.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.377+0000 7fb813ff6700 1 -- 172.21.15.114:0/949180749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80c1a2d60 con 0x7fb80c105a40 2024-02-19T18:59:05.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.378+0000 7fb813ff6700 1 -- 172.21.15.114:0/949180749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb80c109070 con 0x7fb80c105a40 2024-02-19T18:59:05.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.378+0000 7fb80affd700 1 -- 172.21.15.114:0/949180749 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fb800004660 con 0x7fb80c105a40 2024-02-19T18:59:05.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.379+0000 7fb80affd700 1 --2- 172.21.15.114:0/949180749 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb7f406c4e0 0x7fb7f406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:05.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.379+0000 7fb811d92700 1 --2- 172.21.15.114:0/949180749 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb7f406c4e0 0x7fb7f406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:05.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.379+0000 7fb80affd700 1 -- 172.21.15.114:0/949180749 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb80008cf20 con 0x7fb80c105a40 2024-02-19T18:59:05.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.379+0000 7fb811d92700 1 --2- 172.21.15.114:0/949180749 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb7f406c4e0 0x7fb7f406e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb804007900 tx=0x7fb804008040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:05.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.383+0000 7fb80affd700 1 -- 172.21.15.114:0/949180749 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb8000c00e0 con 0x7fb80c105a40 2024-02-19T18:59:05.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.574+0000 7fb813ff6700 1 -- 172.21.15.114:0/949180749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb80c04ea90 con 0x7fb80c105a40 2024-02-19T18:59:05.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.575+0000 7fb80affd700 1 -- 172.21.15.114:0/949180749 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11370 (secure 0 0 0) 0x7fb800057a40 con 0x7fb80c105a40 2024-02-19T18:59:05.593 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:59:05.593 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":41,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","created":"2024-02-19T18:51:33.283071+0000","modified":"2024-02-19T18:58:47.142333+0000","last_up_change":"2024-02-19T18:58:46.137044+0000","last_in_change":"2024-02-19T18:58:26.384214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T18:57:30.943324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"93a2d632-7629-4a15-b026-93b40fba59d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2204928688},{"type":"v1","addr":"172.21.15.114:6803","nonce":2204928688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2204928688},{"type":"v1","addr":"172.21.15.114:6805","nonce":2204928688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2204928688},{"type":"v1","addr":"172.21.15.114:6809","nonce":2204928688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2204928688},{"type":"v1","addr":"172.21.15.114:6807","nonce":2204928688}]},"public_addr":"172.21.15.114:6803/2204928688","cluster_addr":"172.21.15.114:6805/2204928688","heartbeat_back_addr":"172.21.15.114:6809/2204928688","heartbeat_front_addr":"172.21.15.114:6807/2204928688","state":["exists","up"]},{"osd":1,"uuid":"8334d634-9c49-4614-88e0-ca4545fa761b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":828515125},{"type":"v1","addr":"172.21.15.114:6811","nonce":828515125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":828515125},{"type":"v1","addr":"172.21.15.114:6813","nonce":828515125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":828515125},{"type":"v1","addr":"172.21.15.114:6817","nonce":828515125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":828515125},{"type":"v1","addr":"172.21.15.114:6815","nonce":828515125}]},"public_addr":"172.21.15.114:6811/828515125","cluster_addr":"172.21.15.114:6813/828515125","heartbeat_back_addr":"172.21.15.114:6817/828515125","heartbeat_front_addr":"172.21.15.114:6815/828515125","state":["exists","up"]},{"osd":2,"uuid":"2d27485a-1fb1-432d-a5a4-f46c8031a6ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":680148196},{"type":"v1","addr":"172.21.15.114:6819","nonce":680148196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":680148196},{"type":"v1","addr":"172.21.15.114:6821","nonce":680148196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":680148196},{"type":"v1","addr":"172.21.15.114:6825","nonce":680148196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":680148196},{"type":"v1","addr":"172.21.15.114:6823","nonce":680148196}]},"public_addr":"172.21.15.114:6819/680148196","cluster_addr":"172.21.15.114:6821/680148196","heartbeat_back_addr":"172.21.15.114:6825/680148196","heartbeat_front_addr":"172.21.15.114:6823/680148196","state":["exists","up"]},{"osd":3,"uuid":"fdbbfdf3-8e5c-472c-9894-97741a049a31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":2070964484},{"type":"v1","addr":"172.21.15.174:6801","nonce":2070964484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":2070964484},{"type":"v1","addr":"172.21.15.174:6803","nonce":2070964484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":2070964484},{"type":"v1","addr":"172.21.15.174:6807","nonce":2070964484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":2070964484},{"type":"v1","addr":"172.21.15.174:6805","nonce":2070964484}]},"public_addr":"172.21.15.174:6801/2070964484","cluster_addr":"172.21.15.174:6803/2070964484","heartbeat_back_addr":"172.21.15.174:6807/2070964484","heartbeat_front_addr":"172.21.15.174:6805/2070964484","state":["exists","up"]},{"osd":4,"uuid":"ebf7daeb-8e62-4a0d-ab1c-612cd6a8bf07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":2301992532},{"type":"v1","addr":"172.21.15.174:6809","nonce":2301992532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":2301992532},{"type":"v1","addr":"172.21.15.174:6811","nonce":2301992532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":2301992532},{"type":"v1","addr":"172.21.15.174:6815","nonce":2301992532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":2301992532},{"type":"v1","addr":"172.21.15.174:6813","nonce":2301992532}]},"public_addr":"172.21.15.174:6809/2301992532","cluster_addr":"172.21.15.174:6811/2301992532","heartbeat_back_addr":"172.21.15.174:6815/2301992532","heartbeat_front_addr":"172.21.15.174:6813/2301992532","state":["exists","up"]},{"osd":5,"uuid":"9a8599a6-778b-4367-bfb1-be184ee9d468","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1280361687},{"type":"v1","addr":"172.21.15.174:6817","nonce":1280361687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1280361687},{"type":"v1","addr":"172.21.15.174:6819","nonce":1280361687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1280361687},{"type":"v1","addr":"172.21.15.174:6823","nonce":1280361687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1280361687},{"type":"v1","addr":"172.21.15.174:6821","nonce":1280361687}]},"public_addr":"172.21.15.174:6817/1280361687","cluster_addr":"172.21.15.174:6819/1280361687","heartbeat_back_addr":"172.21.15.174:6823/1280361687","heartbeat_front_addr":"172.21.15.174:6821/1280361687","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:56:27.015050+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:56:56.239571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:57:26.893375+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:57:50.935953+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:16.875224+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:44.453528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/1382855124":"2024-02-20T18:55:02.605941+0000","172.21.15.114:0/3946301125":"2024-02-20T18:55:02.605941+0000","172.21.15.114:6801/1719431365":"2024-02-20T18:55:02.605941+0000","172.21.15.114:0/110036562":"2024-02-20T18:55:02.605941+0000","172.21.15.114:6801/971741305":"2024-02-20T18:52:38.087813+0000","172.21.15.114:6800/1719431365":"2024-02-20T18:55:02.605941+0000","172.21.15.114:0/857878097":"2024-02-20T18:52:38.087813+0000","172.21.15.114:6800/744561179":"2024-02-20T18:52:05.548945+0000","172.21.15.114:6800/971741305":"2024-02-20T18:52:38.087813+0000","172.21.15.114:6801/744561179":"2024-02-20T18:52:05.548945+0000","172.21.15.114:0/1056805783":"2024-02-20T18:52:05.548945+0000","172.21.15.114:0/945825118":"2024-02-20T18:52:38.087813+0000","172.21.15.114:0/169338133":"2024-02-20T18:52:05.548945+0000","172.21.15.114:0/1245656641":"2024-02-20T18:52:05.548945+0000","172.21.15.114:0/2533620288":"2024-02-20T18:52:38.087813+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T18:59:05.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.577+0000 7fb813ff6700 1 -- 172.21.15.114:0/949180749 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb7f406c4e0 msgr2=0x7fb7f406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:05.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.577+0000 7fb813ff6700 1 --2- 172.21.15.114:0/949180749 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb7f406c4e0 0x7fb7f406e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb804007900 tx=0x7fb804008040 comp rx=0 tx=0).stop 2024-02-19T18:59:05.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.577+0000 7fb813ff6700 1 -- 172.21.15.114:0/949180749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb80c105a40 msgr2=0x7fb80c101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:05.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.577+0000 7fb813ff6700 1 --2- 172.21.15.114:0/949180749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb80c105a40 0x7fb80c101b50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb800000f80 tx=0x7fb80000cdb0 comp rx=0 tx=0).stop 2024-02-19T18:59:05.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.577+0000 7fb813ff6700 1 -- 172.21.15.114:0/949180749 shutdown_connections 2024-02-19T18:59:05.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.577+0000 7fb813ff6700 1 --2- 172.21.15.114:0/949180749 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb7f406c4e0 0x7fb7f406e9a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:05.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.577+0000 7fb813ff6700 1 --2- 172.21.15.114:0/949180749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb80c0ffbd0 0x7fb80c101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:05.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.577+0000 7fb813ff6700 1 --2- 172.21.15.114:0/949180749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb80c105a40 0x7fb80c101b50 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:05.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.577+0000 7fb813ff6700 1 -- 172.21.15.114:0/949180749 >> 172.21.15.114:0/949180749 conn(0x7fb80c0755a0 msgr2=0x7fb80c068650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:05.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.577+0000 7fb813ff6700 1 -- 172.21.15.114:0/949180749 shutdown_connections 2024-02-19T18:59:05.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:05.577+0000 7fb813ff6700 1 -- 172.21.15.114:0/949180749 wait complete. 2024-02-19T18:59:06.172 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:05 smithi114 ceph-mon[29441]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:06.172 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:05 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/949180749' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T18:59:06.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-19T18:59:06.197 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-19T18:59:06.197 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-19T18:59:06.198 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-19T18:59:06.198 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-19T18:59:06.199 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-19T18:59:06.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:05 smithi174 ceph-mon[41061]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:06.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:05 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/949180749' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T18:59:06.498 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:06.507 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:06.795 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:06.796 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:07.992 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:07.993 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:08.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:07 smithi114 ceph-mon[29441]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:08.344 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:07 smithi174 ceph-mon[41061]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:09.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:09 smithi114 ceph-mon[29441]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:09.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:09 smithi174 ceph-mon[41061]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:11.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.813+0000 7f04612a4700 1 -- 172.21.15.114:0/156585667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045c100dd0 msgr2=0x7f045c1011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:11.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.813+0000 7f04612a4700 1 --2- 172.21.15.114:0/156585667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045c100dd0 0x7f045c1011b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f044c009910 tx=0x7f044c009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:11.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.813+0000 7f04612a4700 1 -- 172.21.15.114:0/156585667 shutdown_connections 2024-02-19T18:59:11.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.813+0000 7f04612a4700 1 --2- 172.21.15.114:0/156585667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f045c101780 0x7f045c1057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:11.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.813+0000 7f04612a4700 1 --2- 172.21.15.114:0/156585667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045c100dd0 0x7f045c1011b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:11.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.813+0000 7f04612a4700 1 -- 172.21.15.114:0/156585667 >> 172.21.15.114:0/156585667 conn(0x7f045c0fc640 msgr2=0x7f045c0fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:11.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.814+0000 7f04612a4700 1 -- 172.21.15.114:0/156585667 shutdown_connections 2024-02-19T18:59:11.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.814+0000 7f04612a4700 1 -- 172.21.15.114:0/156585667 wait complete. 2024-02-19T18:59:11.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.814+0000 7f04612a4700 1 Processor -- start 2024-02-19T18:59:11.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.814+0000 7f04612a4700 1 -- start start 2024-02-19T18:59:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.814+0000 7f04612a4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f045c100dd0 0x7f045c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.814+0000 7f04612a4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045c101780 0x7f045c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.814+0000 7f04612a4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045c073b10 con 0x7f045c101780 2024-02-19T18:59:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.814+0000 7f04612a4700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045c073c50 con 0x7f045c100dd0 2024-02-19T18:59:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.815+0000 7f045a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045c101780 0x7f045c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.815+0000 7f045a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045c101780 0x7f045c0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46788/0 (socket says 172.21.15.114:46788) 2024-02-19T18:59:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.815+0000 7f045a7fc700 1 -- 172.21.15.114:0/161967368 learned_addr learned my addr 172.21.15.114:0/161967368 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.815+0000 7f045affd700 1 --2- 172.21.15.114:0/161967368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f045c100dd0 0x7f045c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.815+0000 7f045a7fc700 1 -- 172.21.15.114:0/161967368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f045c100dd0 msgr2=0x7f045c073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:11.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.815+0000 7f045a7fc700 1 --2- 172.21.15.114:0/161967368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f045c100dd0 0x7f045c073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.815+0000 7f045a7fc700 1 -- 172.21.15.114:0/161967368 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044c0095f0 con 0x7f045c101780 2024-02-19T18:59:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.815+0000 7f045a7fc700 1 --2- 172.21.15.114:0/161967368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045c101780 0x7f045c0735d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f045000b480 tx=0x7f045000b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.815+0000 7f0443fff700 1 -- 172.21.15.114:0/161967368 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0450011800 con 0x7f045c101780 2024-02-19T18:59:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.815+0000 7f0443fff700 1 -- 172.21.15.114:0/161967368 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0450011e10 con 0x7f045c101780 2024-02-19T18:59:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.816+0000 7f04612a4700 1 -- 172.21.15.114:0/161967368 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f045c1a2a50 con 0x7f045c101780 2024-02-19T18:59:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.816+0000 7f0443fff700 1 -- 172.21.15.114:0/161967368 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0450010ea0 con 0x7f045c101780 2024-02-19T18:59:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.816+0000 7f04612a4700 1 -- 172.21.15.114:0/161967368 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f045c1a2e60 con 0x7f045c101780 2024-02-19T18:59:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.817+0000 7f04612a4700 1 -- 172.21.15.114:0/161967368 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f045c10e110 con 0x7f045c101780 2024-02-19T18:59:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.817+0000 7f0443fff700 1 -- 172.21.15.114:0/161967368 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f0450010380 con 0x7f045c101780 2024-02-19T18:59:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.818+0000 7f0443fff700 1 --2- 172.21.15.114:0/161967368 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f044406c4e0 0x7f044406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.818+0000 7f045affd700 1 --2- 172.21.15.114:0/161967368 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f044406c4e0 0x7f044406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.818+0000 7f0443fff700 1 -- 172.21.15.114:0/161967368 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f045008afe0 con 0x7f045c101780 2024-02-19T18:59:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.818+0000 7f0443fff700 1 --2- 172.21.15.114:0/161967368 >> [v2:172.21.15.174:6808/2301992532,v1:172.21.15.174:6809/2301992532] conn(0x7f04440721c0 0x7f04440745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.818+0000 7f0443fff700 1 -- 172.21.15.114:0/161967368 --> [v2:172.21.15.174:6808/2301992532,v1:172.21.15.174:6809/2301992532] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0444074c90 con 0x7f04440721c0 2024-02-19T18:59:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.818+0000 7f0443fff700 1 -- 172.21.15.114:0/161967368 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f045008d6b0 con 0x7f045c101780 2024-02-19T18:59:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.818+0000 7f045affd700 1 --2- 172.21.15.114:0/161967368 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f044406c4e0 0x7f044406e9a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f044c006010 tx=0x7f044c01a040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.818+0000 7f045b7fe700 1 --2- 172.21.15.114:0/161967368 >> [v2:172.21.15.174:6808/2301992532,v1:172.21.15.174:6809/2301992532] conn(0x7f04440721c0 0x7f04440745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.819+0000 7f045b7fe700 1 --2- 172.21.15.114:0/161967368 >> [v2:172.21.15.174:6808/2301992532,v1:172.21.15.174:6809/2301992532] conn(0x7f04440721c0 0x7f04440745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.820+0000 7f0443fff700 1 -- 172.21.15.114:0/161967368 <== osd.4 v2:172.21.15.174:6808/2301992532 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f0444074c90 con 0x7f04440721c0 2024-02-19T18:59:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.840+0000 7f04612a4700 1 -- 172.21.15.114:0/161967368 --> [v2:172.21.15.174:6808/2301992532,v1:172.21.15.174:6809/2301992532] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f045c04ea90 con 0x7f04440721c0 2024-02-19T18:59:11.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.841+0000 7f0443fff700 1 -- 172.21.15.114:0/161967368 <== osd.4 v2:172.21.15.174:6808/2301992532 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f045c04ea90 con 0x7f04440721c0 2024-02-19T18:59:11.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.841+0000 7f04612a4700 1 -- 172.21.15.114:0/161967368 >> [v2:172.21.15.174:6808/2301992532,v1:172.21.15.174:6809/2301992532] conn(0x7f04440721c0 msgr2=0x7f04440745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:11.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.841+0000 7f04612a4700 1 --2- 172.21.15.114:0/161967368 >> [v2:172.21.15.174:6808/2301992532,v1:172.21.15.174:6809/2301992532] conn(0x7f04440721c0 0x7f04440745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:11.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.841+0000 7f04612a4700 1 -- 172.21.15.114:0/161967368 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f044406c4e0 msgr2=0x7f044406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:11.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.841+0000 7f04612a4700 1 --2- 172.21.15.114:0/161967368 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f044406c4e0 0x7f044406e9a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f044c006010 tx=0x7f044c01a040 comp rx=0 tx=0).stop 2024-02-19T18:59:11.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.841+0000 7f04612a4700 1 -- 172.21.15.114:0/161967368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045c101780 msgr2=0x7f045c0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.841+0000 7f04612a4700 1 --2- 172.21.15.114:0/161967368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045c101780 0x7f045c0735d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f045000b480 tx=0x7f045000b790 comp rx=0 tx=0).stop 2024-02-19T18:59:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.841+0000 7f04612a4700 1 -- 172.21.15.114:0/161967368 shutdown_connections 2024-02-19T18:59:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.841+0000 7f04612a4700 1 --2- 172.21.15.114:0/161967368 >> [v2:172.21.15.174:6808/2301992532,v1:172.21.15.174:6809/2301992532] conn(0x7f04440721c0 0x7f04440745e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.841+0000 7f04612a4700 1 --2- 172.21.15.114:0/161967368 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f044406c4e0 0x7f044406e9a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.841+0000 7f04612a4700 1 --2- 172.21.15.114:0/161967368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f045c100dd0 0x7f045c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.841+0000 7f04612a4700 1 --2- 172.21.15.114:0/161967368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f045c101780 0x7f045c0735d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.841+0000 7f04612a4700 1 -- 172.21.15.114:0/161967368 >> 172.21.15.114:0/161967368 conn(0x7f045c0fc640 msgr2=0x7f045c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.842+0000 7f04612a4700 1 -- 172.21.15.114:0/161967368 shutdown_connections 2024-02-19T18:59:11.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:11.842+0000 7f04612a4700 1 -- 172.21.15.114:0/161967368 wait complete. 2024-02-19T18:59:11.931 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:11 smithi114 ceph-mon[29441]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:12.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:11 smithi174 ceph-mon[41061]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:12.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.466+0000 7f0a9b961700 1 -- 172.21.15.114:0/569910312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a94100df0 msgr2=0x7f0a941011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:12.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.466+0000 7f0a9b961700 1 --2- 172.21.15.114:0/569910312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a94100df0 0x7f0a941011d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0a90009910 tx=0x7f0a90009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:12.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.467+0000 7f0a9b961700 1 -- 172.21.15.114:0/569910312 shutdown_connections 2024-02-19T18:59:12.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.467+0000 7f0a9b961700 1 --2- 172.21.15.114:0/569910312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a941017a0 0x7f0a941057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:12.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.467+0000 7f0a9b961700 1 --2- 172.21.15.114:0/569910312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a94100df0 0x7f0a941011d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:12.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.467+0000 7f0a9b961700 1 -- 172.21.15.114:0/569910312 >> 172.21.15.114:0/569910312 conn(0x7f0a940fc6a0 msgr2=0x7f0a940feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:12.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.467+0000 7f0a9b961700 1 -- 172.21.15.114:0/569910312 shutdown_connections 2024-02-19T18:59:12.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.467+0000 7f0a9b961700 1 -- 172.21.15.114:0/569910312 wait complete. 2024-02-19T18:59:12.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.467+0000 7f0a9b961700 1 Processor -- start 2024-02-19T18:59:12.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.467+0000 7f0a9b961700 1 -- start start 2024-02-19T18:59:12.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.467+0000 7f0a9b961700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a94100df0 0x7f0a94073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:12.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.467+0000 7f0a9b961700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a941017a0 0x7f0a940735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:12.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.467+0000 7f0a9b961700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a94073b10 con 0x7f0a941017a0 2024-02-19T18:59:12.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.467+0000 7f0a9b961700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a94073c50 con 0x7f0a94100df0 2024-02-19T18:59:12.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.468+0000 7f0a98efc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a941017a0 0x7f0a940735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:12.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.468+0000 7f0a98efc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a941017a0 0x7f0a940735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46804/0 (socket says 172.21.15.114:46804) 2024-02-19T18:59:12.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.468+0000 7f0a98efc700 1 -- 172.21.15.114:0/2070561213 learned_addr learned my addr 172.21.15.114:0/2070561213 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:12.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.468+0000 7f0a996fd700 1 --2- 172.21.15.114:0/2070561213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a94100df0 0x7f0a94073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:12.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.468+0000 7f0a98efc700 1 -- 172.21.15.114:0/2070561213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a94100df0 msgr2=0x7f0a94073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:12.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.468+0000 7f0a98efc700 1 --2- 172.21.15.114:0/2070561213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a94100df0 0x7f0a94073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:12.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.468+0000 7f0a98efc700 1 -- 172.21.15.114:0/2070561213 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a900095f0 con 0x7f0a941017a0 2024-02-19T18:59:12.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.468+0000 7f0a98efc700 1 --2- 172.21.15.114:0/2070561213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a941017a0 0x7f0a940735d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400c960 tx=0x7f0a8400cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:12.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.468+0000 7f0a8a7fc700 1 -- 172.21.15.114:0/2070561213 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a84007a70 con 0x7f0a941017a0 2024-02-19T18:59:12.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.468+0000 7f0a9b961700 1 -- 172.21.15.114:0/2070561213 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a941a2930 con 0x7f0a941017a0 2024-02-19T18:59:12.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.468+0000 7f0a8a7fc700 1 -- 172.21.15.114:0/2070561213 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0a84007bd0 con 0x7f0a941017a0 2024-02-19T18:59:12.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.468+0000 7f0a8a7fc700 1 -- 172.21.15.114:0/2070561213 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a84018630 con 0x7f0a941017a0 2024-02-19T18:59:12.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.468+0000 7f0a9b961700 1 -- 172.21.15.114:0/2070561213 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a941a2c90 con 0x7f0a941017a0 2024-02-19T18:59:12.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.469+0000 7f0a9b961700 1 -- 172.21.15.114:0/2070561213 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0a9410e0b0 con 0x7f0a941017a0 2024-02-19T18:59:12.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.470+0000 7f0a8a7fc700 1 -- 172.21.15.114:0/2070561213 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f0a8400fa90 con 0x7f0a941017a0 2024-02-19T18:59:12.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.470+0000 7f0a8a7fc700 1 --2- 172.21.15.114:0/2070561213 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0a8006c4e0 0x7f0a8006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:12.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.470+0000 7f0a8a7fc700 1 -- 172.21.15.114:0/2070561213 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0a8408b860 con 0x7f0a941017a0 2024-02-19T18:59:12.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.470+0000 7f0a996fd700 1 --2- 172.21.15.114:0/2070561213 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0a8006c4e0 0x7f0a8006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:12.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.470+0000 7f0a8a7fc700 1 --2- 172.21.15.114:0/2070561213 >> [v2:172.21.15.174:6816/1280361687,v1:172.21.15.174:6817/1280361687] conn(0x7f0a800721c0 0x7f0a800745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:12.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.470+0000 7f0a8a7fc700 1 -- 172.21.15.114:0/2070561213 --> [v2:172.21.15.174:6816/1280361687,v1:172.21.15.174:6817/1280361687] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0a80074c90 con 0x7f0a800721c0 2024-02-19T18:59:12.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.470+0000 7f0a8a7fc700 1 -- 172.21.15.114:0/2070561213 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a8408bcc0 con 0x7f0a941017a0 2024-02-19T18:59:12.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.471+0000 7f0a99efe700 1 --2- 172.21.15.114:0/2070561213 >> [v2:172.21.15.174:6816/1280361687,v1:172.21.15.174:6817/1280361687] conn(0x7f0a800721c0 0x7f0a800745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:12.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.471+0000 7f0a996fd700 1 --2- 172.21.15.114:0/2070561213 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0a8006c4e0 0x7f0a8006e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000b5c0 tx=0x7f0a9001a040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:12.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.471+0000 7f0a99efe700 1 --2- 172.21.15.114:0/2070561213 >> [v2:172.21.15.174:6816/1280361687,v1:172.21.15.174:6817/1280361687] conn(0x7f0a800721c0 0x7f0a800745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:12.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.472+0000 7f0a8a7fc700 1 -- 172.21.15.114:0/2070561213 <== osd.5 v2:172.21.15.174:6816/1280361687 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f0a80074c90 con 0x7f0a800721c0 2024-02-19T18:59:12.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.487+0000 7f0a9b961700 1 -- 172.21.15.114:0/2070561213 --> [v2:172.21.15.174:6816/1280361687,v1:172.21.15.174:6817/1280361687] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0a9404ea90 con 0x7f0a800721c0 2024-02-19T18:59:12.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.487+0000 7f0a8a7fc700 1 -- 172.21.15.114:0/2070561213 <== osd.5 v2:172.21.15.174:6816/1280361687 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0a9404ea90 con 0x7f0a800721c0 2024-02-19T18:59:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.487+0000 7f0a9b961700 1 -- 172.21.15.114:0/2070561213 >> [v2:172.21.15.174:6816/1280361687,v1:172.21.15.174:6817/1280361687] conn(0x7f0a800721c0 msgr2=0x7f0a800745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.487+0000 7f0a9b961700 1 --2- 172.21.15.114:0/2070561213 >> [v2:172.21.15.174:6816/1280361687,v1:172.21.15.174:6817/1280361687] conn(0x7f0a800721c0 0x7f0a800745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.488+0000 7f0a9b961700 1 -- 172.21.15.114:0/2070561213 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0a8006c4e0 msgr2=0x7f0a8006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.488+0000 7f0a9b961700 1 --2- 172.21.15.114:0/2070561213 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0a8006c4e0 0x7f0a8006e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000b5c0 tx=0x7f0a9001a040 comp rx=0 tx=0).stop 2024-02-19T18:59:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.488+0000 7f0a9b961700 1 -- 172.21.15.114:0/2070561213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a941017a0 msgr2=0x7f0a940735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.488+0000 7f0a9b961700 1 --2- 172.21.15.114:0/2070561213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a941017a0 0x7f0a940735d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400c960 tx=0x7f0a8400cd20 comp rx=0 tx=0).stop 2024-02-19T18:59:12.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.488+0000 7f0a9b961700 1 -- 172.21.15.114:0/2070561213 shutdown_connections 2024-02-19T18:59:12.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.488+0000 7f0a9b961700 1 --2- 172.21.15.114:0/2070561213 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0a8006c4e0 0x7f0a8006e9a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:12.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.488+0000 7f0a9b961700 1 --2- 172.21.15.114:0/2070561213 >> [v2:172.21.15.174:6816/1280361687,v1:172.21.15.174:6817/1280361687] conn(0x7f0a800721c0 0x7f0a800745e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:12.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.488+0000 7f0a9b961700 1 --2- 172.21.15.114:0/2070561213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a94100df0 0x7f0a94073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:12.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.488+0000 7f0a9b961700 1 --2- 172.21.15.114:0/2070561213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a941017a0 0x7f0a940735d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:12.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.488+0000 7f0a9b961700 1 -- 172.21.15.114:0/2070561213 >> 172.21.15.114:0/2070561213 conn(0x7f0a940fc6a0 msgr2=0x7f0a940fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:12.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.488+0000 7f0a9b961700 1 -- 172.21.15.114:0/2070561213 shutdown_connections 2024-02-19T18:59:12.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:12.488+0000 7f0a9b961700 1 -- 172.21.15.114:0/2070561213 wait complete. 2024-02-19T18:59:13.329 INFO:teuthology.orchestra.run.smithi114.stdout:150323855373 2024-02-19T18:59:13.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-19T18:59:13.931 INFO:teuthology.orchestra.run.smithi114.stdout:171798691848 2024-02-19T18:59:13.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-19T18:59:14.162 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:13 smithi114 ceph-mon[29441]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:14.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.156+0000 7fb89b860700 1 -- 172.21.15.114:0/2744737312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb894101750 msgr2=0x7fb8941057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:14.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.156+0000 7fb89b860700 1 --2- 172.21.15.114:0/2744737312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb894101750 0x7fb8941057a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb884009910 tx=0x7fb884009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:14.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.156+0000 7fb89b860700 1 -- 172.21.15.114:0/2744737312 shutdown_connections 2024-02-19T18:59:14.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.156+0000 7fb89b860700 1 --2- 172.21.15.114:0/2744737312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb894101750 0x7fb8941057a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:14.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.156+0000 7fb89b860700 1 --2- 172.21.15.114:0/2744737312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb894100da0 0x7fb894101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:14.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.156+0000 7fb89b860700 1 -- 172.21.15.114:0/2744737312 >> 172.21.15.114:0/2744737312 conn(0x7fb8940fc630 msgr2=0x7fb8940fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:14.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.156+0000 7fb89b860700 1 -- 172.21.15.114:0/2744737312 shutdown_connections 2024-02-19T18:59:14.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.156+0000 7fb89b860700 1 -- 172.21.15.114:0/2744737312 wait complete. 2024-02-19T18:59:14.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.157+0000 7fb89b860700 1 Processor -- start 2024-02-19T18:59:14.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.157+0000 7fb89b860700 1 -- start start 2024-02-19T18:59:14.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.157+0000 7fb89b860700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb894100da0 0x7fb89419c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:14.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.157+0000 7fb89b860700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb894101750 0x7fb89419cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:14.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.157+0000 7fb89b860700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89419d170 con 0x7fb894101750 2024-02-19T18:59:14.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.157+0000 7fb89b860700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb894196620 con 0x7fb894100da0 2024-02-19T18:59:14.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.157+0000 7fb898dfb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb894101750 0x7fb89419cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:14.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.157+0000 7fb898dfb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb894101750 0x7fb89419cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46810/0 (socket says 172.21.15.114:46810) 2024-02-19T18:59:14.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.157+0000 7fb898dfb700 1 -- 172.21.15.114:0/1375269160 learned_addr learned my addr 172.21.15.114:0/1375269160 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:14.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.157+0000 7fb8995fc700 1 --2- 172.21.15.114:0/1375269160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb894100da0 0x7fb89419c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:14.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.157+0000 7fb898dfb700 1 -- 172.21.15.114:0/1375269160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb894100da0 msgr2=0x7fb89419c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:14.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.157+0000 7fb898dfb700 1 --2- 172.21.15.114:0/1375269160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb894100da0 0x7fb89419c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:14.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.157+0000 7fb898dfb700 1 -- 172.21.15.114:0/1375269160 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8840095f0 con 0x7fb894101750 2024-02-19T18:59:14.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.158+0000 7fb898dfb700 1 --2- 172.21.15.114:0/1375269160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb894101750 0x7fb89419cae0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb8840092e0 tx=0x7fb88400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:14.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.158+0000 7fb88a7fc700 1 -- 172.21.15.114:0/1375269160 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb88401d070 con 0x7fb894101750 2024-02-19T18:59:14.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.158+0000 7fb89b860700 1 -- 172.21.15.114:0/1375269160 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8941968a0 con 0x7fb894101750 2024-02-19T18:59:14.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.158+0000 7fb88a7fc700 1 -- 172.21.15.114:0/1375269160 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb884022470 con 0x7fb894101750 2024-02-19T18:59:14.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.158+0000 7fb88a7fc700 1 -- 172.21.15.114:0/1375269160 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb88400f690 con 0x7fb894101750 2024-02-19T18:59:14.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.158+0000 7fb89b860700 1 -- 172.21.15.114:0/1375269160 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb894196d90 con 0x7fb894101750 2024-02-19T18:59:14.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.159+0000 7fb89b860700 1 -- 172.21.15.114:0/1375269160 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb89410e0c0 con 0x7fb894101750 2024-02-19T18:59:14.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.159+0000 7fb88a7fc700 1 -- 172.21.15.114:0/1375269160 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fb8840225e0 con 0x7fb894101750 2024-02-19T18:59:14.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.160+0000 7fb88a7fc700 1 --2- 172.21.15.114:0/1375269160 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb88006c4e0 0x7fb88006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:14.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.160+0000 7fb88a7fc700 1 -- 172.21.15.114:0/1375269160 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb88408d2d0 con 0x7fb894101750 2024-02-19T18:59:14.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.160+0000 7fb8995fc700 1 --2- 172.21.15.114:0/1375269160 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb88006c4e0 0x7fb88006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:14.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.160+0000 7fb88a7fc700 1 --2- 172.21.15.114:0/1375269160 >> [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125] conn(0x7fb8800721c0 0x7fb8800745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:14.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.160+0000 7fb88a7fc700 1 -- 172.21.15.114:0/1375269160 --> [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb880074c90 con 0x7fb8800721c0 2024-02-19T18:59:14.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.160+0000 7fb88a7fc700 1 -- 172.21.15.114:0/1375269160 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb88408d730 con 0x7fb894101750 2024-02-19T18:59:14.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.160+0000 7fb899dfd700 1 --2- 172.21.15.114:0/1375269160 >> [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125] conn(0x7fb8800721c0 0x7fb8800745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:14.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.160+0000 7fb8995fc700 1 --2- 172.21.15.114:0/1375269160 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb88006c4e0 0x7fb88006e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb890006fd0 tx=0x7fb890008040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:14.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.160+0000 7fb899dfd700 1 --2- 172.21.15.114:0/1375269160 >> [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125] conn(0x7fb8800721c0 0x7fb8800745e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:14.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.161+0000 7fb88a7fc700 1 -- 172.21.15.114:0/1375269160 <== osd.1 v2:172.21.15.114:6810/828515125 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb880074c90 con 0x7fb8800721c0 2024-02-19T18:59:14.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.176+0000 7fb89b860700 1 -- 172.21.15.114:0/1375269160 --> [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb89404ea90 con 0x7fb8800721c0 2024-02-19T18:59:14.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.176+0000 7fb88a7fc700 1 -- 172.21.15.114:0/1375269160 <== osd.1 v2:172.21.15.114:6810/828515125 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb89404ea90 con 0x7fb8800721c0 2024-02-19T18:59:14.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.177+0000 7fb89b860700 1 -- 172.21.15.114:0/1375269160 >> [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125] conn(0x7fb8800721c0 msgr2=0x7fb8800745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:14.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.177+0000 7fb89b860700 1 --2- 172.21.15.114:0/1375269160 >> [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125] conn(0x7fb8800721c0 0x7fb8800745e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:14.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.177+0000 7fb89b860700 1 -- 172.21.15.114:0/1375269160 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb88006c4e0 msgr2=0x7fb88006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:14.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.177+0000 7fb89b860700 1 --2- 172.21.15.114:0/1375269160 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb88006c4e0 0x7fb88006e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb890006fd0 tx=0x7fb890008040 comp rx=0 tx=0).stop 2024-02-19T18:59:14.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.177+0000 7fb89b860700 1 -- 172.21.15.114:0/1375269160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb894101750 msgr2=0x7fb89419cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:14.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.177+0000 7fb89b860700 1 --2- 172.21.15.114:0/1375269160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb894101750 0x7fb89419cae0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb8840092e0 tx=0x7fb88400be20 comp rx=0 tx=0).stop 2024-02-19T18:59:14.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.177+0000 7fb89b860700 1 -- 172.21.15.114:0/1375269160 shutdown_connections 2024-02-19T18:59:14.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.177+0000 7fb89b860700 1 --2- 172.21.15.114:0/1375269160 >> [v2:172.21.15.114:6810/828515125,v1:172.21.15.114:6811/828515125] conn(0x7fb8800721c0 0x7fb8800745e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:14.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.177+0000 7fb89b860700 1 --2- 172.21.15.114:0/1375269160 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb88006c4e0 0x7fb88006e9a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:14.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.177+0000 7fb89b860700 1 --2- 172.21.15.114:0/1375269160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb894100da0 0x7fb89419c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:14.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.177+0000 7fb89b860700 1 --2- 172.21.15.114:0/1375269160 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb894101750 0x7fb89419cae0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:14.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.177+0000 7fb89b860700 1 -- 172.21.15.114:0/1375269160 >> 172.21.15.114:0/1375269160 conn(0x7fb8940fc630 msgr2=0x7fb8940fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:14.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.177+0000 7fb89b860700 1 -- 172.21.15.114:0/1375269160 shutdown_connections 2024-02-19T18:59:14.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:14.177+0000 7fb89b860700 1 -- 172.21.15.114:0/1375269160 wait complete. 2024-02-19T18:59:14.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:13 smithi174 ceph-mon[41061]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:15.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.193+0000 7f82dd4da700 1 -- 172.21.15.114:0/15675163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d8068670 msgr2=0x7f82d8068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:15.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.193+0000 7f82dd4da700 1 --2- 172.21.15.114:0/15675163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d8068670 0x7f82d8068a50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f82c8009910 tx=0x7f82c8009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:15.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.193+0000 7f82dd4da700 1 -- 172.21.15.114:0/15675163 shutdown_connections 2024-02-19T18:59:15.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.193+0000 7f82dd4da700 1 --2- 172.21.15.114:0/15675163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82d8069020 0x7f82d8105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:15.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.193+0000 7f82dd4da700 1 --2- 172.21.15.114:0/15675163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d8068670 0x7f82d8068a50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:15.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.193+0000 7f82dd4da700 1 -- 172.21.15.114:0/15675163 >> 172.21.15.114:0/15675163 conn(0x7f82d80755a0 msgr2=0x7f82d80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:15.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.193+0000 7f82dd4da700 1 -- 172.21.15.114:0/15675163 shutdown_connections 2024-02-19T18:59:15.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.193+0000 7f82dd4da700 1 -- 172.21.15.114:0/15675163 wait complete. 2024-02-19T18:59:15.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.193+0000 7f82dd4da700 1 Processor -- start 2024-02-19T18:59:15.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.194+0000 7f82dd4da700 1 -- start start 2024-02-19T18:59:15.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.194+0000 7f82dd4da700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d8068670 0x7f82d80ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:15.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.194+0000 7f82dd4da700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82d8069020 0x7f82d81001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:15.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.194+0000 7f82dd4da700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d8100720 con 0x7f82d8068670 2024-02-19T18:59:15.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.194+0000 7f82dd4da700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d8100860 con 0x7f82d8069020 2024-02-19T18:59:15.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.194+0000 7f82d6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d8068670 0x7f82d80ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:15.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.194+0000 7f82d6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d8068670 0x7f82d80ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46830/0 (socket says 172.21.15.114:46830) 2024-02-19T18:59:15.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.194+0000 7f82d6ffd700 1 -- 172.21.15.114:0/2266546019 learned_addr learned my addr 172.21.15.114:0/2266546019 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:15.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.194+0000 7f82ce7fc700 1 --2- 172.21.15.114:0/2266546019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82d8069020 0x7f82d81001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:15.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.194+0000 7f82d6ffd700 1 -- 172.21.15.114:0/2266546019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82d8069020 msgr2=0x7f82d81001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:15.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.194+0000 7f82d6ffd700 1 --2- 172.21.15.114:0/2266546019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82d8069020 0x7f82d81001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:15.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.194+0000 7f82d6ffd700 1 -- 172.21.15.114:0/2266546019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c80095f0 con 0x7f82d8068670 2024-02-19T18:59:15.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.194+0000 7f82d6ffd700 1 --2- 172.21.15.114:0/2266546019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d8068670 0x7f82d80ffca0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f82c8006010 tx=0x7f82c800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:15.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.194+0000 7f82d4ff9700 1 -- 172.21.15.114:0/2266546019 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82c801d070 con 0x7f82d8068670 2024-02-19T18:59:15.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.194+0000 7f82dd4da700 1 -- 172.21.15.114:0/2266546019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d8100ae0 con 0x7f82d8068670 2024-02-19T18:59:15.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.195+0000 7f82d4ff9700 1 -- 172.21.15.114:0/2266546019 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f82c8004fa0 con 0x7f82d8068670 2024-02-19T18:59:15.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.195+0000 7f82d4ff9700 1 -- 172.21.15.114:0/2266546019 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82c800f690 con 0x7f82d8068670 2024-02-19T18:59:15.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.195+0000 7f82dd4da700 1 -- 172.21.15.114:0/2266546019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d81a2db0 con 0x7f82d8068670 2024-02-19T18:59:15.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.195+0000 7f82dd4da700 1 -- 172.21.15.114:0/2266546019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f82d810e080 con 0x7f82d8068670 2024-02-19T18:59:15.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.196+0000 7f82d4ff9700 1 -- 172.21.15.114:0/2266546019 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f82c8022470 con 0x7f82d8068670 2024-02-19T18:59:15.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.196+0000 7f82d4ff9700 1 --2- 172.21.15.114:0/2266546019 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f82c406c4e0 0x7f82c406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:15.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.196+0000 7f82d4ff9700 1 -- 172.21.15.114:0/2266546019 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f82c808d6d0 con 0x7f82d8068670 2024-02-19T18:59:15.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.196+0000 7f82ce7fc700 1 --2- 172.21.15.114:0/2266546019 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f82c406c4e0 0x7f82c406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:15.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.197+0000 7f82d4ff9700 1 --2- 172.21.15.114:0/2266546019 >> [v2:172.21.15.174:6800/2070964484,v1:172.21.15.174:6801/2070964484] conn(0x7f82c40721c0 0x7f82c40745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:15.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.197+0000 7f82d4ff9700 1 -- 172.21.15.114:0/2266546019 --> [v2:172.21.15.174:6800/2070964484,v1:172.21.15.174:6801/2070964484] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82c4074c90 con 0x7f82c40721c0 2024-02-19T18:59:15.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.197+0000 7f82d4ff9700 1 -- 172.21.15.114:0/2266546019 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82c808db30 con 0x7f82d8068670 2024-02-19T18:59:15.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.197+0000 7f82ce7fc700 1 --2- 172.21.15.114:0/2266546019 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f82c406c4e0 0x7f82c406e9a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f82d8101490 tx=0x7f82c0009190 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:15.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.197+0000 7f82d77fe700 1 --2- 172.21.15.114:0/2266546019 >> [v2:172.21.15.174:6800/2070964484,v1:172.21.15.174:6801/2070964484] conn(0x7f82c40721c0 0x7f82c40745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:15.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.197+0000 7f82d77fe700 1 --2- 172.21.15.114:0/2266546019 >> [v2:172.21.15.174:6800/2070964484,v1:172.21.15.174:6801/2070964484] conn(0x7f82c40721c0 0x7f82c40745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:15.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.198+0000 7f82d4ff9700 1 -- 172.21.15.114:0/2266546019 <== osd.3 v2:172.21.15.174:6800/2070964484 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f82c4074c90 con 0x7f82c40721c0 2024-02-19T18:59:15.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.214+0000 7f82dd4da700 1 -- 172.21.15.114:0/2266546019 --> [v2:172.21.15.174:6800/2070964484,v1:172.21.15.174:6801/2070964484] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f82d804ea90 con 0x7f82c40721c0 2024-02-19T18:59:15.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.214+0000 7f82d4ff9700 1 -- 172.21.15.114:0/2266546019 <== osd.3 v2:172.21.15.174:6800/2070964484 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f82d804ea90 con 0x7f82c40721c0 2024-02-19T18:59:15.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.215+0000 7f82dd4da700 1 -- 172.21.15.114:0/2266546019 >> [v2:172.21.15.174:6800/2070964484,v1:172.21.15.174:6801/2070964484] conn(0x7f82c40721c0 msgr2=0x7f82c40745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:15.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.215+0000 7f82dd4da700 1 --2- 172.21.15.114:0/2266546019 >> [v2:172.21.15.174:6800/2070964484,v1:172.21.15.174:6801/2070964484] conn(0x7f82c40721c0 0x7f82c40745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:15.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.215+0000 7f82dd4da700 1 -- 172.21.15.114:0/2266546019 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f82c406c4e0 msgr2=0x7f82c406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:15.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.215+0000 7f82dd4da700 1 --2- 172.21.15.114:0/2266546019 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f82c406c4e0 0x7f82c406e9a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f82d8101490 tx=0x7f82c0009190 comp rx=0 tx=0).stop 2024-02-19T18:59:15.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.215+0000 7f82dd4da700 1 -- 172.21.15.114:0/2266546019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d8068670 msgr2=0x7f82d80ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:15.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.215+0000 7f82dd4da700 1 --2- 172.21.15.114:0/2266546019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d8068670 0x7f82d80ffca0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f82c8006010 tx=0x7f82c800bd70 comp rx=0 tx=0).stop 2024-02-19T18:59:15.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.215+0000 7f82dd4da700 1 -- 172.21.15.114:0/2266546019 shutdown_connections 2024-02-19T18:59:15.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.215+0000 7f82dd4da700 1 --2- 172.21.15.114:0/2266546019 >> [v2:172.21.15.174:6800/2070964484,v1:172.21.15.174:6801/2070964484] conn(0x7f82c40721c0 0x7f82c40745e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:15.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.215+0000 7f82dd4da700 1 --2- 172.21.15.114:0/2266546019 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f82c406c4e0 0x7f82c406e9a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:15.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.215+0000 7f82dd4da700 1 --2- 172.21.15.114:0/2266546019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82d8068670 0x7f82d80ffca0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:15.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.215+0000 7f82dd4da700 1 --2- 172.21.15.114:0/2266546019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82d8069020 0x7f82d81001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:15.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.215+0000 7f82dd4da700 1 -- 172.21.15.114:0/2266546019 >> 172.21.15.114:0/2266546019 conn(0x7f82d80755a0 msgr2=0x7f82d80fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:15.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.215+0000 7f82dd4da700 1 -- 172.21.15.114:0/2266546019 shutdown_connections 2024-02-19T18:59:15.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:15.215+0000 7f82dd4da700 1 -- 172.21.15.114:0/2266546019 wait complete. 2024-02-19T18:59:16.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:15 smithi114 ceph-mon[29441]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:16.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:15 smithi174 ceph-mon[41061]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:16.481 INFO:teuthology.orchestra.run.smithi114.stdout:60129542175 2024-02-19T18:59:16.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-19T18:59:16.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.922+0000 7f0d9e169700 1 -- 172.21.15.114:0/2500679760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98068670 msgr2=0x7f0d98068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:16.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.922+0000 7f0d9e169700 1 --2- 172.21.15.114:0/2500679760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98068670 0x7f0d98068a50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0d88009910 tx=0x7f0d88009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:16.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.922+0000 7f0d9e169700 1 -- 172.21.15.114:0/2500679760 shutdown_connections 2024-02-19T18:59:16.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.922+0000 7f0d9e169700 1 --2- 172.21.15.114:0/2500679760 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d98069020 0x7f0d98105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:16.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.922+0000 7f0d9e169700 1 --2- 172.21.15.114:0/2500679760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98068670 0x7f0d98068a50 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:16.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.922+0000 7f0d9e169700 1 -- 172.21.15.114:0/2500679760 >> 172.21.15.114:0/2500679760 conn(0x7f0d980755a0 msgr2=0x7f0d980759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:16.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.922+0000 7f0d9e169700 1 -- 172.21.15.114:0/2500679760 shutdown_connections 2024-02-19T18:59:16.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.922+0000 7f0d9e169700 1 -- 172.21.15.114:0/2500679760 wait complete. 2024-02-19T18:59:16.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.922+0000 7f0d9e169700 1 Processor -- start 2024-02-19T18:59:16.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.923+0000 7f0d9e169700 1 -- start start 2024-02-19T18:59:16.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.923+0000 7f0d9e169700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d98068670 0x7f0d9819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:16.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.923+0000 7f0d9e169700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98069020 0x7f0d9819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:16.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.923+0000 7f0d9e169700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9819d0e0 con 0x7f0d98069020 2024-02-19T18:59:16.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.923+0000 7f0d9e169700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d98196590 con 0x7f0d98068670 2024-02-19T18:59:16.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.923+0000 7f0d96ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98069020 0x7f0d9819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:16.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.923+0000 7f0d96ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98069020 0x7f0d9819ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44642/0 (socket says 172.21.15.114:44642) 2024-02-19T18:59:16.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.923+0000 7f0d96ffd700 1 -- 172.21.15.114:0/917659567 learned_addr learned my addr 172.21.15.114:0/917659567 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:16.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.923+0000 7f0d977fe700 1 --2- 172.21.15.114:0/917659567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d98068670 0x7f0d9819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:16.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.923+0000 7f0d96ffd700 1 -- 172.21.15.114:0/917659567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d98068670 msgr2=0x7f0d9819c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:16.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.923+0000 7f0d96ffd700 1 --2- 172.21.15.114:0/917659567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d98068670 0x7f0d9819c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:16.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.923+0000 7f0d96ffd700 1 -- 172.21.15.114:0/917659567 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d880095f0 con 0x7f0d98069020 2024-02-19T18:59:16.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.923+0000 7f0d977fe700 1 --2- 172.21.15.114:0/917659567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d98068670 0x7f0d9819c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T18:59:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.923+0000 7f0d96ffd700 1 --2- 172.21.15.114:0/917659567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98069020 0x7f0d9819ca50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00d900 tx=0x7f0d8c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.923+0000 7f0d94ff9700 1 -- 172.21.15.114:0/917659567 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d8c009900 con 0x7f0d98069020 2024-02-19T18:59:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.924+0000 7f0d94ff9700 1 -- 172.21.15.114:0/917659567 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0d8c010460 con 0x7f0d98069020 2024-02-19T18:59:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.924+0000 7f0d9e169700 1 -- 172.21.15.114:0/917659567 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d98196870 con 0x7f0d98069020 2024-02-19T18:59:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.924+0000 7f0d94ff9700 1 -- 172.21.15.114:0/917659567 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d8c00f5f0 con 0x7f0d98069020 2024-02-19T18:59:16.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.924+0000 7f0d9e169700 1 -- 172.21.15.114:0/917659567 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d98196dc0 con 0x7f0d98069020 2024-02-19T18:59:16.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.924+0000 7f0d9e169700 1 -- 172.21.15.114:0/917659567 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0d9810e080 con 0x7f0d98069020 2024-02-19T18:59:16.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.925+0000 7f0d94ff9700 1 -- 172.21.15.114:0/917659567 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f0d8c0105d0 con 0x7f0d98069020 2024-02-19T18:59:16.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.925+0000 7f0d94ff9700 1 --2- 172.21.15.114:0/917659567 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0d8006c4e0 0x7f0d8006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:16.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.926+0000 7f0d977fe700 1 --2- 172.21.15.114:0/917659567 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0d8006c4e0 0x7f0d8006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:16.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.926+0000 7f0d94ff9700 1 -- 172.21.15.114:0/917659567 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0d8c08b330 con 0x7f0d98069020 2024-02-19T18:59:16.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.926+0000 7f0d94ff9700 1 --2- 172.21.15.114:0/917659567 >> [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196] conn(0x7f0d800721c0 0x7f0d800745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:16.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.926+0000 7f0d94ff9700 1 -- 172.21.15.114:0/917659567 --> [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d80074c90 con 0x7f0d800721c0 2024-02-19T18:59:16.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.926+0000 7f0d94ff9700 1 -- 172.21.15.114:0/917659567 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d8c08da00 con 0x7f0d98069020 2024-02-19T18:59:16.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.926+0000 7f0d97fff700 1 --2- 172.21.15.114:0/917659567 >> [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196] conn(0x7f0d800721c0 0x7f0d800745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:16.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.926+0000 7f0d977fe700 1 --2- 172.21.15.114:0/917659567 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0d8006c4e0 0x7f0d8006e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800b5c0 tx=0x7f0d88005960 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:16.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.926+0000 7f0d97fff700 1 --2- 172.21.15.114:0/917659567 >> [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196] conn(0x7f0d800721c0 0x7f0d800745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:16.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.927+0000 7f0d94ff9700 1 -- 172.21.15.114:0/917659567 <== osd.2 v2:172.21.15.114:6818/680148196 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f0d80074c90 con 0x7f0d800721c0 2024-02-19T18:59:16.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.942+0000 7f0d9e169700 1 -- 172.21.15.114:0/917659567 --> [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0d9804ea90 con 0x7f0d800721c0 2024-02-19T18:59:16.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.942+0000 7f0d94ff9700 1 -- 172.21.15.114:0/917659567 <== osd.2 v2:172.21.15.114:6818/680148196 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0d9804ea90 con 0x7f0d800721c0 2024-02-19T18:59:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.942+0000 7f0d9e169700 1 -- 172.21.15.114:0/917659567 >> [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196] conn(0x7f0d800721c0 msgr2=0x7f0d800745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.942+0000 7f0d9e169700 1 --2- 172.21.15.114:0/917659567 >> [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196] conn(0x7f0d800721c0 0x7f0d800745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.943+0000 7f0d9e169700 1 -- 172.21.15.114:0/917659567 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0d8006c4e0 msgr2=0x7f0d8006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.943+0000 7f0d9e169700 1 --2- 172.21.15.114:0/917659567 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0d8006c4e0 0x7f0d8006e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800b5c0 tx=0x7f0d88005960 comp rx=0 tx=0).stop 2024-02-19T18:59:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.943+0000 7f0d9e169700 1 -- 172.21.15.114:0/917659567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98069020 msgr2=0x7f0d9819ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.943+0000 7f0d9e169700 1 --2- 172.21.15.114:0/917659567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98069020 0x7f0d9819ca50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00d900 tx=0x7f0d8c00dcc0 comp rx=0 tx=0).stop 2024-02-19T18:59:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.943+0000 7f0d9e169700 1 -- 172.21.15.114:0/917659567 shutdown_connections 2024-02-19T18:59:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.943+0000 7f0d9e169700 1 --2- 172.21.15.114:0/917659567 >> [v2:172.21.15.114:6818/680148196,v1:172.21.15.114:6819/680148196] conn(0x7f0d800721c0 0x7f0d800745e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.943+0000 7f0d9e169700 1 --2- 172.21.15.114:0/917659567 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0d8006c4e0 0x7f0d8006e9a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.943+0000 7f0d9e169700 1 --2- 172.21.15.114:0/917659567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d98068670 0x7f0d9819c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.943+0000 7f0d9e169700 1 --2- 172.21.15.114:0/917659567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98069020 0x7f0d9819ca50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.943+0000 7f0d9e169700 1 -- 172.21.15.114:0/917659567 >> 172.21.15.114:0/917659567 conn(0x7f0d980755a0 msgr2=0x7f0d980fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.943+0000 7f0d9e169700 1 -- 172.21.15.114:0/917659567 shutdown_connections 2024-02-19T18:59:16.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:16.943+0000 7f0d9e169700 1 -- 172.21.15.114:0/917659567 wait complete. 2024-02-19T18:59:17.330 INFO:teuthology.orchestra.run.smithi114.stdout:115964117011 2024-02-19T18:59:17.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-19T18:59:18.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:17 smithi114 ceph-mon[29441]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:18.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:17 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:18.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:17 smithi174 ceph-mon[41061]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:18.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:17 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:18.411 INFO:teuthology.orchestra.run.smithi114.stdout:81604378648 2024-02-19T18:59:18.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-19T18:59:18.424 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:18.436 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:18.477 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:19.171 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:20.164 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:19 smithi114 ceph-mon[29441]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:20.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:19 smithi174 ceph-mon[41061]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:21.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.588+0000 7fb96b1cc700 1 -- 172.21.15.114:0/3820460667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9640ffa40 msgr2=0x7fb9640ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:21.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.588+0000 7fb96b1cc700 1 --2- 172.21.15.114:0/3820460667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9640ffa40 0x7fb9640ffec0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb950009910 tx=0x7fb950009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:21.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.588+0000 7fb96b1cc700 1 -- 172.21.15.114:0/3820460667 shutdown_connections 2024-02-19T18:59:21.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.588+0000 7fb96b1cc700 1 --2- 172.21.15.114:0/3820460667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9640ffa40 0x7fb9640ffec0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:21.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.588+0000 7fb96b1cc700 1 --2- 172.21.15.114:0/3820460667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb964105a40 0x7fb964105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:21.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.588+0000 7fb96b1cc700 1 -- 172.21.15.114:0/3820460667 >> 172.21.15.114:0/3820460667 conn(0x7fb9640fb570 msgr2=0x7fb9640fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:21.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.588+0000 7fb96b1cc700 1 -- 172.21.15.114:0/3820460667 shutdown_connections 2024-02-19T18:59:21.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.588+0000 7fb96b1cc700 1 -- 172.21.15.114:0/3820460667 wait complete. 2024-02-19T18:59:21.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.589+0000 7fb96b1cc700 1 Processor -- start 2024-02-19T18:59:21.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.589+0000 7fb96b1cc700 1 -- start start 2024-02-19T18:59:21.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.589+0000 7fb96b1cc700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9640ffa40 0x7fb9640fbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:21.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.589+0000 7fb96b1cc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb964105a40 0x7fb9640fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:21.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.589+0000 7fb96b1cc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9640fc900 con 0x7fb964105a40 2024-02-19T18:59:21.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.589+0000 7fb96b1cc700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9640fca40 con 0x7fb9640ffa40 2024-02-19T18:59:21.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.589+0000 7fb963fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb964105a40 0x7fb9640fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:21.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.589+0000 7fb963fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb964105a40 0x7fb9640fc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44662/0 (socket says 172.21.15.114:44662) 2024-02-19T18:59:21.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.589+0000 7fb963fff700 1 -- 172.21.15.114:0/2051213208 learned_addr learned my addr 172.21.15.114:0/2051213208 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:21.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.589+0000 7fb968f68700 1 --2- 172.21.15.114:0/2051213208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9640ffa40 0x7fb9640fbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:21.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.589+0000 7fb963fff700 1 -- 172.21.15.114:0/2051213208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9640ffa40 msgr2=0x7fb9640fbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:21.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.589+0000 7fb963fff700 1 --2- 172.21.15.114:0/2051213208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9640ffa40 0x7fb9640fbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:21.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.589+0000 7fb963fff700 1 -- 172.21.15.114:0/2051213208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9500095f0 con 0x7fb964105a40 2024-02-19T18:59:21.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.589+0000 7fb963fff700 1 --2- 172.21.15.114:0/2051213208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb964105a40 0x7fb9640fc3c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb95000fa00 tx=0x7fb95000fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:21.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.590+0000 7fb961ffb700 1 -- 172.21.15.114:0/2051213208 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb95001d070 con 0x7fb964105a40 2024-02-19T18:59:21.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.590+0000 7fb961ffb700 1 -- 172.21.15.114:0/2051213208 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb95000fe90 con 0x7fb964105a40 2024-02-19T18:59:21.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.590+0000 7fb96b1cc700 1 -- 172.21.15.114:0/2051213208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9640fccc0 con 0x7fb964105a40 2024-02-19T18:59:21.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.590+0000 7fb961ffb700 1 -- 172.21.15.114:0/2051213208 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb950022470 con 0x7fb964105a40 2024-02-19T18:59:21.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.590+0000 7fb96b1cc700 1 -- 172.21.15.114:0/2051213208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9641a2bf0 con 0x7fb964105a40 2024-02-19T18:59:21.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.590+0000 7fb96b1cc700 1 -- 172.21.15.114:0/2051213208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb96410cf40 con 0x7fb964105a40 2024-02-19T18:59:21.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.591+0000 7fb961ffb700 1 -- 172.21.15.114:0/2051213208 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fb950003680 con 0x7fb964105a40 2024-02-19T18:59:21.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.591+0000 7fb961ffb700 1 --2- 172.21.15.114:0/2051213208 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb95406c4e0 0x7fb95406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:21.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.592+0000 7fb961ffb700 1 -- 172.21.15.114:0/2051213208 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb95008cad0 con 0x7fb964105a40 2024-02-19T18:59:21.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.592+0000 7fb968f68700 1 --2- 172.21.15.114:0/2051213208 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb95406c4e0 0x7fb95406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:21.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.592+0000 7fb961ffb700 1 --2- 172.21.15.114:0/2051213208 >> [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688] conn(0x7fb9540721c0 0x7fb9540745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:21.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.592+0000 7fb961ffb700 1 -- 172.21.15.114:0/2051213208 --> [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb954074c90 con 0x7fb9540721c0 2024-02-19T18:59:21.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.592+0000 7fb961ffb700 1 -- 172.21.15.114:0/2051213208 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb95008cf30 con 0x7fb964105a40 2024-02-19T18:59:21.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.592+0000 7fb969769700 1 --2- 172.21.15.114:0/2051213208 >> [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688] conn(0x7fb9540721c0 0x7fb9540745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:21.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.592+0000 7fb968f68700 1 --2- 172.21.15.114:0/2051213208 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb95406c4e0 0x7fb95406e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb958009c60 tx=0x7fb958009260 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:21.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.592+0000 7fb969769700 1 --2- 172.21.15.114:0/2051213208 >> [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688] conn(0x7fb9540721c0 0x7fb9540745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:21.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.593+0000 7fb961ffb700 1 -- 172.21.15.114:0/2051213208 <== osd.0 v2:172.21.15.114:6802/2204928688 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb954074c90 con 0x7fb9540721c0 2024-02-19T18:59:21.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.608+0000 7fb96b1cc700 1 -- 172.21.15.114:0/2051213208 --> [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb96404ea90 con 0x7fb9540721c0 2024-02-19T18:59:21.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.608+0000 7fb961ffb700 1 -- 172.21.15.114:0/2051213208 <== osd.0 v2:172.21.15.114:6802/2204928688 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb96404ea90 con 0x7fb9540721c0 2024-02-19T18:59:21.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.609+0000 7fb96b1cc700 1 -- 172.21.15.114:0/2051213208 >> [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688] conn(0x7fb9540721c0 msgr2=0x7fb9540745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:21.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.609+0000 7fb96b1cc700 1 --2- 172.21.15.114:0/2051213208 >> [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688] conn(0x7fb9540721c0 0x7fb9540745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:21.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.609+0000 7fb96b1cc700 1 -- 172.21.15.114:0/2051213208 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb95406c4e0 msgr2=0x7fb95406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:21.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.609+0000 7fb96b1cc700 1 --2- 172.21.15.114:0/2051213208 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb95406c4e0 0x7fb95406e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb958009c60 tx=0x7fb958009260 comp rx=0 tx=0).stop 2024-02-19T18:59:21.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.609+0000 7fb96b1cc700 1 -- 172.21.15.114:0/2051213208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb964105a40 msgr2=0x7fb9640fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:21.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.609+0000 7fb96b1cc700 1 --2- 172.21.15.114:0/2051213208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb964105a40 0x7fb9640fc3c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb95000fa00 tx=0x7fb95000fa30 comp rx=0 tx=0).stop 2024-02-19T18:59:21.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.609+0000 7fb96b1cc700 1 -- 172.21.15.114:0/2051213208 shutdown_connections 2024-02-19T18:59:21.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.609+0000 7fb96b1cc700 1 --2- 172.21.15.114:0/2051213208 >> [v2:172.21.15.114:6802/2204928688,v1:172.21.15.114:6803/2204928688] conn(0x7fb9540721c0 0x7fb9540745e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:21.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.609+0000 7fb96b1cc700 1 --2- 172.21.15.114:0/2051213208 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb95406c4e0 0x7fb95406e9a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:21.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.609+0000 7fb96b1cc700 1 --2- 172.21.15.114:0/2051213208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9640ffa40 0x7fb9640fbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:21.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.609+0000 7fb96b1cc700 1 --2- 172.21.15.114:0/2051213208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb964105a40 0x7fb9640fc3c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:21.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.609+0000 7fb96b1cc700 1 -- 172.21.15.114:0/2051213208 >> 172.21.15.114:0/2051213208 conn(0x7fb9640fb570 msgr2=0x7fb964068ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:21.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.609+0000 7fb96b1cc700 1 -- 172.21.15.114:0/2051213208 shutdown_connections 2024-02-19T18:59:21.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:19.609+0000 7fb96b1cc700 1 -- 172.21.15.114:0/2051213208 wait complete. 2024-02-19T18:59:22.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:21 smithi114 ceph-mon[29441]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:22.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:21 smithi174 ceph-mon[41061]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:23.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.552+0000 7f350686b700 1 -- 172.21.15.114:0/1003217549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35000ffa40 msgr2=0x7f35000ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:23.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.552+0000 7f350686b700 1 --2- 172.21.15.114:0/1003217549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35000ffa40 0x7f35000ffec0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f34f4009910 tx=0x7f34f4009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:23.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.552+0000 7f350686b700 1 -- 172.21.15.114:0/1003217549 shutdown_connections 2024-02-19T18:59:23.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.552+0000 7f350686b700 1 --2- 172.21.15.114:0/1003217549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35000ffa40 0x7f35000ffec0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:23.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.552+0000 7f350686b700 1 --2- 172.21.15.114:0/1003217549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3500105a40 0x7f3500105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:23.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.552+0000 7f350686b700 1 -- 172.21.15.114:0/1003217549 >> 172.21.15.114:0/1003217549 conn(0x7f35000fb570 msgr2=0x7f35000fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:23.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.552+0000 7f350686b700 1 -- 172.21.15.114:0/1003217549 shutdown_connections 2024-02-19T18:59:23.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.552+0000 7f350686b700 1 -- 172.21.15.114:0/1003217549 wait complete. 2024-02-19T18:59:23.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.553+0000 7f350686b700 1 Processor -- start 2024-02-19T18:59:23.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.553+0000 7f350686b700 1 -- start start 2024-02-19T18:59:23.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.554+0000 7f350686b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35000ffa40 0x7f3500195620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:23.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.554+0000 7f350686b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3500105a40 0x7f3500195b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:23.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.554+0000 7f350686b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35001961b0 con 0x7f35000ffa40 2024-02-19T18:59:23.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.554+0000 7f350686b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35001962f0 con 0x7f3500105a40 2024-02-19T18:59:23.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.554+0000 7f34fffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35000ffa40 0x7f3500195620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:23.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.554+0000 7f34fffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35000ffa40 0x7f3500195620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44674/0 (socket says 172.21.15.114:44674) 2024-02-19T18:59:23.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.554+0000 7f34fffff700 1 -- 172.21.15.114:0/1215110508 learned_addr learned my addr 172.21.15.114:0/1215110508 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:23.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.554+0000 7f34ff7fe700 1 --2- 172.21.15.114:0/1215110508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3500105a40 0x7f3500195b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:23.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.554+0000 7f34fffff700 1 -- 172.21.15.114:0/1215110508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3500105a40 msgr2=0x7f3500195b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:23.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.554+0000 7f34fffff700 1 --2- 172.21.15.114:0/1215110508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3500105a40 0x7f3500195b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:23.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.554+0000 7f34fffff700 1 -- 172.21.15.114:0/1215110508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f40095f0 con 0x7f35000ffa40 2024-02-19T18:59:23.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.554+0000 7f34fffff700 1 --2- 172.21.15.114:0/1215110508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35000ffa40 0x7f3500195620 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f34f000c930 tx=0x7f34f000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:23.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.554+0000 7f34fd7fa700 1 -- 172.21.15.114:0/1215110508 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34f0007ad0 con 0x7f35000ffa40 2024-02-19T18:59:23.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.554+0000 7f350686b700 1 -- 172.21.15.114:0/1215110508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350019a140 con 0x7f35000ffa40 2024-02-19T18:59:23.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.554+0000 7f34fd7fa700 1 -- 172.21.15.114:0/1215110508 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f34f000ce80 con 0x7f35000ffa40 2024-02-19T18:59:23.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.554+0000 7f34fd7fa700 1 -- 172.21.15.114:0/1215110508 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34f0018630 con 0x7f35000ffa40 2024-02-19T18:59:23.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.554+0000 7f350686b700 1 -- 172.21.15.114:0/1215110508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350019a690 con 0x7f35000ffa40 2024-02-19T18:59:23.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.555+0000 7f350686b700 1 -- 172.21.15.114:0/1215110508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3500107fd0 con 0x7f35000ffa40 2024-02-19T18:59:23.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.556+0000 7f34fd7fa700 1 -- 172.21.15.114:0/1215110508 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f34f0007c30 con 0x7f35000ffa40 2024-02-19T18:59:23.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.556+0000 7f34fd7fa700 1 --2- 172.21.15.114:0/1215110508 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f34e806c4e0 0x7f34e806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:23.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.557+0000 7f34ff7fe700 1 --2- 172.21.15.114:0/1215110508 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f34e806c4e0 0x7f34e806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:23.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.557+0000 7f34fd7fa700 1 -- 172.21.15.114:0/1215110508 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f34f008b470 con 0x7f35000ffa40 2024-02-19T18:59:23.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.557+0000 7f34ff7fe700 1 --2- 172.21.15.114:0/1215110508 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f34e806c4e0 0x7f34e806e9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3500196c40 tx=0x7f34f401a040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:23.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.561+0000 7f34fd7fa700 1 -- 172.21.15.114:0/1215110508 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f34f0055f90 con 0x7f35000ffa40 2024-02-19T18:59:23.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.755+0000 7f350686b700 1 -- 172.21.15.114:0/1215110508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f350004ea90 con 0x7f35000ffa40 2024-02-19T18:59:23.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.755+0000 7f34fd7fa700 1 -- 172.21.15.114:0/1215110508 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f34f00595b0 con 0x7f35000ffa40 2024-02-19T18:59:23.774 INFO:teuthology.orchestra.run.smithi114.stdout:171798691850 2024-02-19T18:59:23.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.758+0000 7f350686b700 1 -- 172.21.15.114:0/1215110508 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f34e806c4e0 msgr2=0x7f34e806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:23.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.758+0000 7f350686b700 1 --2- 172.21.15.114:0/1215110508 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f34e806c4e0 0x7f34e806e9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3500196c40 tx=0x7f34f401a040 comp rx=0 tx=0).stop 2024-02-19T18:59:23.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.758+0000 7f350686b700 1 -- 172.21.15.114:0/1215110508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35000ffa40 msgr2=0x7f3500195620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:23.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.758+0000 7f350686b700 1 --2- 172.21.15.114:0/1215110508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35000ffa40 0x7f3500195620 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f34f000c930 tx=0x7f34f000cc40 comp rx=0 tx=0).stop 2024-02-19T18:59:23.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.758+0000 7f350686b700 1 -- 172.21.15.114:0/1215110508 shutdown_connections 2024-02-19T18:59:23.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.758+0000 7f350686b700 1 --2- 172.21.15.114:0/1215110508 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f34e806c4e0 0x7f34e806e9a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:23.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.758+0000 7f350686b700 1 --2- 172.21.15.114:0/1215110508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35000ffa40 0x7f3500195620 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:23.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.758+0000 7f350686b700 1 --2- 172.21.15.114:0/1215110508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3500105a40 0x7f3500195b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:23.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.758+0000 7f350686b700 1 -- 172.21.15.114:0/1215110508 >> 172.21.15.114:0/1215110508 conn(0x7f35000fb570 msgr2=0x7f35000fc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:23.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.758+0000 7f350686b700 1 -- 172.21.15.114:0/1215110508 shutdown_connections 2024-02-19T18:59:23.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:23.758+0000 7f350686b700 1 -- 172.21.15.114:0/1215110508 wait complete. 2024-02-19T18:59:24.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.177+0000 7faf01b14700 1 -- 172.21.15.114:0/2708773542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faefc100dc0 msgr2=0x7faefc1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:24.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.177+0000 7faf01b14700 1 --2- 172.21.15.114:0/2708773542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faefc100dc0 0x7faefc1011a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7faeec009910 tx=0x7faeec009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:24.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.177+0000 7faf01b14700 1 -- 172.21.15.114:0/2708773542 shutdown_connections 2024-02-19T18:59:24.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.177+0000 7faf01b14700 1 --2- 172.21.15.114:0/2708773542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faefc101770 0x7faefc1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:24.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.177+0000 7faf01b14700 1 --2- 172.21.15.114:0/2708773542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faefc100dc0 0x7faefc1011a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:24.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.177+0000 7faf01b14700 1 -- 172.21.15.114:0/2708773542 >> 172.21.15.114:0/2708773542 conn(0x7faefc0fc650 msgr2=0x7faefc0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:24.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.177+0000 7faf01b14700 1 -- 172.21.15.114:0/2708773542 shutdown_connections 2024-02-19T18:59:24.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.177+0000 7faf01b14700 1 -- 172.21.15.114:0/2708773542 wait complete. 2024-02-19T18:59:24.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.178+0000 7faf01b14700 1 Processor -- start 2024-02-19T18:59:24.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.178+0000 7faf01b14700 1 -- start start 2024-02-19T18:59:24.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.178+0000 7faf01b14700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faefc100dc0 0x7faefc19c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:24.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.178+0000 7faf01b14700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faefc101770 0x7faefc19ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:24.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.178+0000 7faf01b14700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faefc19d110 con 0x7faefc101770 2024-02-19T18:59:24.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.178+0000 7faf01b14700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faefc1965c0 con 0x7faefc100dc0 2024-02-19T18:59:24.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.178+0000 7faefaffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faefc101770 0x7faefc19ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:24.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.178+0000 7faefaffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faefc101770 0x7faefc19ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44690/0 (socket says 172.21.15.114:44690) 2024-02-19T18:59:24.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.178+0000 7faefaffd700 1 -- 172.21.15.114:0/3531822113 learned_addr learned my addr 172.21.15.114:0/3531822113 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:24.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.179+0000 7faefb7fe700 1 --2- 172.21.15.114:0/3531822113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faefc100dc0 0x7faefc19c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:24.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.179+0000 7faefaffd700 1 -- 172.21.15.114:0/3531822113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faefc100dc0 msgr2=0x7faefc19c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:24.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.179+0000 7faefaffd700 1 --2- 172.21.15.114:0/3531822113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faefc100dc0 0x7faefc19c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:24.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.179+0000 7faefaffd700 1 -- 172.21.15.114:0/3531822113 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec0095f0 con 0x7faefc101770 2024-02-19T18:59:24.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.179+0000 7faefaffd700 1 --2- 172.21.15.114:0/3531822113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faefc101770 0x7faefc19ca80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7faef000c8f0 tx=0x7faef000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:24.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.179+0000 7faef8ff9700 1 -- 172.21.15.114:0/3531822113 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faef00078a0 con 0x7faefc101770 2024-02-19T18:59:24.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.179+0000 7faf01b14700 1 -- 172.21.15.114:0/3531822113 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faefc1968a0 con 0x7faefc101770 2024-02-19T18:59:24.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.179+0000 7faef8ff9700 1 -- 172.21.15.114:0/3531822113 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faef000f450 con 0x7faefc101770 2024-02-19T18:59:24.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.179+0000 7faef8ff9700 1 -- 172.21.15.114:0/3531822113 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faef0018610 con 0x7faefc101770 2024-02-19T18:59:24.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.179+0000 7faf01b14700 1 -- 172.21.15.114:0/3531822113 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faefc196df0 con 0x7faefc101770 2024-02-19T18:59:24.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.180+0000 7faf01b14700 1 -- 172.21.15.114:0/3531822113 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faefc109160 con 0x7faefc101770 2024-02-19T18:59:24.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.181+0000 7faef8ff9700 1 -- 172.21.15.114:0/3531822113 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7faef000f5c0 con 0x7faefc101770 2024-02-19T18:59:24.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.181+0000 7faef8ff9700 1 --2- 172.21.15.114:0/3531822113 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7faee406c4e0 0x7faee406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:24.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.181+0000 7faefb7fe700 1 --2- 172.21.15.114:0/3531822113 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7faee406c4e0 0x7faee406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:24.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.181+0000 7faef8ff9700 1 -- 172.21.15.114:0/3531822113 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7faef008b400 con 0x7faefc101770 2024-02-19T18:59:24.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.182+0000 7faefb7fe700 1 --2- 172.21.15.114:0/3531822113 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7faee406c4e0 0x7faee406e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faeec006010 tx=0x7faeec01a040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:24.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.186+0000 7faef8ff9700 1 -- 172.21.15.114:0/3531822113 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7faef00599b0 con 0x7faefc101770 2024-02-19T18:59:24.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:23 smithi114 ceph-mon[29441]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:24.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:23 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1215110508' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-19T18:59:24.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:23 smithi174 ceph-mon[41061]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:24.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:23 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1215110508' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-19T18:59:24.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.381+0000 7faf01b14700 1 -- 172.21.15.114:0/3531822113 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7faefc1970a0 con 0x7faefc101770 2024-02-19T18:59:24.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.381+0000 7faef8ff9700 1 -- 172.21.15.114:0/3531822113 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faef0059540 con 0x7faefc101770 2024-02-19T18:59:24.399 INFO:teuthology.orchestra.run.smithi114.stdout:81604378649 2024-02-19T18:59:24.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.383+0000 7faf01b14700 1 -- 172.21.15.114:0/3531822113 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7faee406c4e0 msgr2=0x7faee406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:24.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.383+0000 7faf01b14700 1 --2- 172.21.15.114:0/3531822113 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7faee406c4e0 0x7faee406e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faeec006010 tx=0x7faeec01a040 comp rx=0 tx=0).stop 2024-02-19T18:59:24.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.383+0000 7faf01b14700 1 -- 172.21.15.114:0/3531822113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faefc101770 msgr2=0x7faefc19ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:24.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.383+0000 7faf01b14700 1 --2- 172.21.15.114:0/3531822113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faefc101770 0x7faefc19ca80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7faef000c8f0 tx=0x7faef000cc00 comp rx=0 tx=0).stop 2024-02-19T18:59:24.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.383+0000 7faf01b14700 1 -- 172.21.15.114:0/3531822113 shutdown_connections 2024-02-19T18:59:24.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.383+0000 7faf01b14700 1 --2- 172.21.15.114:0/3531822113 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7faee406c4e0 0x7faee406e9a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:24.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.383+0000 7faf01b14700 1 --2- 172.21.15.114:0/3531822113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faefc100dc0 0x7faefc19c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:24.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.383+0000 7faf01b14700 1 --2- 172.21.15.114:0/3531822113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faefc101770 0x7faefc19ca80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:24.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.383+0000 7faf01b14700 1 -- 172.21.15.114:0/3531822113 >> 172.21.15.114:0/3531822113 conn(0x7faefc0fc650 msgr2=0x7faefc0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:24.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.383+0000 7faf01b14700 1 -- 172.21.15.114:0/3531822113 shutdown_connections 2024-02-19T18:59:24.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:24.383+0000 7faf01b14700 1 -- 172.21.15.114:0/3531822113 wait complete. 2024-02-19T18:59:24.609 INFO:teuthology.orchestra.run.smithi114.stdout:38654705701 2024-02-19T18:59:24.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-19T18:59:24.779 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:25.187 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378648 got 81604378649 for osd.2 2024-02-19T18:59:25.187 DEBUG:teuthology.parallel:result is None 2024-02-19T18:59:25.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:24 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/3531822113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-19T18:59:25.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:24 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/3531822113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-19T18:59:26.238 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:25 smithi114 ceph-mon[29441]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:26.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:25 smithi174 ceph-mon[41061]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:27.479 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691848 got 171798691850 for osd.5 2024-02-19T18:59:27.479 DEBUG:teuthology.parallel:result is None 2024-02-19T18:59:27.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.652+0000 7fe739132700 1 -- 172.21.15.114:0/662744139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe734102680 msgr2=0x7fe73410ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:27.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.652+0000 7fe739132700 1 --2- 172.21.15.114:0/662744139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe734102680 0x7fe73410ab70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe728009940 tx=0x7fe728009c50 comp rx=0 tx=0).stop 2024-02-19T18:59:27.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.652+0000 7fe739132700 1 -- 172.21.15.114:0/662744139 shutdown_connections 2024-02-19T18:59:27.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.652+0000 7fe739132700 1 --2- 172.21.15.114:0/662744139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe734102680 0x7fe73410ab70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:27.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.652+0000 7fe739132700 1 --2- 172.21.15.114:0/662744139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe734101d60 0x7fe734102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:27.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.652+0000 7fe739132700 1 -- 172.21.15.114:0/662744139 >> 172.21.15.114:0/662744139 conn(0x7fe7340fb550 msgr2=0x7fe7340fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:27.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.652+0000 7fe739132700 1 -- 172.21.15.114:0/662744139 shutdown_connections 2024-02-19T18:59:27.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.652+0000 7fe739132700 1 -- 172.21.15.114:0/662744139 wait complete. 2024-02-19T18:59:27.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.653+0000 7fe739132700 1 Processor -- start 2024-02-19T18:59:27.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.653+0000 7fe739132700 1 -- start start 2024-02-19T18:59:27.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.653+0000 7fe739132700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe734101d60 0x7fe73419c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:27.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.653+0000 7fe739132700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe734102680 0x7fe73419cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:27.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.653+0000 7fe739132700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73419d2b0 con 0x7fe734101d60 2024-02-19T18:59:27.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.654+0000 7fe739132700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe734196760 con 0x7fe734102680 2024-02-19T18:59:27.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.654+0000 7fe732d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe734101d60 0x7fe73419c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:27.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.654+0000 7fe732d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe734101d60 0x7fe73419c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51952/0 (socket says 172.21.15.114:51952) 2024-02-19T18:59:27.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.654+0000 7fe732d9d700 1 -- 172.21.15.114:0/747077472 learned_addr learned my addr 172.21.15.114:0/747077472 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:27.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.654+0000 7fe73259c700 1 --2- 172.21.15.114:0/747077472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe734102680 0x7fe73419cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:27.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.654+0000 7fe732d9d700 1 -- 172.21.15.114:0/747077472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe734102680 msgr2=0x7fe73419cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:27.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.654+0000 7fe732d9d700 1 --2- 172.21.15.114:0/747077472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe734102680 0x7fe73419cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:27.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.654+0000 7fe732d9d700 1 -- 172.21.15.114:0/747077472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7280095f0 con 0x7fe734101d60 2024-02-19T18:59:27.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.654+0000 7fe732d9d700 1 --2- 172.21.15.114:0/747077472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe734101d60 0x7fe73419c6e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe72400b4b0 tx=0x7fe72400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:27.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.654+0000 7fe71bfff700 1 -- 172.21.15.114:0/747077472 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe724011800 con 0x7fe734101d60 2024-02-19T18:59:27.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.654+0000 7fe739132700 1 -- 172.21.15.114:0/747077472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe734196a40 con 0x7fe734101d60 2024-02-19T18:59:27.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.654+0000 7fe71bfff700 1 -- 172.21.15.114:0/747077472 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe724011e10 con 0x7fe734101d60 2024-02-19T18:59:27.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.654+0000 7fe71bfff700 1 -- 172.21.15.114:0/747077472 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe72400f2e0 con 0x7fe734101d60 2024-02-19T18:59:27.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.654+0000 7fe739132700 1 -- 172.21.15.114:0/747077472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe734196f60 con 0x7fe734101d60 2024-02-19T18:59:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.655+0000 7fe739132700 1 -- 172.21.15.114:0/747077472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe734108270 con 0x7fe734101d60 2024-02-19T18:59:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.656+0000 7fe71bfff700 1 -- 172.21.15.114:0/747077472 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fe724010380 con 0x7fe734101d60 2024-02-19T18:59:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.656+0000 7fe71bfff700 1 --2- 172.21.15.114:0/747077472 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe71c06c4e0 0x7fe71c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.656+0000 7fe71bfff700 1 -- 172.21.15.114:0/747077472 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe72408b090 con 0x7fe734101d60 2024-02-19T18:59:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.656+0000 7fe73259c700 1 --2- 172.21.15.114:0/747077472 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe71c06c4e0 0x7fe71c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.657+0000 7fe73259c700 1 --2- 172.21.15.114:0/747077472 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe71c06c4e0 0x7fe71c06e9a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe728009910 tx=0x7fe728005dc0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:27.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.661+0000 7fe71bfff700 1 -- 172.21.15.114:0/747077472 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe724059640 con 0x7fe734101d60 2024-02-19T18:59:27.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.853+0000 7fe739132700 1 -- 172.21.15.114:0/747077472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fe73404ea90 con 0x7fe734101d60 2024-02-19T18:59:27.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.853+0000 7fe71bfff700 1 -- 172.21.15.114:0/747077472 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe7240591d0 con 0x7fe734101d60 2024-02-19T18:59:27.872 INFO:teuthology.orchestra.run.smithi114.stdout:150323855376 2024-02-19T18:59:27.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.856+0000 7fe739132700 1 -- 172.21.15.114:0/747077472 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe71c06c4e0 msgr2=0x7fe71c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:27.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.856+0000 7fe739132700 1 --2- 172.21.15.114:0/747077472 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe71c06c4e0 0x7fe71c06e9a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe728009910 tx=0x7fe728005dc0 comp rx=0 tx=0).stop 2024-02-19T18:59:27.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.856+0000 7fe739132700 1 -- 172.21.15.114:0/747077472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe734101d60 msgr2=0x7fe73419c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:27.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.856+0000 7fe739132700 1 --2- 172.21.15.114:0/747077472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe734101d60 0x7fe73419c6e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe72400b4b0 tx=0x7fe72400b7c0 comp rx=0 tx=0).stop 2024-02-19T18:59:27.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.856+0000 7fe739132700 1 -- 172.21.15.114:0/747077472 shutdown_connections 2024-02-19T18:59:27.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.856+0000 7fe739132700 1 --2- 172.21.15.114:0/747077472 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe71c06c4e0 0x7fe71c06e9a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:27.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.856+0000 7fe739132700 1 --2- 172.21.15.114:0/747077472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe734101d60 0x7fe73419c6e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:27.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.856+0000 7fe739132700 1 --2- 172.21.15.114:0/747077472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe734102680 0x7fe73419cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:27.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.856+0000 7fe739132700 1 -- 172.21.15.114:0/747077472 >> 172.21.15.114:0/747077472 conn(0x7fe7340fb550 msgr2=0x7fe7341053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:27.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.856+0000 7fe739132700 1 -- 172.21.15.114:0/747077472 shutdown_connections 2024-02-19T18:59:27.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:27.856+0000 7fe739132700 1 -- 172.21.15.114:0/747077472 wait complete. 2024-02-19T18:59:28.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:27 smithi114 ceph-mon[29441]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:28.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:27 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/747077472' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-19T18:59:28.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:27 smithi174 ceph-mon[41061]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:28.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:27 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/747077472' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-19T18:59:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.832+0000 7f0c7c3e8700 1 -- 172.21.15.114:0/4083476634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c740ffc40 msgr2=0x7f0c74105780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.832+0000 7f0c7c3e8700 1 --2- 172.21.15.114:0/4083476634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c740ffc40 0x7f0c74105780 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0c64009910 tx=0x7f0c64009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.832+0000 7f0c7c3e8700 1 -- 172.21.15.114:0/4083476634 shutdown_connections 2024-02-19T18:59:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.832+0000 7f0c7c3e8700 1 --2- 172.21.15.114:0/4083476634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c740ffc40 0x7f0c74105780 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:28.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.832+0000 7f0c7c3e8700 1 --2- 172.21.15.114:0/4083476634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c740ff290 0x7f0c740ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:28.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.832+0000 7f0c7c3e8700 1 -- 172.21.15.114:0/4083476634 >> 172.21.15.114:0/4083476634 conn(0x7f0c74075e30 msgr2=0x7f0c740fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:28.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.832+0000 7f0c7c3e8700 1 -- 172.21.15.114:0/4083476634 shutdown_connections 2024-02-19T18:59:28.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.832+0000 7f0c7c3e8700 1 -- 172.21.15.114:0/4083476634 wait complete. 2024-02-19T18:59:28.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.833+0000 7f0c7c3e8700 1 Processor -- start 2024-02-19T18:59:28.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.833+0000 7f0c7c3e8700 1 -- start start 2024-02-19T18:59:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.833+0000 7f0c7c3e8700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c740ff290 0x7f0c74100210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.833+0000 7f0c7c3e8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c740ffc40 0x7f0c74100750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.833+0000 7f0c7c3e8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c74100c90 con 0x7f0c740ffc40 2024-02-19T18:59:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.833+0000 7f0c7c3e8700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c74100dd0 con 0x7f0c740ff290 2024-02-19T18:59:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.833+0000 7f0c79983700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c740ffc40 0x7f0c74100750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.833+0000 7f0c79983700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c740ffc40 0x7f0c74100750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51984/0 (socket says 172.21.15.114:51984) 2024-02-19T18:59:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.833+0000 7f0c79983700 1 -- 172.21.15.114:0/3478439514 learned_addr learned my addr 172.21.15.114:0/3478439514 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.833+0000 7f0c7a184700 1 --2- 172.21.15.114:0/3478439514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c740ff290 0x7f0c74100210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.833+0000 7f0c79983700 1 -- 172.21.15.114:0/3478439514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c740ff290 msgr2=0x7f0c74100210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.833+0000 7f0c79983700 1 --2- 172.21.15.114:0/3478439514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c740ff290 0x7f0c74100210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.833+0000 7f0c79983700 1 -- 172.21.15.114:0/3478439514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c640095f0 con 0x7f0c740ffc40 2024-02-19T18:59:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.834+0000 7f0c79983700 1 --2- 172.21.15.114:0/3478439514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c740ffc40 0x7f0c74100750 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0c6400bda0 tx=0x7f0c6400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.834+0000 7f0c6b7fe700 1 -- 172.21.15.114:0/3478439514 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c6401d070 con 0x7f0c740ffc40 2024-02-19T18:59:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.834+0000 7f0c7c3e8700 1 -- 172.21.15.114:0/3478439514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c74101050 con 0x7f0c740ffc40 2024-02-19T18:59:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.834+0000 7f0c7c3e8700 1 -- 172.21.15.114:0/3478439514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c741a2c10 con 0x7f0c740ffc40 2024-02-19T18:59:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.834+0000 7f0c6b7fe700 1 -- 172.21.15.114:0/3478439514 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0c64004fa0 con 0x7f0c740ffc40 2024-02-19T18:59:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.834+0000 7f0c6b7fe700 1 -- 172.21.15.114:0/3478439514 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c6400f690 con 0x7f0c740ffc40 2024-02-19T18:59:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.835+0000 7f0c7c3e8700 1 -- 172.21.15.114:0/3478439514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c58005320 con 0x7f0c740ffc40 2024-02-19T18:59:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.835+0000 7f0c6b7fe700 1 -- 172.21.15.114:0/3478439514 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f0c64004af0 con 0x7f0c740ffc40 2024-02-19T18:59:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.836+0000 7f0c6b7fe700 1 --2- 172.21.15.114:0/3478439514 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0c6006c4e0 0x7f0c6006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.836+0000 7f0c7a184700 1 --2- 172.21.15.114:0/3478439514 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0c6006c4e0 0x7f0c6006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.836+0000 7f0c6b7fe700 1 -- 172.21.15.114:0/3478439514 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0c6408ca60 con 0x7f0c740ffc40 2024-02-19T18:59:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.836+0000 7f0c7a184700 1 --2- 172.21.15.114:0/3478439514 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0c6006c4e0 0x7f0c6006e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0c70005950 tx=0x7f0c7000a300 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.840+0000 7f0c6b7fe700 1 -- 172.21.15.114:0/3478439514 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0c64057600 con 0x7f0c740ffc40 2024-02-19T18:59:29.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:29.062+0000 7f0c7c3e8700 1 -- 172.21.15.114:0/3478439514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f0c58005190 con 0x7f0c740ffc40 2024-02-19T18:59:29.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:29.062+0000 7f0c6b7fe700 1 -- 172.21.15.114:0/3478439514 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0c6405ac20 con 0x7f0c740ffc40 2024-02-19T18:59:29.081 INFO:teuthology.orchestra.run.smithi114.stdout:60129542177 2024-02-19T18:59:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:29.065+0000 7f0c7c3e8700 1 -- 172.21.15.114:0/3478439514 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0c6006c4e0 msgr2=0x7f0c6006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:29.065+0000 7f0c7c3e8700 1 --2- 172.21.15.114:0/3478439514 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0c6006c4e0 0x7f0c6006e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0c70005950 tx=0x7f0c7000a300 comp rx=0 tx=0).stop 2024-02-19T18:59:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:29.065+0000 7f0c7c3e8700 1 -- 172.21.15.114:0/3478439514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c740ffc40 msgr2=0x7f0c74100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:29.065+0000 7f0c7c3e8700 1 --2- 172.21.15.114:0/3478439514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c740ffc40 0x7f0c74100750 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0c6400bda0 tx=0x7f0c6400bdd0 comp rx=0 tx=0).stop 2024-02-19T18:59:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:29.065+0000 7f0c7c3e8700 1 -- 172.21.15.114:0/3478439514 shutdown_connections 2024-02-19T18:59:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:29.065+0000 7f0c7c3e8700 1 --2- 172.21.15.114:0/3478439514 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0c6006c4e0 0x7f0c6006e9a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:29.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:29.065+0000 7f0c7c3e8700 1 --2- 172.21.15.114:0/3478439514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c740ff290 0x7f0c74100210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:29.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:29.065+0000 7f0c7c3e8700 1 --2- 172.21.15.114:0/3478439514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c740ffc40 0x7f0c74100750 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:29.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:29.065+0000 7f0c7c3e8700 1 -- 172.21.15.114:0/3478439514 >> 172.21.15.114:0/3478439514 conn(0x7f0c74075e30 msgr2=0x7f0c740fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:29.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:29.065+0000 7f0c7c3e8700 1 -- 172.21.15.114:0/3478439514 shutdown_connections 2024-02-19T18:59:29.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:29.065+0000 7f0c7c3e8700 1 -- 172.21.15.114:0/3478439514 wait complete. 2024-02-19T18:59:29.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.715+0000 7f799ffe6700 1 -- 172.21.15.114:0/3999120176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7998100dc0 msgr2=0x7f79981011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:29.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.715+0000 7f799ffe6700 1 --2- 172.21.15.114:0/3999120176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7998100dc0 0x7f79981011a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7994009910 tx=0x7f7994009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:29.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.715+0000 7f799ffe6700 1 -- 172.21.15.114:0/3999120176 shutdown_connections 2024-02-19T18:59:29.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.715+0000 7f799ffe6700 1 --2- 172.21.15.114:0/3999120176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7998101770 0x7f79981057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:29.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.715+0000 7f799ffe6700 1 --2- 172.21.15.114:0/3999120176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7998100dc0 0x7f79981011a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:29.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.715+0000 7f799ffe6700 1 -- 172.21.15.114:0/3999120176 >> 172.21.15.114:0/3999120176 conn(0x7f79980fc650 msgr2=0x7f79980fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:29.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.715+0000 7f799ffe6700 1 -- 172.21.15.114:0/3999120176 shutdown_connections 2024-02-19T18:59:29.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.715+0000 7f799ffe6700 1 -- 172.21.15.114:0/3999120176 wait complete. 2024-02-19T18:59:29.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.716+0000 7f799ffe6700 1 Processor -- start 2024-02-19T18:59:29.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.716+0000 7f799ffe6700 1 -- start start 2024-02-19T18:59:29.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.716+0000 7f799ffe6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7998100dc0 0x7f799819c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:29.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.716+0000 7f799ffe6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7998101770 0x7f799819ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:29.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.716+0000 7f799ffe6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799819d110 con 0x7f7998101770 2024-02-19T18:59:29.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.716+0000 7f799ffe6700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79981965c0 con 0x7f7998100dc0 2024-02-19T18:59:29.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.716+0000 7f799d581700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7998101770 0x7f799819ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:29.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.716+0000 7f799d581700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7998101770 0x7f799819ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51964/0 (socket says 172.21.15.114:51964) 2024-02-19T18:59:29.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.716+0000 7f799d581700 1 -- 172.21.15.114:0/3756073787 learned_addr learned my addr 172.21.15.114:0/3756073787 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:29.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.716+0000 7f799dd82700 1 --2- 172.21.15.114:0/3756073787 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7998100dc0 0x7f799819c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:29.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.716+0000 7f799d581700 1 -- 172.21.15.114:0/3756073787 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7998100dc0 msgr2=0x7f799819c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:29.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.716+0000 7f799d581700 1 --2- 172.21.15.114:0/3756073787 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7998100dc0 0x7f799819c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:29.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.716+0000 7f799d581700 1 -- 172.21.15.114:0/3756073787 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79940095f0 con 0x7f7998101770 2024-02-19T18:59:29.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.717+0000 7f799d581700 1 --2- 172.21.15.114:0/3756073787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7998101770 0x7f799819ca80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f798800c8f0 tx=0x7f798800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:29.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.717+0000 7f798effd700 1 -- 172.21.15.114:0/3756073787 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79880078a0 con 0x7f7998101770 2024-02-19T18:59:29.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.717+0000 7f799ffe6700 1 -- 172.21.15.114:0/3756073787 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79981968a0 con 0x7f7998101770 2024-02-19T18:59:29.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.717+0000 7f798effd700 1 -- 172.21.15.114:0/3756073787 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f798800f450 con 0x7f7998101770 2024-02-19T18:59:29.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.717+0000 7f798effd700 1 -- 172.21.15.114:0/3756073787 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7988018610 con 0x7f7998101770 2024-02-19T18:59:29.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.717+0000 7f799ffe6700 1 -- 172.21.15.114:0/3756073787 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7998196df0 con 0x7f7998101770 2024-02-19T18:59:29.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.718+0000 7f799ffe6700 1 -- 172.21.15.114:0/3756073787 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7998109160 con 0x7f7998101770 2024-02-19T18:59:29.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.718+0000 7f798effd700 1 -- 172.21.15.114:0/3756073787 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f798800f5c0 con 0x7f7998101770 2024-02-19T18:59:29.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.719+0000 7f798effd700 1 --2- 172.21.15.114:0/3756073787 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f798406c4e0 0x7f798406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:29.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.719+0000 7f798effd700 1 -- 172.21.15.114:0/3756073787 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f798808b400 con 0x7f7998101770 2024-02-19T18:59:29.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.719+0000 7f799dd82700 1 --2- 172.21.15.114:0/3756073787 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f798406c4e0 0x7f798406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:29.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.719+0000 7f799dd82700 1 --2- 172.21.15.114:0/3756073787 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f798406c4e0 0x7f798406e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7994006010 tx=0x7f799401a040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:29.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.723+0000 7f798effd700 1 -- 172.21.15.114:0/3756073787 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f79880599b0 con 0x7f7998101770 2024-02-19T18:59:29.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.942+0000 7f799ffe6700 1 -- 172.21.15.114:0/3756073787 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f79981970c0 con 0x7f7998101770 2024-02-19T18:59:29.103 INFO:teuthology.orchestra.run.smithi114.stdout:115964117013 2024-02-19T18:59:29.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.943+0000 7f798effd700 1 -- 172.21.15.114:0/3756073787 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7988059540 con 0x7f7998101770 2024-02-19T18:59:29.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.945+0000 7f799ffe6700 1 -- 172.21.15.114:0/3756073787 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f798406c4e0 msgr2=0x7f798406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:29.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.945+0000 7f799ffe6700 1 --2- 172.21.15.114:0/3756073787 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f798406c4e0 0x7f798406e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7994006010 tx=0x7f799401a040 comp rx=0 tx=0).stop 2024-02-19T18:59:29.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.945+0000 7f799ffe6700 1 -- 172.21.15.114:0/3756073787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7998101770 msgr2=0x7f799819ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:29.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.945+0000 7f799ffe6700 1 --2- 172.21.15.114:0/3756073787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7998101770 0x7f799819ca80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f798800c8f0 tx=0x7f798800cc00 comp rx=0 tx=0).stop 2024-02-19T18:59:29.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.945+0000 7f799ffe6700 1 -- 172.21.15.114:0/3756073787 shutdown_connections 2024-02-19T18:59:29.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.945+0000 7f799ffe6700 1 --2- 172.21.15.114:0/3756073787 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f798406c4e0 0x7f798406e9a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:29.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.945+0000 7f799ffe6700 1 --2- 172.21.15.114:0/3756073787 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7998100dc0 0x7f799819c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:29.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.945+0000 7f799ffe6700 1 --2- 172.21.15.114:0/3756073787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7998101770 0x7f799819ca80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:29.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.945+0000 7f799ffe6700 1 -- 172.21.15.114:0/3756073787 >> 172.21.15.114:0/3756073787 conn(0x7f79980fc650 msgr2=0x7f79980fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:29.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.945+0000 7f799ffe6700 1 -- 172.21.15.114:0/3756073787 shutdown_connections 2024-02-19T18:59:29.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:28.945+0000 7f799ffe6700 1 -- 172.21.15.114:0/3756073787 wait complete. 2024-02-19T18:59:29.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:28 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/3756073787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-19T18:59:29.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:28 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/3756073787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-19T18:59:29.692 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855373 got 150323855376 for osd.4 2024-02-19T18:59:29.692 DEBUG:teuthology.parallel:result is None 2024-02-19T18:59:30.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:29 smithi114 ceph-mon[29441]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:30.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:29 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/3478439514' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-19T18:59:30.353 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542175 got 60129542177 for osd.1 2024-02-19T18:59:30.353 DEBUG:teuthology.parallel:result is None 2024-02-19T18:59:30.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:29 smithi174 ceph-mon[41061]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:30.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:29 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/3478439514' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-19T18:59:30.433 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:30.813 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117011 got 115964117013 for osd.3 2024-02-19T18:59:30.813 DEBUG:teuthology.parallel:result is None 2024-02-19T18:59:31.272 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:30 smithi114 ceph-mon[29441]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:31.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:30 smithi174 ceph-mon[41061]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:32.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.335+0000 7fddafe89700 1 -- 172.21.15.114:0/2115997819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda80ffa40 msgr2=0x7fdda80ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:32.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.335+0000 7fddafe89700 1 --2- 172.21.15.114:0/2115997819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda80ffa40 0x7fdda80ffec0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fdd98009910 tx=0x7fdd98009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:32.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.335+0000 7fddafe89700 1 -- 172.21.15.114:0/2115997819 shutdown_connections 2024-02-19T18:59:32.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.335+0000 7fddafe89700 1 --2- 172.21.15.114:0/2115997819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda80ffa40 0x7fdda80ffec0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:32.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.335+0000 7fddafe89700 1 --2- 172.21.15.114:0/2115997819 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdda8105a40 0x7fdda8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:32.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.335+0000 7fddafe89700 1 -- 172.21.15.114:0/2115997819 >> 172.21.15.114:0/2115997819 conn(0x7fdda80fb550 msgr2=0x7fdda80fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:32.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.335+0000 7fddafe89700 1 -- 172.21.15.114:0/2115997819 shutdown_connections 2024-02-19T18:59:32.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.335+0000 7fddafe89700 1 -- 172.21.15.114:0/2115997819 wait complete. 2024-02-19T18:59:32.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.335+0000 7fddafe89700 1 Processor -- start 2024-02-19T18:59:32.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.335+0000 7fddafe89700 1 -- start start 2024-02-19T18:59:32.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.336+0000 7fddafe89700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda80ffa40 0x7fdda81953a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:32.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.336+0000 7fddafe89700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdda8105a40 0x7fdda81958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:32.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.336+0000 7fddafe89700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda8195e20 con 0x7fdda80ffa40 2024-02-19T18:59:32.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.336+0000 7fddafe89700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda8195f90 con 0x7fdda8105a40 2024-02-19T18:59:32.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.336+0000 7fddadc25700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda80ffa40 0x7fdda81953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:32.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.336+0000 7fddadc25700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda80ffa40 0x7fdda81953a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52004/0 (socket says 172.21.15.114:52004) 2024-02-19T18:59:32.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.336+0000 7fddadc25700 1 -- 172.21.15.114:0/3022146681 learned_addr learned my addr 172.21.15.114:0/3022146681 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:32.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.336+0000 7fddad424700 1 --2- 172.21.15.114:0/3022146681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdda8105a40 0x7fdda81958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:32.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.336+0000 7fddadc25700 1 -- 172.21.15.114:0/3022146681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdda8105a40 msgr2=0x7fdda81958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:32.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.336+0000 7fddadc25700 1 --2- 172.21.15.114:0/3022146681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdda8105a40 0x7fdda81958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:32.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.336+0000 7fddadc25700 1 -- 172.21.15.114:0/3022146681 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd980095f0 con 0x7fdda80ffa40 2024-02-19T18:59:32.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.336+0000 7fddadc25700 1 --2- 172.21.15.114:0/3022146681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda80ffa40 0x7fdda81953a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdda400c8a0 tx=0x7fdda400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:32.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.336+0000 7fdd9effd700 1 -- 172.21.15.114:0/3022146681 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdda400cea0 con 0x7fdda80ffa40 2024-02-19T18:59:32.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.336+0000 7fdd9effd700 1 -- 172.21.15.114:0/3022146681 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdda4004830 con 0x7fdda80ffa40 2024-02-19T18:59:32.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.336+0000 7fddafe89700 1 -- 172.21.15.114:0/3022146681 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda81a2a10 con 0x7fdda80ffa40 2024-02-19T18:59:32.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.336+0000 7fdd9effd700 1 -- 172.21.15.114:0/3022146681 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdda40055b0 con 0x7fdda80ffa40 2024-02-19T18:59:32.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.336+0000 7fddafe89700 1 -- 172.21.15.114:0/3022146681 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda81a2d70 con 0x7fdda80ffa40 2024-02-19T18:59:32.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.337+0000 7fddafe89700 1 -- 172.21.15.114:0/3022146681 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdda8107fd0 con 0x7fdda80ffa40 2024-02-19T18:59:32.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.338+0000 7fdd9effd700 1 -- 172.21.15.114:0/3022146681 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fdda4007ad0 con 0x7fdda80ffa40 2024-02-19T18:59:32.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.338+0000 7fdd9effd700 1 --2- 172.21.15.114:0/3022146681 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fdd9406c4e0 0x7fdd9406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:32.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.338+0000 7fdd9effd700 1 -- 172.21.15.114:0/3022146681 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fdda408b110 con 0x7fdda80ffa40 2024-02-19T18:59:32.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.338+0000 7fddad424700 1 --2- 172.21.15.114:0/3022146681 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fdd9406c4e0 0x7fdd9406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:32.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.338+0000 7fddad424700 1 --2- 172.21.15.114:0/3022146681 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fdd9406c4e0 0x7fdd9406e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdda8069710 tx=0x7fdd9801a040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:32.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.343+0000 7fdd9effd700 1 -- 172.21.15.114:0/3022146681 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fdda4059740 con 0x7fdda80ffa40 2024-02-19T18:59:32.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.532+0000 7fddafe89700 1 -- 172.21.15.114:0/3022146681 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fdda804ea90 con 0x7fdda80ffa40 2024-02-19T18:59:32.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.533+0000 7fdd9effd700 1 -- 172.21.15.114:0/3022146681 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdda40592d0 con 0x7fdda80ffa40 2024-02-19T18:59:32.551 INFO:teuthology.orchestra.run.smithi114.stdout:38654705704 2024-02-19T18:59:32.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.534+0000 7fddafe89700 1 -- 172.21.15.114:0/3022146681 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fdd9406c4e0 msgr2=0x7fdd9406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:32.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.534+0000 7fddafe89700 1 --2- 172.21.15.114:0/3022146681 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fdd9406c4e0 0x7fdd9406e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdda8069710 tx=0x7fdd9801a040 comp rx=0 tx=0).stop 2024-02-19T18:59:32.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.534+0000 7fddafe89700 1 -- 172.21.15.114:0/3022146681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda80ffa40 msgr2=0x7fdda81953a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:32.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.534+0000 7fddafe89700 1 --2- 172.21.15.114:0/3022146681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda80ffa40 0x7fdda81953a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdda400c8a0 tx=0x7fdda400cc60 comp rx=0 tx=0).stop 2024-02-19T18:59:32.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.535+0000 7fddafe89700 1 -- 172.21.15.114:0/3022146681 shutdown_connections 2024-02-19T18:59:32.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.535+0000 7fddafe89700 1 --2- 172.21.15.114:0/3022146681 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fdd9406c4e0 0x7fdd9406e9a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:32.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.535+0000 7fddafe89700 1 --2- 172.21.15.114:0/3022146681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda80ffa40 0x7fdda81953a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:32.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.535+0000 7fddafe89700 1 --2- 172.21.15.114:0/3022146681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdda8105a40 0x7fdda81958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:32.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.535+0000 7fddafe89700 1 -- 172.21.15.114:0/3022146681 >> 172.21.15.114:0/3022146681 conn(0x7fdda80fb550 msgr2=0x7fdda80fc010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:32.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.535+0000 7fddafe89700 1 -- 172.21.15.114:0/3022146681 shutdown_connections 2024-02-19T18:59:32.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:32.535+0000 7fddafe89700 1 -- 172.21.15.114:0/3022146681 wait complete. 2024-02-19T18:59:32.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:32 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/3022146681' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-19T18:59:32.906 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:32 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/3022146681' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-19T18:59:33.237 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705701 got 38654705704 for osd.0 2024-02-19T18:59:33.237 DEBUG:teuthology.parallel:result is None 2024-02-19T18:59:33.237 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-19T18:59:33.238 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T18:59:33.442 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:33.892 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:33 smithi174 ceph-mon[41061]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:33.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:33 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:33.919 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:33 smithi114 ceph-mon[29441]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:33.919 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:35.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.017+0000 7f2c9fdbf700 1 -- 172.21.15.114:0/2135386860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c98105a40 msgr2=0x7f2c98105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:35.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.017+0000 7f2c9fdbf700 1 --2- 172.21.15.114:0/2135386860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c98105a40 0x7f2c98105e20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2c9000b0a0 tx=0x7f2c9000b3b0 comp rx=0 tx=0).stop 2024-02-19T18:59:35.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.017+0000 7f2c9fdbf700 1 -- 172.21.15.114:0/2135386860 shutdown_connections 2024-02-19T18:59:35.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.017+0000 7f2c9fdbf700 1 --2- 172.21.15.114:0/2135386860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c980ffbd0 0x7f2c98100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:35.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.017+0000 7f2c9fdbf700 1 --2- 172.21.15.114:0/2135386860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c98105a40 0x7f2c98105e20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:35.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.017+0000 7f2c9fdbf700 1 -- 172.21.15.114:0/2135386860 >> 172.21.15.114:0/2135386860 conn(0x7f2c980755a0 msgr2=0x7f2c980759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:35.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.017+0000 7f2c9fdbf700 1 -- 172.21.15.114:0/2135386860 shutdown_connections 2024-02-19T18:59:35.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.017+0000 7f2c9fdbf700 1 -- 172.21.15.114:0/2135386860 wait complete. 2024-02-19T18:59:35.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.018+0000 7f2c9fdbf700 1 Processor -- start 2024-02-19T18:59:35.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.018+0000 7f2c9fdbf700 1 -- start start 2024-02-19T18:59:35.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.018+0000 7f2c9fdbf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c980ffbd0 0x7f2c98197550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:35.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.018+0000 7f2c9fdbf700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c98105a40 0x7f2c98197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:35.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.018+0000 7f2c9fdbf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c98198170 con 0x7f2c980ffbd0 2024-02-19T18:59:35.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.018+0000 7f2c9fdbf700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c981982b0 con 0x7f2c98105a40 2024-02-19T18:59:35.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.018+0000 7f2c9db5b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c980ffbd0 0x7f2c98197550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:35.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.018+0000 7f2c9db5b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c980ffbd0 0x7f2c98197550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52032/0 (socket says 172.21.15.114:52032) 2024-02-19T18:59:35.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.018+0000 7f2c9db5b700 1 -- 172.21.15.114:0/3757463384 learned_addr learned my addr 172.21.15.114:0/3757463384 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:35.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.018+0000 7f2c9d35a700 1 --2- 172.21.15.114:0/3757463384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c98105a40 0x7f2c98197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:35.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.018+0000 7f2c9db5b700 1 -- 172.21.15.114:0/3757463384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c98105a40 msgr2=0x7f2c98197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:35.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.018+0000 7f2c9db5b700 1 --2- 172.21.15.114:0/3757463384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c98105a40 0x7f2c98197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:35.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.018+0000 7f2c9db5b700 1 -- 172.21.15.114:0/3757463384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c90009d00 con 0x7f2c980ffbd0 2024-02-19T18:59:35.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.018+0000 7f2c9d35a700 1 --2- 172.21.15.114:0/3757463384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c98105a40 0x7f2c98197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T18:59:35.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.018+0000 7f2c9db5b700 1 --2- 172.21.15.114:0/3757463384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c980ffbd0 0x7f2c98197550 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2c90000f80 tx=0x7f2c9000cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:35.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.019+0000 7f2c8effd700 1 -- 172.21.15.114:0/3757463384 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c9000e040 con 0x7f2c980ffbd0 2024-02-19T18:59:35.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.019+0000 7f2c8effd700 1 -- 172.21.15.114:0/3757463384 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2c90003de0 con 0x7f2c980ffbd0 2024-02-19T18:59:35.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.019+0000 7f2c9fdbf700 1 -- 172.21.15.114:0/3757463384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c981a05c0 con 0x7f2c980ffbd0 2024-02-19T18:59:35.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.019+0000 7f2c8effd700 1 -- 172.21.15.114:0/3757463384 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c9001d9a0 con 0x7f2c980ffbd0 2024-02-19T18:59:35.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.019+0000 7f2c9fdbf700 1 -- 172.21.15.114:0/3757463384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c981a0a30 con 0x7f2c980ffbd0 2024-02-19T18:59:35.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.020+0000 7f2c9fdbf700 1 -- 172.21.15.114:0/3757463384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c98108040 con 0x7f2c980ffbd0 2024-02-19T18:59:35.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.020+0000 7f2c8effd700 1 -- 172.21.15.114:0/3757463384 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2c90004040 con 0x7f2c980ffbd0 2024-02-19T18:59:35.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.020+0000 7f2c8effd700 1 --2- 172.21.15.114:0/3757463384 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2c8406c4e0 0x7f2c8406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:35.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.021+0000 7f2c8effd700 1 -- 172.21.15.114:0/3757463384 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2c9008cd70 con 0x7f2c980ffbd0 2024-02-19T18:59:35.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.021+0000 7f2c9d35a700 1 --2- 172.21.15.114:0/3757463384 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2c8406c4e0 0x7f2c8406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:35.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.021+0000 7f2c9d35a700 1 --2- 172.21.15.114:0/3757463384 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2c8406c4e0 0x7f2c8406e9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2c88005950 tx=0x7f2c880058e0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:35.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.025+0000 7f2c8effd700 1 -- 172.21.15.114:0/3757463384 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2c90057910 con 0x7f2c980ffbd0 2024-02-19T18:59:35.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.209+0000 7f2c9fdbf700 1 -- 172.21.15.114:0/3757463384 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2c9802cd80 con 0x7f2c8406c4e0 2024-02-19T18:59:35.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.211+0000 7f2c8effd700 1 -- 172.21.15.114:0/3757463384 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27659 (secure 0 0 0) 0x7f2c9802cd80 con 0x7f2c8406c4e0 2024-02-19T18:59:35.229 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:59:35.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.213+0000 7f2c9fdbf700 1 -- 172.21.15.114:0/3757463384 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2c8406c4e0 msgr2=0x7f2c8406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:35.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.213+0000 7f2c9fdbf700 1 --2- 172.21.15.114:0/3757463384 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2c8406c4e0 0x7f2c8406e9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2c88005950 tx=0x7f2c880058e0 comp rx=0 tx=0).stop 2024-02-19T18:59:35.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.213+0000 7f2c9fdbf700 1 -- 172.21.15.114:0/3757463384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c980ffbd0 msgr2=0x7f2c98197550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:35.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.213+0000 7f2c9fdbf700 1 --2- 172.21.15.114:0/3757463384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c980ffbd0 0x7f2c98197550 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2c90000f80 tx=0x7f2c9000cdd0 comp rx=0 tx=0).stop 2024-02-19T18:59:35.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.213+0000 7f2c9fdbf700 1 -- 172.21.15.114:0/3757463384 shutdown_connections 2024-02-19T18:59:35.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.213+0000 7f2c9fdbf700 1 --2- 172.21.15.114:0/3757463384 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2c8406c4e0 0x7f2c8406e9a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:35.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.213+0000 7f2c9fdbf700 1 --2- 172.21.15.114:0/3757463384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c980ffbd0 0x7f2c98197550 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:35.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.213+0000 7f2c9fdbf700 1 --2- 172.21.15.114:0/3757463384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c98105a40 0x7f2c98197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:35.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.213+0000 7f2c9fdbf700 1 -- 172.21.15.114:0/3757463384 >> 172.21.15.114:0/3757463384 conn(0x7f2c980755a0 msgr2=0x7f2c980fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:35.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.213+0000 7f2c9fdbf700 1 -- 172.21.15.114:0/3757463384 shutdown_connections 2024-02-19T18:59:35.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:35.213+0000 7f2c9fdbf700 1 -- 172.21.15.114:0/3757463384 wait complete. 2024-02-19T18:59:35.233 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-02-19T18:59:35.760 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-02-19T18:59:34.657129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165636,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305276,"statfs":{"total":575970213888,"available":575800602624,"internally_reserved":0,"allocated":4018176,"data_stored":2627442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002063"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":185,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-19T18:58:46.140640+0000","last_change":"2024-02-19T18:58:18.578701+0000","last_active":"2024-02-19T18:58:46.140640+0000","last_peered":"2024-02-19T18:58:46.140640+0000","last_clean":"2024-02-19T18:58:46.140640+0000","last_became_active":"2024-02-19T18:58:18.578439+0000","last_became_peered":"2024-02-19T18:58:18.578439+0000","last_unstale":"2024-02-19T18:58:46.140640+0000","last_undegraded":"2024-02-19T18:58:46.140640+0000","last_fullsized":"2024-02-19T18:58:46.140640+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T18:57:30.993433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T18:57:30.993433+0000","last_clean_scrub_stamp":"2024-02-19T18:57:30.993433+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:35.661914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.61799999999999999}]},{"osd":1,"last update":"Mon Feb 19 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.63500000000000001}]},{"osd":2,"last update":"Mon Feb 19 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.56899999999999995}]},{"osd":3,"last update":"Mon Feb 19 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.375},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.40300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 18:58:55 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.47399999999999998}]},{"osd":1,"last update":"Mon Feb 19 18:58:55 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.54900000000000004}]},{"osd":2,"last update":"Mon Feb 19 18:58:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.59699999999999998}]},{"osd":4,"last update":"Mon Feb 19 18:59:19 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.315}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378651,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 18:59:28 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.34000000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.45500000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.35899999999999999,"15min":0.36299999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.46000000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.35999999999999999}]},{"osd":1,"last update":"Mon Feb 19 18:59:28 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.373,"15min":0.38},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.47699999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.38,"15min":0.38600000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.496,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.32700000000000001}]},{"osd":3,"last update":"Mon Feb 19 18:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.63600000000000001}]},{"osd":4,"last update":"Mon Feb 19 18:59:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.67600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 18:59:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.29799999999999999,"15min":0.28999999999999998},"min":{"1min":0.23699999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.374,"5min":0.32600000000000001,"15min":0.318},"min":{"1min":0.24399999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.59699999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.23599999999999999}]},{"osd":2,"last update":"Mon Feb 19 18:58:30 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.315},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.30299999999999999}]},{"osd":3,"last update":"Mon Feb 19 18:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.44900000000000001}]},{"osd":4,"last update":"Mon Feb 19 18:59:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.42199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 18:59:05 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.308,"15min":0.30499999999999999},"min":{"1min":0.23400000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.32200000000000001,"15min":0.316},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.26400000000000001}]},{"osd":2,"last update":"Mon Feb 19 18:58:33 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.30099999999999999}]},{"osd":3,"last update":"Mon Feb 19 18:58:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.372},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.39600000000000002}]},{"osd":4,"last update":"Mon Feb 19 18:59:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T18:59:35.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T18:59:35.966 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:36.036 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:35 smithi114 ceph-mon[29441]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:36.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:35 smithi174 ceph-mon[41061]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:37.005 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:36 smithi114 ceph-mon[29441]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T18:59:37.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:36 smithi174 ceph-mon[41061]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T18:59:37.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.505+0000 7fa584f18700 1 -- 172.21.15.114:0/3284912185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5800731c0 msgr2=0x7fa5800735a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:37.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.505+0000 7fa584f18700 1 --2- 172.21.15.114:0/3284912185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5800731c0 0x7fa5800735a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa570009910 tx=0x7fa570009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:37.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.505+0000 7fa584f18700 1 -- 172.21.15.114:0/3284912185 shutdown_connections 2024-02-19T18:59:37.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.505+0000 7fa584f18700 1 --2- 172.21.15.114:0/3284912185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa580073ae0 0x7fa58010ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:37.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.505+0000 7fa584f18700 1 --2- 172.21.15.114:0/3284912185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5800731c0 0x7fa5800735a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:37.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.505+0000 7fa584f18700 1 -- 172.21.15.114:0/3284912185 >> 172.21.15.114:0/3284912185 conn(0x7fa5800fc670 msgr2=0x7fa5800fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:37.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.505+0000 7fa584f18700 1 -- 172.21.15.114:0/3284912185 shutdown_connections 2024-02-19T18:59:37.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.505+0000 7fa584f18700 1 -- 172.21.15.114:0/3284912185 wait complete. 2024-02-19T18:59:37.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.506+0000 7fa584f18700 1 Processor -- start 2024-02-19T18:59:37.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.506+0000 7fa584f18700 1 -- start start 2024-02-19T18:59:37.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.506+0000 7fa584f18700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5800731c0 0x7fa58019e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:37.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.506+0000 7fa584f18700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa580073ae0 0x7fa58019ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:37.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.506+0000 7fa584f18700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58019f2c0 con 0x7fa580073ae0 2024-02-19T18:59:37.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.506+0000 7fa584f18700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5801987c0 con 0x7fa5800731c0 2024-02-19T18:59:37.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.506+0000 7fa575d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa580073ae0 0x7fa58019ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:37.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.506+0000 7fa575d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa580073ae0 0x7fa58019ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36062/0 (socket says 172.21.15.114:36062) 2024-02-19T18:59:37.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.506+0000 7fa575d9b700 1 -- 172.21.15.114:0/3570190775 learned_addr learned my addr 172.21.15.114:0/3570190775 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:37.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.506+0000 7fa57e59c700 1 --2- 172.21.15.114:0/3570190775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5800731c0 0x7fa58019e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:37.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.506+0000 7fa575d9b700 1 -- 172.21.15.114:0/3570190775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5800731c0 msgr2=0x7fa58019e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:37.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.506+0000 7fa575d9b700 1 --2- 172.21.15.114:0/3570190775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5800731c0 0x7fa58019e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:37.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.506+0000 7fa575d9b700 1 -- 172.21.15.114:0/3570190775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5700095f0 con 0x7fa580073ae0 2024-02-19T18:59:37.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.506+0000 7fa57e59c700 1 --2- 172.21.15.114:0/3570190775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5800731c0 0x7fa58019e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T18:59:37.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.507+0000 7fa575d9b700 1 --2- 172.21.15.114:0/3570190775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa580073ae0 0x7fa58019ec30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa56800d8d0 tx=0x7fa56800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:37.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.507+0000 7fa577fff700 1 -- 172.21.15.114:0/3570190775 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa568009940 con 0x7fa580073ae0 2024-02-19T18:59:37.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.507+0000 7fa577fff700 1 -- 172.21.15.114:0/3570190775 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa568010460 con 0x7fa580073ae0 2024-02-19T18:59:37.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.507+0000 7fa584f18700 1 -- 172.21.15.114:0/3570190775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa580198aa0 con 0x7fa580073ae0 2024-02-19T18:59:37.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.507+0000 7fa577fff700 1 -- 172.21.15.114:0/3570190775 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa56800f5f0 con 0x7fa580073ae0 2024-02-19T18:59:37.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.507+0000 7fa584f18700 1 -- 172.21.15.114:0/3570190775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa580198ff0 con 0x7fa580073ae0 2024-02-19T18:59:37.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.508+0000 7fa584f18700 1 -- 172.21.15.114:0/3570190775 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa58010a560 con 0x7fa580073ae0 2024-02-19T18:59:37.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.508+0000 7fa577fff700 1 -- 172.21.15.114:0/3570190775 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fa5680105d0 con 0x7fa580073ae0 2024-02-19T18:59:37.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.509+0000 7fa577fff700 1 --2- 172.21.15.114:0/3570190775 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fa56c06c4e0 0x7fa56c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:37.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.509+0000 7fa57e59c700 1 --2- 172.21.15.114:0/3570190775 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fa56c06c4e0 0x7fa56c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:37.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.509+0000 7fa577fff700 1 -- 172.21.15.114:0/3570190775 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa56808b3f0 con 0x7fa580073ae0 2024-02-19T18:59:37.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.509+0000 7fa57e59c700 1 --2- 172.21.15.114:0/3570190775 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fa56c06c4e0 0x7fa56c06e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa5700059d0 tx=0x7fa57001a040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:37.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.513+0000 7fa577fff700 1 -- 172.21.15.114:0/3570190775 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa568059a20 con 0x7fa580073ae0 2024-02-19T18:59:37.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.705+0000 7fa584f18700 1 -- 172.21.15.114:0/3570190775 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa580199320 con 0x7fa56c06c4e0 2024-02-19T18:59:37.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.707+0000 7fa577fff700 1 -- 172.21.15.114:0/3570190775 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27700 (secure 0 0 0) 0x7fa580199320 con 0x7fa56c06c4e0 2024-02-19T18:59:37.726 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:59:37.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.710+0000 7fa584f18700 1 -- 172.21.15.114:0/3570190775 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fa56c06c4e0 msgr2=0x7fa56c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:37.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.710+0000 7fa584f18700 1 --2- 172.21.15.114:0/3570190775 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fa56c06c4e0 0x7fa56c06e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa5700059d0 tx=0x7fa57001a040 comp rx=0 tx=0).stop 2024-02-19T18:59:37.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.710+0000 7fa584f18700 1 -- 172.21.15.114:0/3570190775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa580073ae0 msgr2=0x7fa58019ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:37.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.710+0000 7fa584f18700 1 --2- 172.21.15.114:0/3570190775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa580073ae0 0x7fa58019ec30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa56800d8d0 tx=0x7fa56800dc90 comp rx=0 tx=0).stop 2024-02-19T18:59:37.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.710+0000 7fa584f18700 1 -- 172.21.15.114:0/3570190775 shutdown_connections 2024-02-19T18:59:37.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.710+0000 7fa584f18700 1 --2- 172.21.15.114:0/3570190775 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fa56c06c4e0 0x7fa56c06e9a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:37.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.710+0000 7fa584f18700 1 --2- 172.21.15.114:0/3570190775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5800731c0 0x7fa58019e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:37.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.710+0000 7fa584f18700 1 --2- 172.21.15.114:0/3570190775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa580073ae0 0x7fa58019ec30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:37.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.710+0000 7fa584f18700 1 -- 172.21.15.114:0/3570190775 >> 172.21.15.114:0/3570190775 conn(0x7fa5800fc670 msgr2=0x7fa5801076a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:37.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.710+0000 7fa584f18700 1 -- 172.21.15.114:0/3570190775 shutdown_connections 2024-02-19T18:59:37.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:37.710+0000 7fa584f18700 1 -- 172.21.15.114:0/3570190775 wait complete. 2024-02-19T18:59:37.729 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-02-19T18:59:38.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:37 smithi114 ceph-mon[29441]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:38.142 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:37 smithi174 ceph-mon[41061]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:38.340 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-02-19T18:59:36.657486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165636,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305276,"statfs":{"total":575970213888,"available":575800602624,"internally_reserved":0,"allocated":4018176,"data_stored":2627442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001907"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":185,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-19T18:58:46.140640+0000","last_change":"2024-02-19T18:58:18.578701+0000","last_active":"2024-02-19T18:58:46.140640+0000","last_peered":"2024-02-19T18:58:46.140640+0000","last_clean":"2024-02-19T18:58:46.140640+0000","last_became_active":"2024-02-19T18:58:18.578439+0000","last_became_peered":"2024-02-19T18:58:18.578439+0000","last_unstale":"2024-02-19T18:58:46.140640+0000","last_undegraded":"2024-02-19T18:58:46.140640+0000","last_fullsized":"2024-02-19T18:58:46.140640+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T18:57:30.993433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T18:57:30.993433+0000","last_clean_scrub_stamp":"2024-02-19T18:57:30.993433+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T02:45:35.661914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.61799999999999999}]},{"osd":1,"last update":"Mon Feb 19 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.63500000000000001}]},{"osd":2,"last update":"Mon Feb 19 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.56899999999999995}]},{"osd":3,"last update":"Mon Feb 19 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.375},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.40300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 18:58:55 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.47399999999999998}]},{"osd":1,"last update":"Mon Feb 19 18:58:55 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.54900000000000004}]},{"osd":2,"last update":"Mon Feb 19 18:58:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.59699999999999998}]},{"osd":4,"last update":"Mon Feb 19 18:59:19 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.315}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27316,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717836,"statfs":{"total":95995035648,"available":95967064064,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 18:59:28 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.34000000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.45500000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.35899999999999999,"15min":0.36299999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.46000000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.36599999999999999}]},{"osd":1,"last update":"Mon Feb 19 18:59:28 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.373,"15min":0.38},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.47699999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.38,"15min":0.38600000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.496,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.38600000000000001}]},{"osd":3,"last update":"Mon Feb 19 18:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.747},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.71799999999999997}]},{"osd":4,"last update":"Mon Feb 19 18:59:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.72799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 18:59:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.29799999999999999,"15min":0.28999999999999998},"min":{"1min":0.23699999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.32600000000000001,"15min":0.318},"min":{"1min":0.24399999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.59699999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.54200000000000004}]},{"osd":2,"last update":"Mon Feb 19 18:59:31 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.35199999999999998,"15min":0.35399999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.53800000000000003,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.36699999999999999,"15min":0.37},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.49399999999999999}]},{"osd":3,"last update":"Mon Feb 19 18:58:54 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.53200000000000003}]},{"osd":4,"last update":"Mon Feb 19 18:59:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 18:59:05 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.308,"15min":0.30499999999999999},"min":{"1min":0.23400000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.32200000000000001,"15min":0.316},"min":{"1min":0.25600000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.26400000000000001}]},{"osd":2,"last update":"Mon Feb 19 18:58:33 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.30099999999999999}]},{"osd":3,"last update":"Mon Feb 19 18:58:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.372},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.39600000000000002}]},{"osd":4,"last update":"Mon Feb 19 18:59:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T18:59:38.341 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-19T18:59:38.341 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-19T18:59:38.341 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-19T18:59:38.342 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph health --format=json 2024-02-19T18:59:38.547 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:39.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:38 smithi114 ceph-mon[29441]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T18:59:39.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:38 smithi174 ceph-mon[41061]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T18:59:40.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.052+0000 7f8ca96a1700 1 -- 172.21.15.114:0/2192657072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ca4100de0 msgr2=0x7f8ca41011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:40.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.052+0000 7f8ca96a1700 1 --2- 172.21.15.114:0/2192657072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ca4100de0 0x7f8ca41011c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8c94009910 tx=0x7f8c94009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:40.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.052+0000 7f8ca96a1700 1 -- 172.21.15.114:0/2192657072 shutdown_connections 2024-02-19T18:59:40.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.052+0000 7f8ca96a1700 1 --2- 172.21.15.114:0/2192657072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ca4101790 0x7f8ca41057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:40.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.052+0000 7f8ca96a1700 1 --2- 172.21.15.114:0/2192657072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ca4100de0 0x7f8ca41011c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:40.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.052+0000 7f8ca96a1700 1 -- 172.21.15.114:0/2192657072 >> 172.21.15.114:0/2192657072 conn(0x7f8ca40fc6b0 msgr2=0x7f8ca40fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:40.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.053+0000 7f8ca96a1700 1 -- 172.21.15.114:0/2192657072 shutdown_connections 2024-02-19T18:59:40.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.053+0000 7f8ca96a1700 1 -- 172.21.15.114:0/2192657072 wait complete. 2024-02-19T18:59:40.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.053+0000 7f8ca96a1700 1 Processor -- start 2024-02-19T18:59:40.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.053+0000 7f8ca96a1700 1 -- start start 2024-02-19T18:59:40.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.053+0000 7f8ca96a1700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ca4100de0 0x7f8ca419c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:40.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.053+0000 7f8ca96a1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ca4101790 0x7f8ca419cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:40.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.053+0000 7f8ca96a1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca419d100 con 0x7f8ca4101790 2024-02-19T18:59:40.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.053+0000 7f8ca96a1700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca4196640 con 0x7f8ca4100de0 2024-02-19T18:59:40.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.054+0000 7f8ca27fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ca4101790 0x7f8ca419cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:40.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.054+0000 7f8ca27fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ca4101790 0x7f8ca419cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36078/0 (socket says 172.21.15.114:36078) 2024-02-19T18:59:40.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.054+0000 7f8ca27fc700 1 -- 172.21.15.114:0/1805105027 learned_addr learned my addr 172.21.15.114:0/1805105027 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:40.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.054+0000 7f8ca2ffd700 1 --2- 172.21.15.114:0/1805105027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ca4100de0 0x7f8ca419c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:40.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.054+0000 7f8ca27fc700 1 -- 172.21.15.114:0/1805105027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ca4100de0 msgr2=0x7f8ca419c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:40.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.054+0000 7f8ca27fc700 1 --2- 172.21.15.114:0/1805105027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ca4100de0 0x7f8ca419c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:40.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.054+0000 7f8ca27fc700 1 -- 172.21.15.114:0/1805105027 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c940095f0 con 0x7f8ca4101790 2024-02-19T18:59:40.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.054+0000 7f8ca27fc700 1 --2- 172.21.15.114:0/1805105027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ca4101790 0x7f8ca419cb00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8c9800c960 tx=0x7f8c9800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:40.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.054+0000 7f8c8bfff700 1 -- 172.21.15.114:0/1805105027 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c98007a70 con 0x7f8ca4101790 2024-02-19T18:59:40.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.054+0000 7f8ca96a1700 1 -- 172.21.15.114:0/1805105027 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca4196920 con 0x7f8ca4101790 2024-02-19T18:59:40.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.054+0000 7f8c8bfff700 1 -- 172.21.15.114:0/1805105027 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8c98007bd0 con 0x7f8ca4101790 2024-02-19T18:59:40.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.054+0000 7f8c8bfff700 1 -- 172.21.15.114:0/1805105027 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c98018630 con 0x7f8ca4101790 2024-02-19T18:59:40.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.054+0000 7f8ca96a1700 1 -- 172.21.15.114:0/1805105027 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca4196e70 con 0x7f8ca4101790 2024-02-19T18:59:40.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.055+0000 7f8ca96a1700 1 -- 172.21.15.114:0/1805105027 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca4109180 con 0x7f8ca4101790 2024-02-19T18:59:40.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.056+0000 7f8c8bfff700 1 -- 172.21.15.114:0/1805105027 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f8c9800f450 con 0x7f8ca4101790 2024-02-19T18:59:40.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.056+0000 7f8c8bfff700 1 --2- 172.21.15.114:0/1805105027 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8c8c06c4e0 0x7f8c8c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:40.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.056+0000 7f8ca2ffd700 1 --2- 172.21.15.114:0/1805105027 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8c8c06c4e0 0x7f8c8c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:40.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.056+0000 7f8c8bfff700 1 -- 172.21.15.114:0/1805105027 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8c9808b440 con 0x7f8ca4101790 2024-02-19T18:59:40.076 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.057+0000 7f8ca2ffd700 1 --2- 172.21.15.114:0/1805105027 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8c8c06c4e0 0x7f8c8c06e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8c94006010 tx=0x7f8c9401a040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:40.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.061+0000 7f8c8bfff700 1 -- 172.21.15.114:0/1805105027 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8c980599f0 con 0x7f8ca4101790 2024-02-19T18:59:40.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:39 smithi114 ceph-mon[29441]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:40.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.297+0000 7f8ca96a1700 1 -- 172.21.15.114:0/1805105027 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8ca404ea90 con 0x7f8ca4101790 2024-02-19T18:59:40.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.297+0000 7f8c8bfff700 1 -- 172.21.15.114:0/1805105027 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f8c98059580 con 0x7f8ca4101790 2024-02-19T18:59:40.315 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T18:59:40.315 INFO:teuthology.orchestra.run.smithi114.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-19T18:59:40.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.299+0000 7f8ca96a1700 1 -- 172.21.15.114:0/1805105027 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8c8c06c4e0 msgr2=0x7f8c8c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:40.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.299+0000 7f8ca96a1700 1 --2- 172.21.15.114:0/1805105027 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8c8c06c4e0 0x7f8c8c06e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8c94006010 tx=0x7f8c9401a040 comp rx=0 tx=0).stop 2024-02-19T18:59:40.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.299+0000 7f8ca96a1700 1 -- 172.21.15.114:0/1805105027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ca4101790 msgr2=0x7f8ca419cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:40.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.299+0000 7f8ca96a1700 1 --2- 172.21.15.114:0/1805105027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ca4101790 0x7f8ca419cb00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8c9800c960 tx=0x7f8c9800cc70 comp rx=0 tx=0).stop 2024-02-19T18:59:40.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.299+0000 7f8ca96a1700 1 -- 172.21.15.114:0/1805105027 shutdown_connections 2024-02-19T18:59:40.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.299+0000 7f8ca96a1700 1 --2- 172.21.15.114:0/1805105027 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f8c8c06c4e0 0x7f8c8c06e9a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:40.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.299+0000 7f8ca96a1700 1 --2- 172.21.15.114:0/1805105027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ca4100de0 0x7f8ca419c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:40.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.299+0000 7f8ca96a1700 1 --2- 172.21.15.114:0/1805105027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ca4101790 0x7f8ca419cb00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:40.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.299+0000 7f8ca96a1700 1 -- 172.21.15.114:0/1805105027 >> 172.21.15.114:0/1805105027 conn(0x7f8ca40fc6b0 msgr2=0x7f8ca40fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:40.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.299+0000 7f8ca96a1700 1 -- 172.21.15.114:0/1805105027 shutdown_connections 2024-02-19T18:59:40.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:40.299+0000 7f8ca96a1700 1 -- 172.21.15.114:0/1805105027 wait complete. 2024-02-19T18:59:40.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:39 smithi174 ceph-mon[41061]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:40.981 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-19T18:59:40.981 INFO:tasks.cephadm:Setup complete, yielding 2024-02-19T18:59:40.981 INFO:teuthology.run_tasks:Running task print... 2024-02-19T18:59:40.990 INFO:teuthology.task.print:**** done end installing v18.2.1 cephadm ... 2024-02-19T18:59:40.991 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T18:59:41.004 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-02-19T18:59:41.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-02-19T18:59:41.209 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:41.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:40 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1805105027' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-19T18:59:41.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:40 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1805105027' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-19T18:59:42.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:41 smithi114 ceph-mon[29441]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:42.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:41 smithi174 ceph-mon[41061]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:42.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.813+0000 7feb5cf27700 1 -- 172.21.15.114:0/4292561945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb58100bc0 msgr2=0x7feb58100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:42.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.813+0000 7feb5cf27700 1 --2- 172.21.15.114:0/4292561945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb58100bc0 0x7feb58100fe0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7feb48009910 tx=0x7feb48009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:42.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.815+0000 7feb5cf27700 1 -- 172.21.15.114:0/4292561945 shutdown_connections 2024-02-19T18:59:42.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.815+0000 7feb5cf27700 1 --2- 172.21.15.114:0/4292561945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb58101dc0 0x7feb58102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:42.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.815+0000 7feb5cf27700 1 --2- 172.21.15.114:0/4292561945 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb58100bc0 0x7feb58100fe0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:42.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.815+0000 7feb5cf27700 1 -- 172.21.15.114:0/4292561945 >> 172.21.15.114:0/4292561945 conn(0x7feb580fc140 msgr2=0x7feb580fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:42.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.815+0000 7feb5cf27700 1 -- 172.21.15.114:0/4292561945 shutdown_connections 2024-02-19T18:59:42.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.815+0000 7feb5cf27700 1 -- 172.21.15.114:0/4292561945 wait complete. 2024-02-19T18:59:42.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.815+0000 7feb5cf27700 1 Processor -- start 2024-02-19T18:59:42.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.815+0000 7feb5cf27700 1 -- start start 2024-02-19T18:59:42.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb5cf27700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb58100bc0 0x7feb58198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:42.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb5cf27700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb58101dc0 0x7feb58198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:42.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb5cf27700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb581990e0 con 0x7feb58100bc0 2024-02-19T18:59:42.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb5cf27700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb58199220 con 0x7feb58101dc0 2024-02-19T18:59:42.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb5659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb58100bc0 0x7feb58198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:42.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb5659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb58100bc0 0x7feb58198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36106/0 (socket says 172.21.15.114:36106) 2024-02-19T18:59:42.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb5659c700 1 -- 172.21.15.114:0/3015450042 learned_addr learned my addr 172.21.15.114:0/3015450042 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:42.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb4dd9b700 1 --2- 172.21.15.114:0/3015450042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb58101dc0 0x7feb58198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:42.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb5659c700 1 -- 172.21.15.114:0/3015450042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb58101dc0 msgr2=0x7feb58198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:42.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb5659c700 1 --2- 172.21.15.114:0/3015450042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb58101dc0 0x7feb58198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:42.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb5659c700 1 -- 172.21.15.114:0/3015450042 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb480095f0 con 0x7feb58100bc0 2024-02-19T18:59:42.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb4dd9b700 1 --2- 172.21.15.114:0/3015450042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb58101dc0 0x7feb58198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T18:59:42.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb5659c700 1 --2- 172.21.15.114:0/3015450042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb58100bc0 0x7feb58198580 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7feb4800bda0 tx=0x7feb4800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:42.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb4ffff700 1 -- 172.21.15.114:0/3015450042 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb4801d070 con 0x7feb58100bc0 2024-02-19T18:59:42.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb5cf27700 1 -- 172.21.15.114:0/3015450042 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb5819dc70 con 0x7feb58100bc0 2024-02-19T18:59:42.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb4ffff700 1 -- 172.21.15.114:0/3015450042 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7feb4800f460 con 0x7feb58100bc0 2024-02-19T18:59:42.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb4ffff700 1 -- 172.21.15.114:0/3015450042 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb48021640 con 0x7feb58100bc0 2024-02-19T18:59:42.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.816+0000 7feb5cf27700 1 -- 172.21.15.114:0/3015450042 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb5819e160 con 0x7feb58100bc0 2024-02-19T18:59:42.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.818+0000 7feb5cf27700 1 -- 172.21.15.114:0/3015450042 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb58105ea0 con 0x7feb58100bc0 2024-02-19T18:59:42.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.818+0000 7feb4ffff700 1 -- 172.21.15.114:0/3015450042 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7feb4800fa00 con 0x7feb58100bc0 2024-02-19T18:59:42.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.819+0000 7feb4ffff700 1 --2- 172.21.15.114:0/3015450042 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7feb4406c4e0 0x7feb4406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:42.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.819+0000 7feb4dd9b700 1 --2- 172.21.15.114:0/3015450042 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7feb4406c4e0 0x7feb4406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:42.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.819+0000 7feb4ffff700 1 -- 172.21.15.114:0/3015450042 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7feb4808c7e0 con 0x7feb58100bc0 2024-02-19T18:59:42.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.819+0000 7feb4dd9b700 1 --2- 172.21.15.114:0/3015450042 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7feb4406c4e0 0x7feb4406e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7feb40005950 tx=0x7feb400058e0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:42.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:42.823+0000 7feb4ffff700 1 -- 172.21.15.114:0/3015450042 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7feb4805ad90 con 0x7feb58100bc0 2024-02-19T18:59:43.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:43.016+0000 7feb5cf27700 1 -- 172.21.15.114:0/3015450042 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7feb5802cc70 con 0x7feb58100bc0 2024-02-19T18:59:43.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:43.020+0000 7feb4ffff700 1 -- 172.21.15.114:0/3015450042 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7feb4805a920 con 0x7feb58100bc0 2024-02-19T18:59:43.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:43.022+0000 7feb5cf27700 1 -- 172.21.15.114:0/3015450042 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7feb4406c4e0 msgr2=0x7feb4406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:43.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:43.022+0000 7feb5cf27700 1 --2- 172.21.15.114:0/3015450042 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7feb4406c4e0 0x7feb4406e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7feb40005950 tx=0x7feb400058e0 comp rx=0 tx=0).stop 2024-02-19T18:59:43.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:43.022+0000 7feb5cf27700 1 -- 172.21.15.114:0/3015450042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb58100bc0 msgr2=0x7feb58198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:43.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:43.022+0000 7feb5cf27700 1 --2- 172.21.15.114:0/3015450042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb58100bc0 0x7feb58198580 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7feb4800bda0 tx=0x7feb4800be80 comp rx=0 tx=0).stop 2024-02-19T18:59:43.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:43.022+0000 7feb5cf27700 1 -- 172.21.15.114:0/3015450042 shutdown_connections 2024-02-19T18:59:43.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:43.022+0000 7feb5cf27700 1 --2- 172.21.15.114:0/3015450042 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7feb4406c4e0 0x7feb4406e9a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:43.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:43.022+0000 7feb5cf27700 1 --2- 172.21.15.114:0/3015450042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb58100bc0 0x7feb58198580 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:43.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:43.022+0000 7feb5cf27700 1 --2- 172.21.15.114:0/3015450042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb58101dc0 0x7feb58198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:43.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:43.022+0000 7feb5cf27700 1 -- 172.21.15.114:0/3015450042 >> 172.21.15.114:0/3015450042 conn(0x7feb580fc140 msgr2=0x7feb58102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:43.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:43.022+0000 7feb5cf27700 1 -- 172.21.15.114:0/3015450042 shutdown_connections 2024-02-19T18:59:43.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:43.022+0000 7feb5cf27700 1 -- 172.21.15.114:0/3015450042 wait complete. 2024-02-19T18:59:43.652 INFO:teuthology.run_tasks:Running task print... 2024-02-19T18:59:43.662 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-02-19T18:59:43.662 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T18:59:43.671 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-02-19T18:59:43.671 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph orch status' 2024-02-19T18:59:43.885 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:44.219 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:43 smithi114 ceph-mon[29441]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:44.219 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:43 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/3015450042' entity='client.admin' 2024-02-19T18:59:44.219 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:43 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:59:44.220 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:43 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:44.220 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:43 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:59:44.220 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:43 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:59:44.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:43 smithi174 ceph-mon[41061]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:44.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:43 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/3015450042' entity='client.admin' 2024-02-19T18:59:44.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:43 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:59:44.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:43 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:44.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:43 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:59:44.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:43 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T18:59:45.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.405+0000 7f7b7f627700 1 -- 172.21.15.114:0/2076787196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b78103f70 msgr2=0x7f7b78106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:45.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.405+0000 7f7b7f627700 1 --2- 172.21.15.114:0/2076787196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b78103f70 0x7f7b78106360 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009910 tx=0x7f7b68009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:45.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.405+0000 7f7b7f627700 1 -- 172.21.15.114:0/2076787196 shutdown_connections 2024-02-19T18:59:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.405+0000 7f7b7f627700 1 --2- 172.21.15.114:0/2076787196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b78103f70 0x7f7b78106360 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.405+0000 7f7b7f627700 1 --2- 172.21.15.114:0/2076787196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b78101640 0x7f7b78103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.405+0000 7f7b7f627700 1 -- 172.21.15.114:0/2076787196 >> 172.21.15.114:0/2076787196 conn(0x7f7b780fb050 msgr2=0x7f7b780fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.405+0000 7f7b7f627700 1 -- 172.21.15.114:0/2076787196 shutdown_connections 2024-02-19T18:59:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.405+0000 7f7b7f627700 1 -- 172.21.15.114:0/2076787196 wait complete. 2024-02-19T18:59:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.406+0000 7f7b7f627700 1 Processor -- start 2024-02-19T18:59:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.406+0000 7f7b7f627700 1 -- start start 2024-02-19T18:59:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.406+0000 7f7b7f627700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b78101640 0x7f7b78196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.406+0000 7f7b7f627700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b78103f70 0x7f7b781968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.406+0000 7f7b7f627700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b78196ef0 con 0x7f7b78103f70 2024-02-19T18:59:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.406+0000 7f7b7f627700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b78197030 con 0x7f7b78101640 2024-02-19T18:59:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.406+0000 7f7b7cbc2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b78103f70 0x7f7b781968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.406+0000 7f7b7cbc2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b78103f70 0x7f7b781968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36130/0 (socket says 172.21.15.114:36130) 2024-02-19T18:59:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.406+0000 7f7b7cbc2700 1 -- 172.21.15.114:0/2211950661 learned_addr learned my addr 172.21.15.114:0/2211950661 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:45.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.406+0000 7f7b7d3c3700 1 --2- 172.21.15.114:0/2211950661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b78101640 0x7f7b78196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:45.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.406+0000 7f7b7cbc2700 1 -- 172.21.15.114:0/2211950661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b78101640 msgr2=0x7f7b78196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:45.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.406+0000 7f7b7cbc2700 1 --2- 172.21.15.114:0/2211950661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b78101640 0x7f7b78196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:45.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.406+0000 7f7b7cbc2700 1 -- 172.21.15.114:0/2211950661 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b680095f0 con 0x7f7b78103f70 2024-02-19T18:59:45.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.406+0000 7f7b7cbc2700 1 --2- 172.21.15.114:0/2211950661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b78103f70 0x7f7b781968d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7b68006010 tx=0x7f7b6800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:45.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.407+0000 7f7b6e7fc700 1 -- 172.21.15.114:0/2211950661 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b6801d070 con 0x7f7b78103f70 2024-02-19T18:59:45.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.407+0000 7f7b6e7fc700 1 -- 172.21.15.114:0/2211950661 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7b6800f460 con 0x7f7b78103f70 2024-02-19T18:59:45.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.407+0000 7f7b7f627700 1 -- 172.21.15.114:0/2211950661 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7819ba80 con 0x7f7b78103f70 2024-02-19T18:59:45.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.407+0000 7f7b6e7fc700 1 -- 172.21.15.114:0/2211950661 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b68021640 con 0x7f7b78103f70 2024-02-19T18:59:45.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.407+0000 7f7b7f627700 1 -- 172.21.15.114:0/2211950661 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7819bff0 con 0x7f7b78103f70 2024-02-19T18:59:45.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.408+0000 7f7b7f627700 1 -- 172.21.15.114:0/2211950661 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b781905a0 con 0x7f7b78103f70 2024-02-19T18:59:45.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.408+0000 7f7b6e7fc700 1 -- 172.21.15.114:0/2211950661 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f7b6800fab0 con 0x7f7b78103f70 2024-02-19T18:59:45.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.408+0000 7f7b6e7fc700 1 --2- 172.21.15.114:0/2211950661 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7b6406c4e0 0x7f7b6406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:45.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.409+0000 7f7b6e7fc700 1 -- 172.21.15.114:0/2211950661 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7b6808c970 con 0x7f7b78103f70 2024-02-19T18:59:45.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.409+0000 7f7b7d3c3700 1 --2- 172.21.15.114:0/2211950661 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7b6406c4e0 0x7f7b6406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:45.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.409+0000 7f7b7d3c3700 1 --2- 172.21.15.114:0/2211950661 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7b6406c4e0 0x7f7b6406e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7b70009f20 tx=0x7f7b700094f0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:45.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.413+0000 7f7b6e7fc700 1 -- 172.21.15.114:0/2211950661 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7b6805b040 con 0x7f7b78103f70 2024-02-19T18:59:45.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.619+0000 7f7b7f627700 1 -- 172.21.15.114:0/2211950661 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b780611d0 con 0x7f7b6406c4e0 2024-02-19T18:59:45.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.620+0000 7f7b6e7fc700 1 -- 172.21.15.114:0/2211950661 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f7b780611d0 con 0x7f7b6406c4e0 2024-02-19T18:59:45.638 INFO:teuthology.orchestra.run.smithi114.stdout:Backend: cephadm 2024-02-19T18:59:45.639 INFO:teuthology.orchestra.run.smithi114.stdout:Available: Yes 2024-02-19T18:59:45.639 INFO:teuthology.orchestra.run.smithi114.stdout:Paused: No 2024-02-19T18:59:45.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.622+0000 7f7b7f627700 1 -- 172.21.15.114:0/2211950661 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7b6406c4e0 msgr2=0x7f7b6406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:45.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.622+0000 7f7b7f627700 1 --2- 172.21.15.114:0/2211950661 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7b6406c4e0 0x7f7b6406e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7b70009f20 tx=0x7f7b700094f0 comp rx=0 tx=0).stop 2024-02-19T18:59:45.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.622+0000 7f7b7f627700 1 -- 172.21.15.114:0/2211950661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b78103f70 msgr2=0x7f7b781968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:45.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.622+0000 7f7b7f627700 1 --2- 172.21.15.114:0/2211950661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b78103f70 0x7f7b781968d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7b68006010 tx=0x7f7b6800be20 comp rx=0 tx=0).stop 2024-02-19T18:59:45.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.622+0000 7f7b7f627700 1 -- 172.21.15.114:0/2211950661 shutdown_connections 2024-02-19T18:59:45.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.622+0000 7f7b7f627700 1 --2- 172.21.15.114:0/2211950661 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7b6406c4e0 0x7f7b6406e9a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:45.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.622+0000 7f7b7f627700 1 --2- 172.21.15.114:0/2211950661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b78101640 0x7f7b78196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:45.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.622+0000 7f7b7f627700 1 --2- 172.21.15.114:0/2211950661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b78103f70 0x7f7b781968d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:45.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.622+0000 7f7b7f627700 1 -- 172.21.15.114:0/2211950661 >> 172.21.15.114:0/2211950661 conn(0x7f7b780fb050 msgr2=0x7f7b780fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:45.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.622+0000 7f7b7f627700 1 -- 172.21.15.114:0/2211950661 shutdown_connections 2024-02-19T18:59:45.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:45.622+0000 7f7b7f627700 1 -- 172.21.15.114:0/2211950661 wait complete. 2024-02-19T18:59:46.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:45 smithi114 ceph-mon[29441]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:46.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-19T18:59:46.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:45 smithi174 ceph-mon[41061]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:46.590 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:47.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:46 smithi114 ceph-mon[29441]: from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:47.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:46 smithi174 ceph-mon[41061]: from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:48.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.046+0000 7f7baf84a700 1 -- 172.21.15.114:0/441467899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ba8101e60 msgr2=0x7f7ba81022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:48.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.046+0000 7f7baf84a700 1 --2- 172.21.15.114:0/441467899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ba8101e60 0x7f7ba81022c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7b98009910 tx=0x7f7b98009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:48.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.046+0000 7f7baf84a700 1 -- 172.21.15.114:0/441467899 shutdown_connections 2024-02-19T18:59:48.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.046+0000 7f7baf84a700 1 --2- 172.21.15.114:0/441467899 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ba8101e60 0x7f7ba81022c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:48.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.046+0000 7f7baf84a700 1 --2- 172.21.15.114:0/441467899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ba80fee80 0x7f7ba80ff2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:48.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.046+0000 7f7baf84a700 1 -- 172.21.15.114:0/441467899 >> 172.21.15.114:0/441467899 conn(0x7f7ba80fa7c0 msgr2=0x7f7ba80fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:48.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.046+0000 7f7baf84a700 1 -- 172.21.15.114:0/441467899 shutdown_connections 2024-02-19T18:59:48.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.046+0000 7f7baf84a700 1 -- 172.21.15.114:0/441467899 wait complete. 2024-02-19T18:59:48.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.047+0000 7f7baf84a700 1 Processor -- start 2024-02-19T18:59:48.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.047+0000 7f7baf84a700 1 -- start start 2024-02-19T18:59:48.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.047+0000 7f7baf84a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ba80fee80 0x7f7ba8196350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:48.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.047+0000 7f7baf84a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ba8101e60 0x7f7ba8196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:48.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.047+0000 7f7baf84a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba8196eb0 con 0x7f7ba80fee80 2024-02-19T18:59:48.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.047+0000 7f7baf84a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba8196ff0 con 0x7f7ba8101e60 2024-02-19T18:59:48.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.047+0000 7f7bad5e6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ba80fee80 0x7f7ba8196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:48.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.047+0000 7f7bad5e6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ba80fee80 0x7f7ba8196350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33464/0 (socket says 172.21.15.114:33464) 2024-02-19T18:59:48.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.047+0000 7f7bad5e6700 1 -- 172.21.15.114:0/1913085963 learned_addr learned my addr 172.21.15.114:0/1913085963 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:48.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.047+0000 7f7bad5e6700 1 -- 172.21.15.114:0/1913085963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ba8101e60 msgr2=0x7f7ba8196890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:48.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.047+0000 7f7bad5e6700 1 --2- 172.21.15.114:0/1913085963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ba8101e60 0x7f7ba8196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:48.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.047+0000 7f7bad5e6700 1 -- 172.21.15.114:0/1913085963 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b980095f0 con 0x7f7ba80fee80 2024-02-19T18:59:48.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.048+0000 7f7bad5e6700 1 --2- 172.21.15.114:0/1913085963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ba80fee80 0x7f7ba8196350 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0007e90 tx=0x7f7ba000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:48.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.048+0000 7f7b9e7fc700 1 -- 172.21.15.114:0/1913085963 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ba0017070 con 0x7f7ba80fee80 2024-02-19T18:59:48.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.048+0000 7f7b9e7fc700 1 -- 172.21.15.114:0/1913085963 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7ba000f040 con 0x7f7ba80fee80 2024-02-19T18:59:48.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.048+0000 7f7baf84a700 1 -- 172.21.15.114:0/1913085963 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba819baa0 con 0x7f7ba80fee80 2024-02-19T18:59:48.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.048+0000 7f7b9e7fc700 1 -- 172.21.15.114:0/1913085963 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ba0004880 con 0x7f7ba80fee80 2024-02-19T18:59:48.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.048+0000 7f7baf84a700 1 -- 172.21.15.114:0/1913085963 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba819bff0 con 0x7f7ba80fee80 2024-02-19T18:59:48.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.049+0000 7f7baf84a700 1 -- 172.21.15.114:0/1913085963 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba8109720 con 0x7f7ba80fee80 2024-02-19T18:59:48.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.049+0000 7f7b9e7fc700 1 -- 172.21.15.114:0/1913085963 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f7ba00040b0 con 0x7f7ba80fee80 2024-02-19T18:59:48.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.050+0000 7f7b9e7fc700 1 --2- 172.21.15.114:0/1913085963 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7b9406c490 0x7f7b9406e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:48.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.050+0000 7f7bacde5700 1 --2- 172.21.15.114:0/1913085963 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7b9406c490 0x7f7b9406e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:48.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.050+0000 7f7b9e7fc700 1 -- 172.21.15.114:0/1913085963 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7ba008af60 con 0x7f7ba80fee80 2024-02-19T18:59:48.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.050+0000 7f7bacde5700 1 --2- 172.21.15.114:0/1913085963 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7b9406c490 0x7f7b9406e950 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800b5c0 tx=0x7f7b98005fd0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:48.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.054+0000 7f7b9e7fc700 1 -- 172.21.15.114:0/1913085963 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7ba00be0e0 con 0x7f7ba80fee80 2024-02-19T18:59:48.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.249+0000 7f7baf84a700 1 -- 172.21.15.114:0/1913085963 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ba81067b0 con 0x7f7b9406c490 2024-02-19T18:59:48.268 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:47 smithi114 ceph-mon[29441]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:48.268 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:47 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:48.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.259+0000 7f7b9e7fc700 1 -- 172.21.15.114:0/1913085963 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2820 (secure 0 0 0) 0x7f7ba81067b0 con 0x7f7b9406c490 2024-02-19T18:59:48.277 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T18:59:48.277 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (3m) 2m ago 6m 27.7M - 0.25.0 c8568f914cd2 30444b676ca4 2024-02-19T18:59:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (6m) 2m ago 6m 7084k - 18.2.1 94eadd990e0a 4918cd87e312 2024-02-19T18:59:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi174 smithi174 running (80s) 63s ago 4m 6941k - 18.2.1 94eadd990e0a 516c34431b93 2024-02-19T18:59:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (6m) 2m ago 6m 6660k - 18.2.1 94eadd990e0a ecfcd98907ba 2024-02-19T18:59:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi174 smithi174 running (4m) 63s ago 4m 6660k - 18.2.1 94eadd990e0a de88985f5a07 2024-02-19T18:59:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (3m) 2m ago 5m 80.5M - 9.4.7 954c08fa6188 478bd533e827 2024-02-19T18:59:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.pcvtrk smithi114 *:9283,8765,8443 running (8m) 2m ago 8m 481M - 18.2.1 94eadd990e0a 9d81af39d736 2024-02-19T18:59:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi174.lydvnj smithi174 *:8443,9283,8765 running (4m) 63s ago 4m 442M - 18.2.1 94eadd990e0a 9b31a7d679b9 2024-02-19T18:59:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (8m) 2m ago 8m 44.8M 2048M 18.2.1 94eadd990e0a 41468f811fc0 2024-02-19T18:59:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi174 smithi174 running (4m) 63s ago 4m 34.2M 2048M 18.2.1 94eadd990e0a 70dd7bf69fd9 2024-02-19T18:59:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (6m) 2m ago 6m 14.6M - 1.5.0 0da6a335fe13 b6b258074cce 2024-02-19T18:59:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi174 smithi174 *:9100 running (4m) 63s ago 4m 15.8M - 1.5.0 0da6a335fe13 d0b4fdf67b22 2024-02-19T18:59:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (3m) 2m ago 3m 39.4M 3946M 18.2.1 94eadd990e0a 990afc982b6c 2024-02-19T18:59:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (2m) 2m ago 2m 40.5M 3946M 18.2.1 94eadd990e0a 932e3ad37db5 2024-02-19T18:59:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (2m) 2m ago 2m 41.4M 3946M 18.2.1 94eadd990e0a 12953988af86 2024-02-19T18:59:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi174 running (2m) 63s ago 2m 40.3M 4970M 18.2.1 94eadd990e0a 979f45cabbce 2024-02-19T18:59:48.278 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi174 running (96s) 63s ago 96s 41.7M 4970M 18.2.1 94eadd990e0a 8b8c3745b8c5 2024-02-19T18:59:48.279 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi174 running (69s) 63s ago 68s 16.0M 4970M 18.2.1 94eadd990e0a 5771a995ed75 2024-02-19T18:59:48.279 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (3m) 2m ago 4m 37.8M - 2.43.0 a07b618ecd1d 52087f218c92 2024-02-19T18:59:48.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.261+0000 7f7baf84a700 1 -- 172.21.15.114:0/1913085963 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7b9406c490 msgr2=0x7f7b9406e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:48.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.261+0000 7f7baf84a700 1 --2- 172.21.15.114:0/1913085963 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7b9406c490 0x7f7b9406e950 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800b5c0 tx=0x7f7b98005fd0 comp rx=0 tx=0).stop 2024-02-19T18:59:48.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.261+0000 7f7baf84a700 1 -- 172.21.15.114:0/1913085963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ba80fee80 msgr2=0x7f7ba8196350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:48.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.261+0000 7f7baf84a700 1 --2- 172.21.15.114:0/1913085963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ba80fee80 0x7f7ba8196350 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0007e90 tx=0x7f7ba000d3b0 comp rx=0 tx=0).stop 2024-02-19T18:59:48.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.261+0000 7f7baf84a700 1 -- 172.21.15.114:0/1913085963 shutdown_connections 2024-02-19T18:59:48.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.261+0000 7f7baf84a700 1 --2- 172.21.15.114:0/1913085963 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7b9406c490 0x7f7b9406e950 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:48.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.261+0000 7f7baf84a700 1 --2- 172.21.15.114:0/1913085963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7ba80fee80 0x7f7ba8196350 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:48.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.261+0000 7f7baf84a700 1 --2- 172.21.15.114:0/1913085963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ba8101e60 0x7f7ba8196890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:48.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.261+0000 7f7baf84a700 1 -- 172.21.15.114:0/1913085963 >> 172.21.15.114:0/1913085963 conn(0x7f7ba80fa7c0 msgr2=0x7f7ba8105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:48.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.262+0000 7f7baf84a700 1 -- 172.21.15.114:0/1913085963 shutdown_connections 2024-02-19T18:59:48.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:48.262+0000 7f7baf84a700 1 -- 172.21.15.114:0/1913085963 wait complete. 2024-02-19T18:59:48.294 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:47 smithi174 ceph-mon[41061]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:48.294 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:47 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:48.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-19T18:59:49.142 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:50.182 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:49 smithi114 ceph-mon[29441]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:50.182 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:49 smithi114 ceph-mon[29441]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:50.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:49 smithi174 ceph-mon[41061]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:50.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:49 smithi174 ceph-mon[41061]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:50.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.652+0000 7f397995c700 1 -- 172.21.15.114:0/1508482801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3974101dc0 msgr2=0x7f3974102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:50.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.652+0000 7f397995c700 1 --2- 172.21.15.114:0/1508482801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3974101dc0 0x7f3974102220 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f395c009910 tx=0x7f395c009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:50.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.652+0000 7f397995c700 1 -- 172.21.15.114:0/1508482801 shutdown_connections 2024-02-19T18:59:50.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.652+0000 7f397995c700 1 --2- 172.21.15.114:0/1508482801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3974101dc0 0x7f3974102220 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:50.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.652+0000 7f397995c700 1 --2- 172.21.15.114:0/1508482801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3974100bc0 0x7f3974100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:50.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.652+0000 7f397995c700 1 -- 172.21.15.114:0/1508482801 >> 172.21.15.114:0/1508482801 conn(0x7f39740fc140 msgr2=0x7f39740fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:50.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.652+0000 7f397995c700 1 -- 172.21.15.114:0/1508482801 shutdown_connections 2024-02-19T18:59:50.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.652+0000 7f397995c700 1 -- 172.21.15.114:0/1508482801 wait complete. 2024-02-19T18:59:50.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.653+0000 7f397995c700 1 Processor -- start 2024-02-19T18:59:50.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.653+0000 7f397995c700 1 -- start start 2024-02-19T18:59:50.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.653+0000 7f397995c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3974100bc0 0x7f3974196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:50.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.653+0000 7f397995c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3974101dc0 0x7f39741968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:50.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.653+0000 7f397995c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3974196e40 con 0x7f3974101dc0 2024-02-19T18:59:50.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.653+0000 7f397995c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3974196f80 con 0x7f3974100bc0 2024-02-19T18:59:50.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.653+0000 7f39727fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3974101dc0 0x7f39741968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:50.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.653+0000 7f39727fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3974101dc0 0x7f39741968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33484/0 (socket says 172.21.15.114:33484) 2024-02-19T18:59:50.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.653+0000 7f39727fc700 1 -- 172.21.15.114:0/1669960347 learned_addr learned my addr 172.21.15.114:0/1669960347 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:50.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.653+0000 7f3972ffd700 1 --2- 172.21.15.114:0/1669960347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3974100bc0 0x7f3974196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:50.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.653+0000 7f39727fc700 1 -- 172.21.15.114:0/1669960347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3974100bc0 msgr2=0x7f3974196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:50.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.653+0000 7f39727fc700 1 --2- 172.21.15.114:0/1669960347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3974100bc0 0x7f3974196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:50.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.653+0000 7f39727fc700 1 -- 172.21.15.114:0/1669960347 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395c0095f0 con 0x7f3974101dc0 2024-02-19T18:59:50.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.653+0000 7f39727fc700 1 --2- 172.21.15.114:0/1669960347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3974101dc0 0x7f39741968b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f395c00bd70 tx=0x7f395c00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:50.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.654+0000 7f397895a700 1 -- 172.21.15.114:0/1669960347 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f395c01d070 con 0x7f3974101dc0 2024-02-19T18:59:50.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.654+0000 7f397995c700 1 -- 172.21.15.114:0/1669960347 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397419b9e0 con 0x7f3974101dc0 2024-02-19T18:59:50.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.654+0000 7f397895a700 1 -- 172.21.15.114:0/1669960347 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f395c005940 con 0x7f3974101dc0 2024-02-19T18:59:50.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.654+0000 7f397895a700 1 -- 172.21.15.114:0/1669960347 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f395c021430 con 0x7f3974101dc0 2024-02-19T18:59:50.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.654+0000 7f397995c700 1 -- 172.21.15.114:0/1669960347 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397419bea0 con 0x7f3974101dc0 2024-02-19T18:59:50.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.655+0000 7f397995c700 1 -- 172.21.15.114:0/1669960347 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3974109720 con 0x7f3974101dc0 2024-02-19T18:59:50.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.655+0000 7f397895a700 1 -- 172.21.15.114:0/1669960347 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f395c02b440 con 0x7f3974101dc0 2024-02-19T18:59:50.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.655+0000 7f397895a700 1 --2- 172.21.15.114:0/1669960347 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f396006c4e0 0x7f396006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:50.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.655+0000 7f397895a700 1 -- 172.21.15.114:0/1669960347 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f395c08c930 con 0x7f3974101dc0 2024-02-19T18:59:50.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.656+0000 7f3972ffd700 1 --2- 172.21.15.114:0/1669960347 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f396006c4e0 0x7f396006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:50.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.656+0000 7f3972ffd700 1 --2- 172.21.15.114:0/1669960347 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f396006c4e0 0x7f396006e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3974101c20 tx=0x7f396400a3b0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:50.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.660+0000 7f397895a700 1 -- 172.21.15.114:0/1669960347 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f395c05af50 con 0x7f3974101dc0 2024-02-19T18:59:50.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.858+0000 7f397995c700 1 -- 172.21.15.114:0/1669960347 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3974106710 con 0x7f396006c4e0 2024-02-19T18:59:50.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.863+0000 7f397895a700 1 -- 172.21.15.114:0/1669960347 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1310 (secure 0 0 0) 0x7f3974106710 con 0x7f396006c4e0 2024-02-19T18:59:50.881 INFO:teuthology.orchestra.run.smithi114.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-19T18:59:50.881 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager ?:9093,9094 1/1 2m ago 7m count:1 2024-02-19T18:59:50.881 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter 2/2 2m ago 7m * 2024-02-19T18:59:50.881 INFO:teuthology.orchestra.run.smithi114.stdout:crash 2/2 2m ago 7m * 2024-02-19T18:59:50.881 INFO:teuthology.orchestra.run.smithi114.stdout:grafana ?:3000 1/1 2m ago 7m count:1 2024-02-19T18:59:50.881 INFO:teuthology.orchestra.run.smithi114.stdout:mgr 2/2 2m ago 7m count:2 2024-02-19T18:59:50.881 INFO:teuthology.orchestra.run.smithi114.stdout:mon 2/2 2m ago 6m smithi114:172.21.15.114=smithi114;smithi174:172.21.15.174=smithi174;count:2 2024-02-19T18:59:50.881 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter ?:9100 2/2 2m ago 7m * 2024-02-19T18:59:50.881 INFO:teuthology.orchestra.run.smithi114.stdout:osd 6 2m ago - 2024-02-19T18:59:50.882 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus ?:9095 1/1 2m ago 7m count:1 2024-02-19T18:59:50.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.864+0000 7f397995c700 1 -- 172.21.15.114:0/1669960347 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f396006c4e0 msgr2=0x7f396006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:50.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.864+0000 7f397995c700 1 --2- 172.21.15.114:0/1669960347 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f396006c4e0 0x7f396006e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3974101c20 tx=0x7f396400a3b0 comp rx=0 tx=0).stop 2024-02-19T18:59:50.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.864+0000 7f397995c700 1 -- 172.21.15.114:0/1669960347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3974101dc0 msgr2=0x7f39741968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:50.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.864+0000 7f397995c700 1 --2- 172.21.15.114:0/1669960347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3974101dc0 0x7f39741968b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f395c00bd70 tx=0x7f395c00bda0 comp rx=0 tx=0).stop 2024-02-19T18:59:50.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.865+0000 7f397995c700 1 -- 172.21.15.114:0/1669960347 shutdown_connections 2024-02-19T18:59:50.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.865+0000 7f397995c700 1 --2- 172.21.15.114:0/1669960347 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f396006c4e0 0x7f396006e9a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:50.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.865+0000 7f397995c700 1 --2- 172.21.15.114:0/1669960347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3974100bc0 0x7f3974196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:50.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.865+0000 7f397995c700 1 --2- 172.21.15.114:0/1669960347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3974101dc0 0x7f39741968b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:50.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.865+0000 7f397995c700 1 -- 172.21.15.114:0/1669960347 >> 172.21.15.114:0/1669960347 conn(0x7f39740fc140 msgr2=0x7f3974104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:50.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.865+0000 7f397995c700 1 -- 172.21.15.114:0/1669960347 shutdown_connections 2024-02-19T18:59:50.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:50.865+0000 7f397995c700 1 -- 172.21.15.114:0/1669960347 wait complete. 2024-02-19T18:59:51.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-19T18:59:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:52.109 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:51 smithi114 ceph-mon[29441]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:52.109 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:51 smithi114 ceph-mon[29441]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:52.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:51 smithi174 ceph-mon[41061]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:52.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:51 smithi174 ceph-mon[41061]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:53.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.317+0000 7fbeae4c7700 1 -- 172.21.15.114:0/268863582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbea80ffae0 msgr2=0x7fbea80fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:53.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.317+0000 7fbeae4c7700 1 --2- 172.21.15.114:0/268863582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbea80ffae0 0x7fbea80fff00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbe98009910 tx=0x7fbe98009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:53.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.317+0000 7fbeae4c7700 1 -- 172.21.15.114:0/268863582 shutdown_connections 2024-02-19T18:59:53.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.317+0000 7fbeae4c7700 1 --2- 172.21.15.114:0/268863582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbea8100ce0 0x7fbea8101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:53.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.317+0000 7fbeae4c7700 1 --2- 172.21.15.114:0/268863582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbea80ffae0 0x7fbea80fff00 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:53.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.317+0000 7fbeae4c7700 1 -- 172.21.15.114:0/268863582 >> 172.21.15.114:0/268863582 conn(0x7fbea80fb080 msgr2=0x7fbea80fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:53.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.317+0000 7fbeae4c7700 1 -- 172.21.15.114:0/268863582 shutdown_connections 2024-02-19T18:59:53.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.317+0000 7fbeae4c7700 1 -- 172.21.15.114:0/268863582 wait complete. 2024-02-19T18:59:53.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.318+0000 7fbeae4c7700 1 Processor -- start 2024-02-19T18:59:53.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.318+0000 7fbeae4c7700 1 -- start start 2024-02-19T18:59:53.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.318+0000 7fbeae4c7700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbea80ffae0 0x7fbea81985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:53.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.318+0000 7fbeae4c7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbea8100ce0 0x7fbea8198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:53.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.318+0000 7fbeae4c7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea8199130 con 0x7fbea8100ce0 2024-02-19T18:59:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.318+0000 7fbeae4c7700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea8199270 con 0x7fbea80ffae0 2024-02-19T18:59:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.318+0000 7fbea77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbea8100ce0 0x7fbea8198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.318+0000 7fbea77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbea8100ce0 0x7fbea8198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33500/0 (socket says 172.21.15.114:33500) 2024-02-19T18:59:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.318+0000 7fbea77fe700 1 -- 172.21.15.114:0/1558201532 learned_addr learned my addr 172.21.15.114:0/1558201532 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.318+0000 7fbea77fe700 1 -- 172.21.15.114:0/1558201532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbea80ffae0 msgr2=0x7fbea81985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.318+0000 7fbea77fe700 1 --2- 172.21.15.114:0/1558201532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbea80ffae0 0x7fbea81985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.318+0000 7fbea77fe700 1 -- 172.21.15.114:0/1558201532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe980095f0 con 0x7fbea8100ce0 2024-02-19T18:59:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.318+0000 7fbea77fe700 1 --2- 172.21.15.114:0/1558201532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbea8100ce0 0x7fbea8198b10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c00d8d0 tx=0x7fbe9c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.319+0000 7fbea57fa700 1 -- 172.21.15.114:0/1558201532 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe9c009940 con 0x7fbea8100ce0 2024-02-19T18:59:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.319+0000 7fbeae4c7700 1 -- 172.21.15.114:0/1558201532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea819dcd0 con 0x7fbea8100ce0 2024-02-19T18:59:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.319+0000 7fbea57fa700 1 -- 172.21.15.114:0/1558201532 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbe9c010460 con 0x7fbea8100ce0 2024-02-19T18:59:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.319+0000 7fbea57fa700 1 -- 172.21.15.114:0/1558201532 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe9c00f5f0 con 0x7fbea8100ce0 2024-02-19T18:59:53.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.319+0000 7fbeae4c7700 1 -- 172.21.15.114:0/1558201532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea819e220 con 0x7fbea8100ce0 2024-02-19T18:59:53.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.320+0000 7fbeae4c7700 1 -- 172.21.15.114:0/1558201532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea8108640 con 0x7fbea8100ce0 2024-02-19T18:59:53.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.320+0000 7fbea57fa700 1 -- 172.21.15.114:0/1558201532 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fbe9c0105d0 con 0x7fbea8100ce0 2024-02-19T18:59:53.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.320+0000 7fbea57fa700 1 --2- 172.21.15.114:0/1558201532 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fbe9006c4e0 0x7fbe9006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:53.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.321+0000 7fbea57fa700 1 -- 172.21.15.114:0/1558201532 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fbe9c08b4d0 con 0x7fbea8100ce0 2024-02-19T18:59:53.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.321+0000 7fbea7fff700 1 --2- 172.21.15.114:0/1558201532 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fbe9006c4e0 0x7fbe9006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:53.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.321+0000 7fbea7fff700 1 --2- 172.21.15.114:0/1558201532 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fbe9006c4e0 0x7fbe9006e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbe9800b5c0 tx=0x7fbe98005960 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:53.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.325+0000 7fbea57fa700 1 -- 172.21.15.114:0/1558201532 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbe9c059a80 con 0x7fbea8100ce0 2024-02-19T18:59:53.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.518+0000 7fbeae4c7700 1 -- 172.21.15.114:0/1558201532 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fbea8069440 con 0x7fbe9006c4e0 2024-02-19T18:59:53.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.520+0000 7fbea57fa700 1 -- 172.21.15.114:0/1558201532 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7fbea8069440 con 0x7fbe9006c4e0 2024-02-19T18:59:53.538 INFO:teuthology.orchestra.run.smithi114.stdout:HOST ADDR LABELS STATUS 2024-02-19T18:59:53.538 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114 172.21.15.114 2024-02-19T18:59:53.538 INFO:teuthology.orchestra.run.smithi114.stdout:smithi174 172.21.15.174 2024-02-19T18:59:53.538 INFO:teuthology.orchestra.run.smithi114.stdout:2 hosts in cluster 2024-02-19T18:59:53.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.522+0000 7fbeae4c7700 1 -- 172.21.15.114:0/1558201532 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fbe9006c4e0 msgr2=0x7fbe9006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:53.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.522+0000 7fbeae4c7700 1 --2- 172.21.15.114:0/1558201532 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fbe9006c4e0 0x7fbe9006e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbe9800b5c0 tx=0x7fbe98005960 comp rx=0 tx=0).stop 2024-02-19T18:59:53.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.522+0000 7fbeae4c7700 1 -- 172.21.15.114:0/1558201532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbea8100ce0 msgr2=0x7fbea8198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:53.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.522+0000 7fbeae4c7700 1 --2- 172.21.15.114:0/1558201532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbea8100ce0 0x7fbea8198b10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c00d8d0 tx=0x7fbe9c00dc90 comp rx=0 tx=0).stop 2024-02-19T18:59:53.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.522+0000 7fbeae4c7700 1 -- 172.21.15.114:0/1558201532 shutdown_connections 2024-02-19T18:59:53.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.522+0000 7fbeae4c7700 1 --2- 172.21.15.114:0/1558201532 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fbe9006c4e0 0x7fbe9006e9a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:53.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.522+0000 7fbeae4c7700 1 --2- 172.21.15.114:0/1558201532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbea80ffae0 0x7fbea81985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:53.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.522+0000 7fbeae4c7700 1 --2- 172.21.15.114:0/1558201532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbea8100ce0 0x7fbea8198b10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:53.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.522+0000 7fbeae4c7700 1 -- 172.21.15.114:0/1558201532 >> 172.21.15.114:0/1558201532 conn(0x7fbea80fb080 msgr2=0x7fbea8068d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:53.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.522+0000 7fbeae4c7700 1 -- 172.21.15.114:0/1558201532 shutdown_connections 2024-02-19T18:59:53.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:53.522+0000 7fbeae4c7700 1 -- 172.21.15.114:0/1558201532 wait complete. 2024-02-19T18:59:54.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:53 smithi114 ceph-mon[29441]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:54.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-19T18:59:54.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:53 smithi174 ceph-mon[41061]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:54.514 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:55.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:54 smithi114 ceph-mon[29441]: from='client.14498 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:55.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:54 smithi174 ceph-mon[41061]: from='client.14498 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:56.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.079+0000 7fb0be0ba700 1 -- 172.21.15.114:0/3693940466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b8100bc0 msgr2=0x7fb0b8100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:56.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.079+0000 7fb0be0ba700 1 --2- 172.21.15.114:0/3693940466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b8100bc0 0x7fb0b8100fe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009910 tx=0x7fb0a8009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:56.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.079+0000 7fb0be0ba700 1 -- 172.21.15.114:0/3693940466 shutdown_connections 2024-02-19T18:59:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.079+0000 7fb0be0ba700 1 --2- 172.21.15.114:0/3693940466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0b8101dc0 0x7fb0b8102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.079+0000 7fb0be0ba700 1 --2- 172.21.15.114:0/3693940466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b8100bc0 0x7fb0b8100fe0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.079+0000 7fb0be0ba700 1 -- 172.21.15.114:0/3693940466 >> 172.21.15.114:0/3693940466 conn(0x7fb0b80fc140 msgr2=0x7fb0b80fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.080+0000 7fb0be0ba700 1 -- 172.21.15.114:0/3693940466 shutdown_connections 2024-02-19T18:59:56.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.080+0000 7fb0be0ba700 1 -- 172.21.15.114:0/3693940466 wait complete. 2024-02-19T18:59:56.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.080+0000 7fb0be0ba700 1 Processor -- start 2024-02-19T18:59:56.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0be0ba700 1 -- start start 2024-02-19T18:59:56.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0be0ba700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b8100bc0 0x7fb0b8196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:56.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0be0ba700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0b8101dc0 0x7fb0b81968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:56.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0be0ba700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b8196e60 con 0x7fb0b8100bc0 2024-02-19T18:59:56.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0be0ba700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b8196fa0 con 0x7fb0b8101dc0 2024-02-19T18:59:56.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0b77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b8100bc0 0x7fb0b8196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:56.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0b77fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b8100bc0 0x7fb0b8196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35316/0 (socket says 172.21.15.114:35316) 2024-02-19T18:59:56.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0b77fe700 1 -- 172.21.15.114:0/3156780656 learned_addr learned my addr 172.21.15.114:0/3156780656 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:56.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0b6ffd700 1 --2- 172.21.15.114:0/3156780656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0b8101dc0 0x7fb0b81968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:56.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0b77fe700 1 -- 172.21.15.114:0/3156780656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0b8101dc0 msgr2=0x7fb0b81968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:56.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0b77fe700 1 --2- 172.21.15.114:0/3156780656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0b8101dc0 0x7fb0b81968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:56.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0b77fe700 1 -- 172.21.15.114:0/3156780656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a80095f0 con 0x7fb0b8100bc0 2024-02-19T18:59:56.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0b77fe700 1 --2- 172.21.15.114:0/3156780656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b8100bc0 0x7fb0b8196390 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800bd70 tx=0x7fb0a800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:56.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0b4ff9700 1 -- 172.21.15.114:0/3156780656 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0a801d070 con 0x7fb0b8100bc0 2024-02-19T18:59:56.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0b4ff9700 1 -- 172.21.15.114:0/3156780656 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb0a800f460 con 0x7fb0b8100bc0 2024-02-19T18:59:56.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0be0ba700 1 -- 172.21.15.114:0/3156780656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b819ba00 con 0x7fb0b8100bc0 2024-02-19T18:59:56.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0b4ff9700 1 -- 172.21.15.114:0/3156780656 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0a8021640 con 0x7fb0b8100bc0 2024-02-19T18:59:56.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.081+0000 7fb0be0ba700 1 -- 172.21.15.114:0/3156780656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b819bec0 con 0x7fb0b8100bc0 2024-02-19T18:59:56.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.082+0000 7fb0be0ba700 1 -- 172.21.15.114:0/3156780656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b8109720 con 0x7fb0b8100bc0 2024-02-19T18:59:56.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.083+0000 7fb0b4ff9700 1 -- 172.21.15.114:0/3156780656 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fb0a800f5d0 con 0x7fb0b8100bc0 2024-02-19T18:59:56.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.083+0000 7fb0b4ff9700 1 --2- 172.21.15.114:0/3156780656 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb0a006c4e0 0x7fb0a006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:56.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.083+0000 7fb0b6ffd700 1 --2- 172.21.15.114:0/3156780656 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb0a006c4e0 0x7fb0a006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:56.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.083+0000 7fb0b4ff9700 1 -- 172.21.15.114:0/3156780656 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb0a808c950 con 0x7fb0b8100bc0 2024-02-19T18:59:56.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.084+0000 7fb0b6ffd700 1 --2- 172.21.15.114:0/3156780656 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb0a006c4e0 0x7fb0a006e9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac005ea0 tx=0x7fb0ac005e30 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:56.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.088+0000 7fb0b4ff9700 1 -- 172.21.15.114:0/3156780656 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb0a805af00 con 0x7fb0b8100bc0 2024-02-19T18:59:56.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:55 smithi114 ceph-mon[29441]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:56.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.286+0000 7fb0be0ba700 1 -- 172.21.15.114:0/3156780656 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0b8106710 con 0x7fb0a006c4e0 2024-02-19T18:59:56.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.287+0000 7fb0b4ff9700 1 -- 172.21.15.114:0/3156780656 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7fb0b8106710 con 0x7fb0a006c4e0 2024-02-19T18:59:56.306 INFO:teuthology.orchestra.run.smithi114.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-02-19T18:59:56.306 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400WB400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-02-19T18:59:56.306 INFO:teuthology.orchestra.run.smithi114.stdout:smithi174 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62050074400BGN 372G No 68s ago Has a FileSystem, LVM detected 2024-02-19T18:59:56.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.289+0000 7fb0be0ba700 1 -- 172.21.15.114:0/3156780656 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb0a006c4e0 msgr2=0x7fb0a006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:56.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.289+0000 7fb0be0ba700 1 --2- 172.21.15.114:0/3156780656 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb0a006c4e0 0x7fb0a006e9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac005ea0 tx=0x7fb0ac005e30 comp rx=0 tx=0).stop 2024-02-19T18:59:56.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.289+0000 7fb0be0ba700 1 -- 172.21.15.114:0/3156780656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b8100bc0 msgr2=0x7fb0b8196390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:56.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.289+0000 7fb0be0ba700 1 --2- 172.21.15.114:0/3156780656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b8100bc0 0x7fb0b8196390 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800bd70 tx=0x7fb0a800be50 comp rx=0 tx=0).stop 2024-02-19T18:59:56.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.289+0000 7fb0be0ba700 1 -- 172.21.15.114:0/3156780656 shutdown_connections 2024-02-19T18:59:56.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.289+0000 7fb0be0ba700 1 --2- 172.21.15.114:0/3156780656 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fb0a006c4e0 0x7fb0a006e9a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:56.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.289+0000 7fb0be0ba700 1 --2- 172.21.15.114:0/3156780656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0b8100bc0 0x7fb0b8196390 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:56.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.289+0000 7fb0be0ba700 1 --2- 172.21.15.114:0/3156780656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0b8101dc0 0x7fb0b81968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:56.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.290+0000 7fb0be0ba700 1 -- 172.21.15.114:0/3156780656 >> 172.21.15.114:0/3156780656 conn(0x7fb0b80fc140 msgr2=0x7fb0b8104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:56.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.290+0000 7fb0be0ba700 1 -- 172.21.15.114:0/3156780656 shutdown_connections 2024-02-19T18:59:56.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:56.290+0000 7fb0be0ba700 1 -- 172.21.15.114:0/3156780656 wait complete. 2024-02-19T18:59:56.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:55 smithi174 ceph-mon[41061]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:56.957 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-02-19T18:59:56.967 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T18:59:56.977 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-02-19T18:59:56.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-02-19T18:59:57.185 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:56 smithi114 ceph-mon[29441]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:57.185 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:56 smithi114 ceph-mon[29441]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:57.185 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T18:59:57.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:56 smithi174 ceph-mon[41061]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:57.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:56 smithi174 ceph-mon[41061]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T18:59:58.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.751+0000 7f007dccd700 1 -- 172.21.15.114:0/1243795183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0078101dc0 msgr2=0x7f0078102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:58.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.751+0000 7f007dccd700 1 --2- 172.21.15.114:0/1243795183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0078101dc0 0x7f0078102220 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f006c009910 tx=0x7f006c009c20 comp rx=0 tx=0).stop 2024-02-19T18:59:58.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.751+0000 7f007dccd700 1 -- 172.21.15.114:0/1243795183 shutdown_connections 2024-02-19T18:59:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.751+0000 7f007dccd700 1 --2- 172.21.15.114:0/1243795183 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0078101dc0 0x7f0078102220 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.751+0000 7f007dccd700 1 --2- 172.21.15.114:0/1243795183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0078100bc0 0x7f0078100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.751+0000 7f007dccd700 1 -- 172.21.15.114:0/1243795183 >> 172.21.15.114:0/1243795183 conn(0x7f00780fc140 msgr2=0x7f00780fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.751+0000 7f007dccd700 1 -- 172.21.15.114:0/1243795183 shutdown_connections 2024-02-19T18:59:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.751+0000 7f007dccd700 1 -- 172.21.15.114:0/1243795183 wait complete. 2024-02-19T18:59:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.751+0000 7f007dccd700 1 Processor -- start 2024-02-19T18:59:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.752+0000 7f007dccd700 1 -- start start 2024-02-19T18:59:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.752+0000 7f007dccd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0078100bc0 0x7f0078196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.752+0000 7f007dccd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0078101dc0 0x7f00781968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.752+0000 7f007dccd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0078196ed0 con 0x7f0078100bc0 2024-02-19T18:59:58.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.752+0000 7f007dccd700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0078197010 con 0x7f0078101dc0 2024-02-19T18:59:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.752+0000 7f00777fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0078100bc0 0x7f0078196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.752+0000 7f00777fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0078100bc0 0x7f0078196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35334/0 (socket says 172.21.15.114:35334) 2024-02-19T18:59:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.752+0000 7f00777fe700 1 -- 172.21.15.114:0/3711988512 learned_addr learned my addr 172.21.15.114:0/3711988512 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T18:59:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.752+0000 7f0076ffd700 1 --2- 172.21.15.114:0/3711988512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0078101dc0 0x7f00781968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.752+0000 7f00777fe700 1 -- 172.21.15.114:0/3711988512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0078101dc0 msgr2=0x7f00781968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.752+0000 7f00777fe700 1 --2- 172.21.15.114:0/3711988512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0078101dc0 0x7f00781968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.752+0000 7f00777fe700 1 -- 172.21.15.114:0/3711988512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c0095f0 con 0x7f0078100bc0 2024-02-19T18:59:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.752+0000 7f00777fe700 1 --2- 172.21.15.114:0/3711988512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0078100bc0 0x7f0078196370 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f006800d900 tx=0x7f006800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.753+0000 7f0074ff9700 1 -- 172.21.15.114:0/3711988512 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00680041f0 con 0x7f0078100bc0 2024-02-19T18:59:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.753+0000 7f0074ff9700 1 -- 172.21.15.114:0/3711988512 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0068004350 con 0x7f0078100bc0 2024-02-19T18:59:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.753+0000 7f0074ff9700 1 -- 172.21.15.114:0/3711988512 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0068003d20 con 0x7f0078100bc0 2024-02-19T18:59:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.753+0000 7f007dccd700 1 -- 172.21.15.114:0/3711988512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f007819bac0 con 0x7f0078100bc0 2024-02-19T18:59:58.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.753+0000 7f007dccd700 1 -- 172.21.15.114:0/3711988512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f007819c010 con 0x7f0078100bc0 2024-02-19T18:59:58.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.754+0000 7f007dccd700 1 -- 172.21.15.114:0/3711988512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0078109720 con 0x7f0078100bc0 2024-02-19T18:59:58.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.754+0000 7f0074ff9700 1 -- 172.21.15.114:0/3711988512 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f0068009730 con 0x7f0078100bc0 2024-02-19T18:59:58.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.754+0000 7f0074ff9700 1 --2- 172.21.15.114:0/3711988512 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f006006c4e0 0x7f006006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:58.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.755+0000 7f0074ff9700 1 -- 172.21.15.114:0/3711988512 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f006808b2f0 con 0x7f0078100bc0 2024-02-19T18:59:58.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.755+0000 7f0076ffd700 1 --2- 172.21.15.114:0/3711988512 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f006006c4e0 0x7f006006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:58.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.755+0000 7f0076ffd700 1 --2- 172.21.15.114:0/3711988512 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f006006c4e0 0x7f006006e9a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f006c00b5c0 tx=0x7f006c005ee0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:58.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.759+0000 7f0074ff9700 1 -- 172.21.15.114:0/3711988512 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0068059920 con 0x7f0078100bc0 2024-02-19T18:59:58.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T18:59:58.975+0000 7f007dccd700 1 -- 172.21.15.114:0/3711988512 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0078106710 con 0x7f006006c4e0 2024-02-19T19:00:00.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:59 smithi114 ceph-mon[29441]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:00.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 18:59:59 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-19T19:00:00.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:59 smithi174 ceph-mon[41061]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:00.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 18:59:59 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-19T19:00:01.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:00.996+0000 7f0074ff9700 1 -- 172.21.15.114:0/3711988512 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0078106710 con 0x7f006006c4e0 2024-02-19T19:00:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:01.000+0000 7f007dccd700 1 -- 172.21.15.114:0/3711988512 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f006006c4e0 msgr2=0x7f006006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:01.000+0000 7f007dccd700 1 --2- 172.21.15.114:0/3711988512 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f006006c4e0 0x7f006006e9a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f006c00b5c0 tx=0x7f006c005ee0 comp rx=0 tx=0).stop 2024-02-19T19:00:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:01.000+0000 7f007dccd700 1 -- 172.21.15.114:0/3711988512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0078100bc0 msgr2=0x7f0078196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:01.000+0000 7f007dccd700 1 --2- 172.21.15.114:0/3711988512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0078100bc0 0x7f0078196370 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f006800d900 tx=0x7f006800dc10 comp rx=0 tx=0).stop 2024-02-19T19:00:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:01.000+0000 7f007dccd700 1 -- 172.21.15.114:0/3711988512 shutdown_connections 2024-02-19T19:00:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:01.000+0000 7f007dccd700 1 --2- 172.21.15.114:0/3711988512 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f006006c4e0 0x7f006006e9a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:01.000+0000 7f007dccd700 1 --2- 172.21.15.114:0/3711988512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0078100bc0 0x7f0078196370 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:01.000+0000 7f007dccd700 1 --2- 172.21.15.114:0/3711988512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0078101dc0 0x7f00781968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:01.000+0000 7f007dccd700 1 -- 172.21.15.114:0/3711988512 >> 172.21.15.114:0/3711988512 conn(0x7f00780fc140 msgr2=0x7f0078104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:01.001+0000 7f007dccd700 1 -- 172.21.15.114:0/3711988512 shutdown_connections 2024-02-19T19:00:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:01.001+0000 7f007dccd700 1 -- 172.21.15.114:0/3711988512 wait complete. 2024-02-19T19:00:01.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:00 smithi114 ceph-mon[29441]: from='client.14506 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:01.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:00 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-19T19:00:01.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:00 smithi114 ceph-mon[29441]: osdmap e42: 6 total, 6 up, 6 in 2024-02-19T19:00:01.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:00 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-19T19:00:01.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:00 smithi114 ceph-mon[29441]: overall HEALTH_OK 2024-02-19T19:00:01.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[29437]: 2024-02-19T19:00:00.979+0000 7f35c7d12700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T19:00:01.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:00 smithi174 ceph-mon[41061]: from='client.14506 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:01.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:00 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-19T19:00:01.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:00 smithi174 ceph-mon[41061]: osdmap e42: 6 total, 6 up, 6 in 2024-02-19T19:00:01.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:00 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-19T19:00:01.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:00 smithi174 ceph-mon[41061]: overall HEALTH_OK 2024-02-19T19:00:01.603 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-19T19:00:01.820 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T19:00:02.225 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: pgmap v179: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:02.225 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T19:00:02.225 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-19T19:00:02.225 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: osdmap e43: 6 total, 6 up, 6 in 2024-02-19T19:00:02.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-19T19:00:02.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T19:00:02.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-19T19:00:02.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: osdmap e44: 6 total, 6 up, 6 in 2024-02-19T19:00:02.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-19T19:00:02.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: fsmap cephfs:0 2024-02-19T19:00:02.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:02.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:02.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:02.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:02.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:02.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.vpoeop", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:00:02.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.vpoeop", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T19:00:02.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:01 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:02.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: pgmap v179: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:02.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T19:00:02.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-19T19:00:02.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: osdmap e43: 6 total, 6 up, 6 in 2024-02-19T19:00:02.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-19T19:00:02.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T19:00:02.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-19T19:00:02.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: osdmap e44: 6 total, 6 up, 6 in 2024-02-19T19:00:02.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-19T19:00:02.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: fsmap cephfs:0 2024-02-19T19:00:02.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:02.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:02.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:02.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:02.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:02.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.vpoeop", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:00:02.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.vpoeop", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T19:00:02.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:01 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:03.249 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:02 smithi114 ceph-mon[29441]: Saving service mds.cephfs spec with placement count:4 2024-02-19T19:00:03.249 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:02 smithi114 ceph-mon[29441]: Deploying daemon mds.cephfs.smithi114.vpoeop on smithi114 2024-02-19T19:00:03.249 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:02 smithi114 ceph-mon[29441]: osdmap e45: 6 total, 6 up, 6 in 2024-02-19T19:00:03.249 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:02 smithi114 ceph-mon[29441]: pgmap v183: 65 pgs: 51 unknown, 14 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:03.249 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:02 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:03.249 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:02 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:03.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:02 smithi174 ceph-mon[41061]: Saving service mds.cephfs spec with placement count:4 2024-02-19T19:00:03.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:02 smithi174 ceph-mon[41061]: Deploying daemon mds.cephfs.smithi114.vpoeop on smithi114 2024-02-19T19:00:03.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:02 smithi174 ceph-mon[41061]: osdmap e45: 6 total, 6 up, 6 in 2024-02-19T19:00:03.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:02 smithi174 ceph-mon[41061]: pgmap v183: 65 pgs: 51 unknown, 14 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:03.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:02 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:03.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:02 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:04.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:03 smithi174 ceph-mon[41061]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-19T19:00:04.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:03 smithi174 ceph-mon[41061]: osdmap e46: 6 total, 6 up, 6 in 2024-02-19T19:00:04.243 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:03 smithi114 ceph-mon[29441]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-19T19:00:04.244 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:03 smithi114 ceph-mon[29441]: osdmap e46: 6 total, 6 up, 6 in 2024-02-19T19:00:04.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.480+0000 7f11a111f700 1 -- 172.21.15.114:0/2919260169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f119c1016c0 msgr2=0x7f119c103ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:04.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.480+0000 7f11a111f700 1 --2- 172.21.15.114:0/2919260169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f119c1016c0 0x7f119c103ab0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f118c009910 tx=0x7f118c009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:04.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.480+0000 7f11a111f700 1 -- 172.21.15.114:0/2919260169 shutdown_connections 2024-02-19T19:00:04.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.480+0000 7f11a111f700 1 --2- 172.21.15.114:0/2919260169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f119c103ff0 0x7f119c1063e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:04.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.480+0000 7f11a111f700 1 --2- 172.21.15.114:0/2919260169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f119c1016c0 0x7f119c103ab0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:04.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.480+0000 7f11a111f700 1 -- 172.21.15.114:0/2919260169 >> 172.21.15.114:0/2919260169 conn(0x7f119c0fb030 msgr2=0x7f119c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:04.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.480+0000 7f11a111f700 1 -- 172.21.15.114:0/2919260169 shutdown_connections 2024-02-19T19:00:04.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.480+0000 7f11a111f700 1 -- 172.21.15.114:0/2919260169 wait complete. 2024-02-19T19:00:04.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.481+0000 7f11a111f700 1 Processor -- start 2024-02-19T19:00:04.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.482+0000 7f11a111f700 1 -- start start 2024-02-19T19:00:04.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.482+0000 7f11a111f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f119c1016c0 0x7f119c101020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:04.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.482+0000 7f11a111f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f119c103ff0 0x7f119c0ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:04.505 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.482+0000 7f11a111f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119c0ffbb0 con 0x7f119c1016c0 2024-02-19T19:00:04.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.482+0000 7f11a111f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119c0ffcf0 con 0x7f119c103ff0 2024-02-19T19:00:04.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.482+0000 7f119ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f119c1016c0 0x7f119c101020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:04.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.482+0000 7f119ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f119c1016c0 0x7f119c101020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35346/0 (socket says 172.21.15.114:35346) 2024-02-19T19:00:04.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.482+0000 7f119ad9d700 1 -- 172.21.15.114:0/2655243 learned_addr learned my addr 172.21.15.114:0/2655243 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:04.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.482+0000 7f119a59c700 1 --2- 172.21.15.114:0/2655243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f119c103ff0 0x7f119c0ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.482+0000 7f119ad9d700 1 -- 172.21.15.114:0/2655243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f119c103ff0 msgr2=0x7f119c0ff670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.482+0000 7f119ad9d700 1 --2- 172.21.15.114:0/2655243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f119c103ff0 0x7f119c0ff670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.482+0000 7f119ad9d700 1 -- 172.21.15.114:0/2655243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c0095f0 con 0x7f119c1016c0 2024-02-19T19:00:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.482+0000 7f119a59c700 1 --2- 172.21.15.114:0/2655243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f119c103ff0 0x7f119c0ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:00:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.483+0000 7f119ad9d700 1 --2- 172.21.15.114:0/2655243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f119c1016c0 0x7f119c101020 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f118c000c00 tx=0x7f118c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.483+0000 7f1183fff700 1 -- 172.21.15.114:0/2655243 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f118c01d070 con 0x7f119c1016c0 2024-02-19T19:00:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.483+0000 7f1183fff700 1 -- 172.21.15.114:0/2655243 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f118c005dc0 con 0x7f119c1016c0 2024-02-19T19:00:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.483+0000 7f1183fff700 1 -- 172.21.15.114:0/2655243 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f118c0226a0 con 0x7f119c1016c0 2024-02-19T19:00:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.483+0000 7f11a111f700 1 -- 172.21.15.114:0/2655243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f119c0fff70 con 0x7f119c1016c0 2024-02-19T19:00:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.483+0000 7f11a111f700 1 -- 172.21.15.114:0/2655243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f119c1003a0 con 0x7f119c1016c0 2024-02-19T19:00:04.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.484+0000 7f11a111f700 1 -- 172.21.15.114:0/2655243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f119c190520 con 0x7f119c1016c0 2024-02-19T19:00:04.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.484+0000 7f1183fff700 1 -- 172.21.15.114:0/2655243 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f118c0038f0 con 0x7f119c1016c0 2024-02-19T19:00:04.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.485+0000 7f1183fff700 1 --2- 172.21.15.114:0/2655243 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f118406c4e0 0x7f118406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:04.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.485+0000 7f1183fff700 1 -- 172.21.15.114:0/2655243 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f118c08cc60 con 0x7f119c1016c0 2024-02-19T19:00:04.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.485+0000 7f119a59c700 1 --2- 172.21.15.114:0/2655243 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f118406c4e0 0x7f118406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:04.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.485+0000 7f119a59c700 1 --2- 172.21.15.114:0/2655243 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f118406c4e0 0x7f118406e9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1190011490 tx=0x7f1190009260 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:04.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.490+0000 7f1183fff700 1 -- 172.21.15.114:0/2655243 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f118c05af50 con 0x7f119c1016c0 2024-02-19T19:00:04.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.743+0000 7f11a111f700 1 -- 172.21.15.114:0/2655243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f119c02cc70 con 0x7f119c1016c0 2024-02-19T19:00:04.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.743+0000 7f1183fff700 1 -- 172.21.15.114:0/2655243 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1093 (secure 0 0 0) 0x7f118c003680 con 0x7f119c1016c0 2024-02-19T19:00:04.762 INFO:teuthology.orchestra.run.smithi114.stdout:e2 2024-02-19T19:00:04.762 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:00:04.762 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:00:04.762 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-02-19T19:00:04.762 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:00:04.762 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:00:04.763 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-02-19T19:00:04.763 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 2 2024-02-19T19:00:04.763 INFO:teuthology.orchestra.run.smithi114.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-19T19:00:04.763 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:00:04.763 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-02-19T19:00:00.979901+0000 2024-02-19T19:00:04.763 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-02-19T19:00:04.763 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-02-19T19:00:04.763 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-02-19T19:00:04.763 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-02-19T19:00:04.763 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-02-19T19:00:04.763 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-02-19T19:00:04.763 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-02-19T19:00:04.763 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 0 2024-02-19T19:00:04.763 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:00:04.764 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 1 2024-02-19T19:00:04.764 INFO:teuthology.orchestra.run.smithi114.stdout:in 2024-02-19T19:00:04.764 INFO:teuthology.orchestra.run.smithi114.stdout:up {} 2024-02-19T19:00:04.764 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-02-19T19:00:04.764 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-02-19T19:00:04.764 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-02-19T19:00:04.764 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-02-19T19:00:04.764 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-02-19T19:00:04.764 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data disabled 2024-02-19T19:00:04.764 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-02-19T19:00:04.764 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-02-19T19:00:04.764 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 0 2024-02-19T19:00:04.764 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:00:04.764 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:00:04.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.746+0000 7f11a111f700 1 -- 172.21.15.114:0/2655243 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f118406c4e0 msgr2=0x7f118406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:04.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.746+0000 7f11a111f700 1 --2- 172.21.15.114:0/2655243 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f118406c4e0 0x7f118406e9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1190011490 tx=0x7f1190009260 comp rx=0 tx=0).stop 2024-02-19T19:00:04.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.746+0000 7f11a111f700 1 -- 172.21.15.114:0/2655243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f119c1016c0 msgr2=0x7f119c101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:04.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.746+0000 7f11a111f700 1 --2- 172.21.15.114:0/2655243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f119c1016c0 0x7f119c101020 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f118c000c00 tx=0x7f118c00f690 comp rx=0 tx=0).stop 2024-02-19T19:00:04.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.747+0000 7f11a111f700 1 -- 172.21.15.114:0/2655243 shutdown_connections 2024-02-19T19:00:04.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.747+0000 7f11a111f700 1 --2- 172.21.15.114:0/2655243 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f118406c4e0 0x7f118406e9a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:04.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.747+0000 7f11a111f700 1 --2- 172.21.15.114:0/2655243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f119c1016c0 0x7f119c101020 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:04.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.747+0000 7f11a111f700 1 --2- 172.21.15.114:0/2655243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f119c103ff0 0x7f119c0ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:04.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.747+0000 7f11a111f700 1 -- 172.21.15.114:0/2655243 >> 172.21.15.114:0/2655243 conn(0x7f119c0fb030 msgr2=0x7f119c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:04.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.747+0000 7f11a111f700 1 -- 172.21.15.114:0/2655243 shutdown_connections 2024-02-19T19:00:04.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:04.747+0000 7f11a111f700 1 -- 172.21.15.114:0/2655243 wait complete. 2024-02-19T19:00:04.766 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 2 2024-02-19T19:00:05.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:04 smithi114 ceph-mon[29441]: pgmap v185: 65 pgs: 18 unknown, 47 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:05.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:04 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/2655243' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:00:05.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:04 smithi174 ceph-mon[41061]: pgmap v185: 65 pgs: 18 unknown, 47 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:05.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:04 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/2655243' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:00:05.903 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T19:00:05.913 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-02-19T19:00:05.913 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-02-19T19:00:06.127 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T19:00:06.787 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:06 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:06.787 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:06 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:06.787 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:06 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:06 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi174.znhuvd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:00:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:06 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi174.znhuvd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T19:00:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:06 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:06 smithi114 ceph-mon[29441]: Deploying daemon mds.cephfs.smithi174.znhuvd on smithi174 2024-02-19T19:00:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:06 smithi114 ceph-mon[29441]: mds.? [v2:172.21.15.114:6826/98255571,v1:172.21.15.114:6827/98255571] up:boot 2024-02-19T19:00:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:06 smithi114 ceph-mon[29441]: daemon mds.cephfs.smithi114.vpoeop assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-19T19:00:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:06 smithi114 ceph-mon[29441]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-19T19:00:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:06 smithi114 ceph-mon[29441]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-19T19:00:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:06 smithi114 ceph-mon[29441]: Cluster is now healthy 2024-02-19T19:00:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:06 smithi114 ceph-mon[29441]: fsmap cephfs:0 1 up:standby 2024-02-19T19:00:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:06 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.vpoeop"}]: dispatch 2024-02-19T19:00:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:06 smithi114 ceph-mon[29441]: fsmap cephfs:1 {0=cephfs.smithi114.vpoeop=up:creating} 2024-02-19T19:00:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:06 smithi114 ceph-mon[29441]: daemon mds.cephfs.smithi114.vpoeop is now active in filesystem cephfs as rank 0 2024-02-19T19:00:06.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:06 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:06.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:06 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:06.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:06 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:06.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:06 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi174.znhuvd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:00:06.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:06 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi174.znhuvd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T19:00:06.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:06 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:06.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:06 smithi174 ceph-mon[41061]: Deploying daemon mds.cephfs.smithi174.znhuvd on smithi174 2024-02-19T19:00:06.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:06 smithi174 ceph-mon[41061]: mds.? [v2:172.21.15.114:6826/98255571,v1:172.21.15.114:6827/98255571] up:boot 2024-02-19T19:00:06.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:06 smithi174 ceph-mon[41061]: daemon mds.cephfs.smithi114.vpoeop assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-19T19:00:06.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:06 smithi174 ceph-mon[41061]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-19T19:00:06.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:06 smithi174 ceph-mon[41061]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-19T19:00:06.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:06 smithi174 ceph-mon[41061]: Cluster is now healthy 2024-02-19T19:00:06.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:06 smithi174 ceph-mon[41061]: fsmap cephfs:0 1 up:standby 2024-02-19T19:00:06.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:06 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.vpoeop"}]: dispatch 2024-02-19T19:00:06.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:06 smithi174 ceph-mon[41061]: fsmap cephfs:1 {0=cephfs.smithi114.vpoeop=up:creating} 2024-02-19T19:00:06.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:06 smithi174 ceph-mon[41061]: daemon mds.cephfs.smithi114.vpoeop is now active in filesystem cephfs as rank 0 2024-02-19T19:00:07.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:07 smithi174 ceph-mon[41061]: pgmap v186: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:07.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:07 smithi174 ceph-mon[41061]: mds.? [v2:172.21.15.114:6826/98255571,v1:172.21.15.114:6827/98255571] up:active 2024-02-19T19:00:07.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:07 smithi174 ceph-mon[41061]: fsmap cephfs:1 {0=cephfs.smithi114.vpoeop=up:active} 2024-02-19T19:00:07.787 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:07 smithi114 ceph-mon[29441]: pgmap v186: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:00:07.787 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:07 smithi114 ceph-mon[29441]: mds.? [v2:172.21.15.114:6826/98255571,v1:172.21.15.114:6827/98255571] up:active 2024-02-19T19:00:07.787 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:07 smithi114 ceph-mon[29441]: fsmap cephfs:1 {0=cephfs.smithi114.vpoeop=up:active} 2024-02-19T19:00:07.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.771+0000 7f2a335ae700 1 -- 172.21.15.114:0/2887370685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a2c1021a0 msgr2=0x7f2a2c1025c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:07.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.771+0000 7f2a335ae700 1 --2- 172.21.15.114:0/2887370685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a2c1021a0 0x7f2a2c1025c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009910 tx=0x7f2a28009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:07.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.771+0000 7f2a335ae700 1 -- 172.21.15.114:0/2887370685 shutdown_connections 2024-02-19T19:00:07.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.771+0000 7f2a335ae700 1 --2- 172.21.15.114:0/2887370685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a2c0fec20 0x7f2a2c0ff080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:07.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.771+0000 7f2a335ae700 1 --2- 172.21.15.114:0/2887370685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a2c1021a0 0x7f2a2c1025c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:07.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.771+0000 7f2a335ae700 1 -- 172.21.15.114:0/2887370685 >> 172.21.15.114:0/2887370685 conn(0x7f2a2c0fa7c0 msgr2=0x7f2a2c0fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:07.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.771+0000 7f2a335ae700 1 -- 172.21.15.114:0/2887370685 shutdown_connections 2024-02-19T19:00:07.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.771+0000 7f2a335ae700 1 -- 172.21.15.114:0/2887370685 wait complete. 2024-02-19T19:00:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.772+0000 7f2a335ae700 1 Processor -- start 2024-02-19T19:00:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.772+0000 7f2a335ae700 1 -- start start 2024-02-19T19:00:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.772+0000 7f2a335ae700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a2c0fec20 0x7f2a2c1963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.772+0000 7f2a335ae700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a2c1021a0 0x7f2a2c196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.772+0000 7f2a335ae700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c196f50 con 0x7f2a2c0fec20 2024-02-19T19:00:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.772+0000 7f2a335ae700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c197090 con 0x7f2a2c1021a0 2024-02-19T19:00:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.772+0000 7f2a3134a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a2c0fec20 0x7f2a2c1963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.772+0000 7f2a3134a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a2c0fec20 0x7f2a2c1963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38648/0 (socket says 172.21.15.114:38648) 2024-02-19T19:00:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.772+0000 7f2a3134a700 1 -- 172.21.15.114:0/1007390662 learned_addr learned my addr 172.21.15.114:0/1007390662 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.772+0000 7f2a30b49700 1 --2- 172.21.15.114:0/1007390662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a2c1021a0 0x7f2a2c196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.772+0000 7f2a3134a700 1 -- 172.21.15.114:0/1007390662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a2c1021a0 msgr2=0x7f2a2c196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.772+0000 7f2a3134a700 1 --2- 172.21.15.114:0/1007390662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a2c1021a0 0x7f2a2c196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.772+0000 7f2a3134a700 1 -- 172.21.15.114:0/1007390662 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a280095f0 con 0x7f2a2c0fec20 2024-02-19T19:00:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.773+0000 7f2a3134a700 1 --2- 172.21.15.114:0/1007390662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a2c0fec20 0x7f2a2c1963f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2a28000c00 tx=0x7f2a2800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.773+0000 7f2a227fc700 1 -- 172.21.15.114:0/1007390662 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a2801d070 con 0x7f2a2c0fec20 2024-02-19T19:00:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.773+0000 7f2a335ae700 1 -- 172.21.15.114:0/1007390662 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2c19bae0 con 0x7f2a2c0fec20 2024-02-19T19:00:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.773+0000 7f2a227fc700 1 -- 172.21.15.114:0/1007390662 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2a28005dc0 con 0x7f2a2c0fec20 2024-02-19T19:00:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.773+0000 7f2a227fc700 1 -- 172.21.15.114:0/1007390662 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a28022470 con 0x7f2a2c0fec20 2024-02-19T19:00:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.773+0000 7f2a335ae700 1 -- 172.21.15.114:0/1007390662 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2c19bfd0 con 0x7f2a2c0fec20 2024-02-19T19:00:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.774+0000 7f2a335ae700 1 -- 172.21.15.114:0/1007390662 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2c109730 con 0x7f2a2c0fec20 2024-02-19T19:00:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.774+0000 7f2a227fc700 1 -- 172.21.15.114:0/1007390662 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2a28003810 con 0x7f2a2c0fec20 2024-02-19T19:00:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.775+0000 7f2a227fc700 1 --2- 172.21.15.114:0/1007390662 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2a1806c4e0 0x7f2a1806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:07.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.775+0000 7f2a30b49700 1 --2- 172.21.15.114:0/1007390662 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2a1806c4e0 0x7f2a1806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:07.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.775+0000 7f2a227fc700 1 -- 172.21.15.114:0/1007390662 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f2a2808cd50 con 0x7f2a2c0fec20 2024-02-19T19:00:07.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.775+0000 7f2a30b49700 1 --2- 172.21.15.114:0/1007390662 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2a1806c4e0 0x7f2a1806e9a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c009c60 tx=0x7f2a1c009260 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:07.779+0000 7f2a227fc700 1 -- 172.21.15.114:0/1007390662 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2a2805afc0 con 0x7f2a2c0fec20 2024-02-19T19:00:08.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:08.011+0000 7f2a335ae700 1 -- 172.21.15.114:0/1007390662 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f2a2c19c640 con 0x7f2a2c0fec20 2024-02-19T19:00:08.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:08.958+0000 7f2a227fc700 1 -- 172.21.15.114:0/1007390662 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f2a2805ab50 con 0x7f2a2c0fec20 2024-02-19T19:00:08.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:08.960+0000 7f2a335ae700 1 -- 172.21.15.114:0/1007390662 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2a1806c4e0 msgr2=0x7f2a1806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:08.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:08.960+0000 7f2a335ae700 1 --2- 172.21.15.114:0/1007390662 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2a1806c4e0 0x7f2a1806e9a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c009c60 tx=0x7f2a1c009260 comp rx=0 tx=0).stop 2024-02-19T19:00:08.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:08.960+0000 7f2a335ae700 1 -- 172.21.15.114:0/1007390662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a2c0fec20 msgr2=0x7f2a2c1963f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:08.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:08.960+0000 7f2a335ae700 1 --2- 172.21.15.114:0/1007390662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a2c0fec20 0x7f2a2c1963f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2a28000c00 tx=0x7f2a2800f690 comp rx=0 tx=0).stop 2024-02-19T19:00:08.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:08.960+0000 7f2a335ae700 1 -- 172.21.15.114:0/1007390662 shutdown_connections 2024-02-19T19:00:08.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:08.960+0000 7f2a335ae700 1 --2- 172.21.15.114:0/1007390662 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f2a1806c4e0 0x7f2a1806e9a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:08.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:08.960+0000 7f2a335ae700 1 --2- 172.21.15.114:0/1007390662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a2c0fec20 0x7f2a2c1963f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:08.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:08.960+0000 7f2a335ae700 1 --2- 172.21.15.114:0/1007390662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a2c1021a0 0x7f2a2c196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:08.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:08.960+0000 7f2a335ae700 1 -- 172.21.15.114:0/1007390662 >> 172.21.15.114:0/1007390662 conn(0x7f2a2c0fa7c0 msgr2=0x7f2a2c105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:08.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:08.960+0000 7f2a335ae700 1 -- 172.21.15.114:0/1007390662 shutdown_connections 2024-02-19T19:00:08.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:08.961+0000 7f2a335ae700 1 -- 172.21.15.114:0/1007390662 wait complete. 2024-02-19T19:00:09.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:08 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1007390662' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-02-19T19:00:09.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:08 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:09.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:08 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:09.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:08 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:09.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:08 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.tcwpod", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:00:09.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:08 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.tcwpod", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T19:00:09.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:08 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:09.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:08 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1007390662' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-02-19T19:00:09.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:08 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:09.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:08 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:09.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:08 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:09.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:08 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.tcwpod", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:00:09.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:08 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.tcwpod", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T19:00:09.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:08 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:09.740 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T19:00:09.750 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-02-19T19:00:09.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-02-19T19:00:10.259 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T19:00:10.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:09 smithi114 ceph-mon[29441]: Deploying daemon mds.cephfs.smithi114.tcwpod on smithi114 2024-02-19T19:00:10.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:09 smithi114 ceph-mon[29441]: pgmap v187: 65 pgs: 65 active+clean; 578 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 533 B/s wr, 2 op/s 2024-02-19T19:00:10.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:09 smithi114 ceph-mon[29441]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-19T19:00:10.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:09 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1007390662' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-02-19T19:00:10.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:09 smithi114 ceph-mon[29441]: mds.? [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] up:boot 2024-02-19T19:00:10.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:09 smithi114 ceph-mon[29441]: daemon mds.cephfs.smithi174.znhuvd assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-02-19T19:00:10.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:09 smithi114 ceph-mon[29441]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-19T19:00:10.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:09 smithi114 ceph-mon[29441]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-19T19:00:10.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:09 smithi114 ceph-mon[29441]: fsmap cephfs:1 {0=cephfs.smithi114.vpoeop=up:active} 1 up:standby 2024-02-19T19:00:10.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:09 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.znhuvd"}]: dispatch 2024-02-19T19:00:10.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:09 smithi114 ceph-mon[29441]: fsmap cephfs:2 {0=cephfs.smithi114.vpoeop=up:active,1=cephfs.smithi174.znhuvd=up:creating} 2024-02-19T19:00:10.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:09 smithi114 ceph-mon[29441]: daemon mds.cephfs.smithi174.znhuvd is now active in filesystem cephfs as rank 1 2024-02-19T19:00:10.319 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:09 smithi174 ceph-mon[41061]: Deploying daemon mds.cephfs.smithi114.tcwpod on smithi114 2024-02-19T19:00:10.319 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:09 smithi174 ceph-mon[41061]: pgmap v187: 65 pgs: 65 active+clean; 578 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 533 B/s wr, 2 op/s 2024-02-19T19:00:10.319 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:09 smithi174 ceph-mon[41061]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-19T19:00:10.320 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:09 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1007390662' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-02-19T19:00:10.320 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:09 smithi174 ceph-mon[41061]: mds.? [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] up:boot 2024-02-19T19:00:10.320 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:09 smithi174 ceph-mon[41061]: daemon mds.cephfs.smithi174.znhuvd assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-02-19T19:00:10.320 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:09 smithi174 ceph-mon[41061]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-19T19:00:10.320 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:09 smithi174 ceph-mon[41061]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-19T19:00:10.320 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:09 smithi174 ceph-mon[41061]: fsmap cephfs:1 {0=cephfs.smithi114.vpoeop=up:active} 1 up:standby 2024-02-19T19:00:10.320 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:09 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.znhuvd"}]: dispatch 2024-02-19T19:00:10.320 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:09 smithi174 ceph-mon[41061]: fsmap cephfs:2 {0=cephfs.smithi114.vpoeop=up:active,1=cephfs.smithi174.znhuvd=up:creating} 2024-02-19T19:00:10.320 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:09 smithi174 ceph-mon[41061]: daemon mds.cephfs.smithi174.znhuvd is now active in filesystem cephfs as rank 1 2024-02-19T19:00:11.245 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:10 smithi114 ceph-mon[29441]: mds.? [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] up:active 2024-02-19T19:00:11.245 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:10 smithi114 ceph-mon[29441]: fsmap cephfs:2 {0=cephfs.smithi114.vpoeop=up:active,1=cephfs.smithi174.znhuvd=up:active} 2024-02-19T19:00:11.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:10 smithi174 ceph-mon[41061]: mds.? [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] up:active 2024-02-19T19:00:11.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:10 smithi174 ceph-mon[41061]: fsmap cephfs:2 {0=cephfs.smithi114.vpoeop=up:active,1=cephfs.smithi174.znhuvd=up:active} 2024-02-19T19:00:12.056 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:11 smithi114 ceph-mon[29441]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-02-19T19:00:12.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.043+0000 7f4b1f228700 1 -- 172.21.15.114:0/1562627028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b18101db0 msgr2=0x7f4b18102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:12.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.043+0000 7f4b1f228700 1 --2- 172.21.15.114:0/1562627028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b18101db0 0x7f4b18102210 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009910 tx=0x7f4b04009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:12.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.043+0000 7f4b1f228700 1 -- 172.21.15.114:0/1562627028 shutdown_connections 2024-02-19T19:00:12.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.043+0000 7f4b1f228700 1 --2- 172.21.15.114:0/1562627028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b18101db0 0x7f4b18102210 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:12.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.043+0000 7f4b1f228700 1 --2- 172.21.15.114:0/1562627028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b18100bb0 0x7f4b18100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:12.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.043+0000 7f4b1f228700 1 -- 172.21.15.114:0/1562627028 >> 172.21.15.114:0/1562627028 conn(0x7f4b180fc110 msgr2=0x7f4b180fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:12.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.043+0000 7f4b1f228700 1 -- 172.21.15.114:0/1562627028 shutdown_connections 2024-02-19T19:00:12.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.043+0000 7f4b1f228700 1 -- 172.21.15.114:0/1562627028 wait complete. 2024-02-19T19:00:12.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.044+0000 7f4b1f228700 1 Processor -- start 2024-02-19T19:00:12.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.044+0000 7f4b1f228700 1 -- start start 2024-02-19T19:00:12.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.044+0000 7f4b1f228700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b18100bb0 0x7f4b18196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:12.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.044+0000 7f4b1f228700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b18101db0 0x7f4b18196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:12.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.044+0000 7f4b1f228700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b18196f60 con 0x7f4b18101db0 2024-02-19T19:00:12.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.044+0000 7f4b1f228700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b181970a0 con 0x7f4b18100bb0 2024-02-19T19:00:12.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.044+0000 7f4b17fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b18101db0 0x7f4b18196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:12.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.044+0000 7f4b17fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b18101db0 0x7f4b18196940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38668/0 (socket says 172.21.15.114:38668) 2024-02-19T19:00:12.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.044+0000 7f4b17fff700 1 -- 172.21.15.114:0/343636990 learned_addr learned my addr 172.21.15.114:0/343636990 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:12.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.044+0000 7f4b1cfc4700 1 --2- 172.21.15.114:0/343636990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b18100bb0 0x7f4b18196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:12.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.044+0000 7f4b17fff700 1 -- 172.21.15.114:0/343636990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b18100bb0 msgr2=0x7f4b18196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:12.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.044+0000 7f4b17fff700 1 --2- 172.21.15.114:0/343636990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b18100bb0 0x7f4b18196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:12.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.044+0000 7f4b17fff700 1 -- 172.21.15.114:0/343636990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b040095f0 con 0x7f4b18101db0 2024-02-19T19:00:12.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.045+0000 7f4b1cfc4700 1 --2- 172.21.15.114:0/343636990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b18100bb0 0x7f4b18196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:00:12.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.045+0000 7f4b17fff700 1 --2- 172.21.15.114:0/343636990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b18101db0 0x7f4b18196940 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400b5c0 tx=0x7f4b0400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:12.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.045+0000 7f4b15ffb700 1 -- 172.21.15.114:0/343636990 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b0401d070 con 0x7f4b18101db0 2024-02-19T19:00:12.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.045+0000 7f4b1f228700 1 -- 172.21.15.114:0/343636990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1819baf0 con 0x7f4b18101db0 2024-02-19T19:00:12.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.045+0000 7f4b15ffb700 1 -- 172.21.15.114:0/343636990 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4b04022470 con 0x7f4b18101db0 2024-02-19T19:00:12.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.045+0000 7f4b15ffb700 1 -- 172.21.15.114:0/343636990 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b0400f690 con 0x7f4b18101db0 2024-02-19T19:00:12.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.045+0000 7f4b1f228700 1 -- 172.21.15.114:0/343636990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1819c060 con 0x7f4b18101db0 2024-02-19T19:00:12.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.046+0000 7f4b1f228700 1 -- 172.21.15.114:0/343636990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b181097b0 con 0x7f4b18101db0 2024-02-19T19:00:12.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.046+0000 7f4b15ffb700 1 -- 172.21.15.114:0/343636990 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f4b04004b70 con 0x7f4b18101db0 2024-02-19T19:00:12.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.047+0000 7f4b15ffb700 1 --2- 172.21.15.114:0/343636990 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f4b0806c4e0 0x7f4b0806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:12.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.047+0000 7f4b1cfc4700 1 --2- 172.21.15.114:0/343636990 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f4b0806c4e0 0x7f4b0806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:12.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.047+0000 7f4b15ffb700 1 -- 172.21.15.114:0/343636990 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f4b0408cf80 con 0x7f4b18101db0 2024-02-19T19:00:12.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.047+0000 7f4b1cfc4700 1 --2- 172.21.15.114:0/343636990 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f4b0806c4e0 0x7f4b0806e9a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4b18101c10 tx=0x7f4b0c008040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:12.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.052+0000 7f4b15ffb700 1 -- 172.21.15.114:0/343636990 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4b0405b2a0 con 0x7f4b18101db0 2024-02-19T19:00:12.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.289+0000 7f4b1f228700 1 -- 172.21.15.114:0/343636990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f4b181099a0 con 0x7f4b18101db0 2024-02-19T19:00:12.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:11 smithi174 ceph-mon[41061]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-02-19T19:00:12.973 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.955+0000 7f4b15ffb700 1 -- 172.21.15.114:0/343636990 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v9) v1 ==== 121+0+0 (secure 0 0 0) 0x7f4b0405ae30 con 0x7f4b18101db0 2024-02-19T19:00:12.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.957+0000 7f4b1f228700 1 -- 172.21.15.114:0/343636990 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f4b0806c4e0 msgr2=0x7f4b0806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:12.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.957+0000 7f4b1f228700 1 --2- 172.21.15.114:0/343636990 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f4b0806c4e0 0x7f4b0806e9a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4b18101c10 tx=0x7f4b0c008040 comp rx=0 tx=0).stop 2024-02-19T19:00:12.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.957+0000 7f4b1f228700 1 -- 172.21.15.114:0/343636990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b18101db0 msgr2=0x7f4b18196940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:12.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.957+0000 7f4b1f228700 1 --2- 172.21.15.114:0/343636990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b18101db0 0x7f4b18196940 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400b5c0 tx=0x7f4b0400be20 comp rx=0 tx=0).stop 2024-02-19T19:00:12.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.957+0000 7f4b1f228700 1 -- 172.21.15.114:0/343636990 shutdown_connections 2024-02-19T19:00:12.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.957+0000 7f4b1f228700 1 --2- 172.21.15.114:0/343636990 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f4b0806c4e0 0x7f4b0806e9a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:12.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.957+0000 7f4b1f228700 1 --2- 172.21.15.114:0/343636990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b18100bb0 0x7f4b18196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:12.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.957+0000 7f4b1f228700 1 --2- 172.21.15.114:0/343636990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b18101db0 0x7f4b18196940 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:12.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.957+0000 7f4b1f228700 1 -- 172.21.15.114:0/343636990 >> 172.21.15.114:0/343636990 conn(0x7f4b180fc110 msgr2=0x7f4b18104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:12.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.957+0000 7f4b1f228700 1 -- 172.21.15.114:0/343636990 shutdown_connections 2024-02-19T19:00:12.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:12.957+0000 7f4b1f228700 1 -- 172.21.15.114:0/343636990 wait complete. 2024-02-19T19:00:13.281 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:12 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/343636990' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-19T19:00:13.301 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:12 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:13.301 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:12 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:13.301 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:12 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:13.301 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:12 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi174.stnfnf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:00:13.301 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:12 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi174.stnfnf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T19:00:13.301 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:12 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:13.301 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:12 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:13.302 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:12 smithi114 ceph-mon[29441]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-19T19:00:13.302 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:12 smithi114 ceph-mon[29441]: Cluster is now healthy 2024-02-19T19:00:13.302 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:12 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/343636990' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-19T19:00:13.302 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:12 smithi114 ceph-mon[29441]: mds.? [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] up:boot 2024-02-19T19:00:13.302 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:12 smithi114 ceph-mon[29441]: fsmap cephfs:2 {0=cephfs.smithi114.vpoeop=up:active,1=cephfs.smithi174.znhuvd=up:active} 1 up:standby-replay 2024-02-19T19:00:13.302 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:12 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.tcwpod"}]: dispatch 2024-02-19T19:00:13.302 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:12 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/343636990' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-19T19:00:13.303 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:12 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:13.303 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:12 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:13.303 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:12 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:13.303 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:12 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi174.stnfnf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:00:13.303 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:12 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi174.stnfnf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T19:00:13.303 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:12 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:13.303 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:12 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:13.303 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:12 smithi174 ceph-mon[41061]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-19T19:00:13.303 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:12 smithi174 ceph-mon[41061]: Cluster is now healthy 2024-02-19T19:00:13.303 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:12 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/343636990' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-19T19:00:13.303 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:12 smithi174 ceph-mon[41061]: mds.? [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] up:boot 2024-02-19T19:00:13.303 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:12 smithi174 ceph-mon[41061]: fsmap cephfs:2 {0=cephfs.smithi114.vpoeop=up:active,1=cephfs.smithi174.znhuvd=up:active} 1 up:standby-replay 2024-02-19T19:00:13.304 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:12 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.tcwpod"}]: dispatch 2024-02-19T19:00:13.524 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T19:00:13.535 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-02-19T19:00:13.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-02-19T19:00:13.747 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T19:00:14.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:13 smithi114 ceph-mon[29441]: Deploying daemon mds.cephfs.smithi174.stnfnf on smithi174 2024-02-19T19:00:14.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:13 smithi114 ceph-mon[29441]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s wr, 5 op/s 2024-02-19T19:00:14.370 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:13 smithi174 ceph-mon[41061]: Deploying daemon mds.cephfs.smithi174.stnfnf on smithi174 2024-02-19T19:00:14.370 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:13 smithi174 ceph-mon[41061]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s wr, 5 op/s 2024-02-19T19:00:15.125 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:15 smithi114 ceph-mon[29441]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 175 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-02-19T19:00:15.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.348+0000 7f546b148700 1 -- 172.21.15.114:0/901554370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5464100cb0 msgr2=0x7f5464101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:15.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.348+0000 7f546b148700 1 --2- 172.21.15.114:0/901554370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5464100cb0 0x7f5464101110 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5450009910 tx=0x7f5450009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:15.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.348+0000 7f546b148700 1 -- 172.21.15.114:0/901554370 shutdown_connections 2024-02-19T19:00:15.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.348+0000 7f546b148700 1 --2- 172.21.15.114:0/901554370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5464100cb0 0x7f5464101110 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:15.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.348+0000 7f546b148700 1 --2- 172.21.15.114:0/901554370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54640ffab0 0x7f54640ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:15.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.348+0000 7f546b148700 1 -- 172.21.15.114:0/901554370 >> 172.21.15.114:0/901554370 conn(0x7f54640fb050 msgr2=0x7f54640fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:15.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.348+0000 7f546b148700 1 -- 172.21.15.114:0/901554370 shutdown_connections 2024-02-19T19:00:15.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.348+0000 7f546b148700 1 -- 172.21.15.114:0/901554370 wait complete. 2024-02-19T19:00:15.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.349+0000 7f546b148700 1 Processor -- start 2024-02-19T19:00:15.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.349+0000 7f546b148700 1 -- start start 2024-02-19T19:00:15.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.349+0000 7f546b148700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54640ffab0 0x7f5464195250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:15.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.349+0000 7f546b148700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5464100cb0 0x7f5464195790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:15.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.349+0000 7f546b148700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5464195db0 con 0x7f54640ffab0 2024-02-19T19:00:15.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.349+0000 7f546b148700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5464195ef0 con 0x7f5464100cb0 2024-02-19T19:00:15.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.349+0000 7f5468ee4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54640ffab0 0x7f5464195250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:15.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.349+0000 7f5468ee4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54640ffab0 0x7f5464195250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38710/0 (socket says 172.21.15.114:38710) 2024-02-19T19:00:15.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.349+0000 7f5468ee4700 1 -- 172.21.15.114:0/3525557604 learned_addr learned my addr 172.21.15.114:0/3525557604 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:15.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.349+0000 7f5463fff700 1 --2- 172.21.15.114:0/3525557604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5464100cb0 0x7f5464195790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:15.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.350+0000 7f5468ee4700 1 -- 172.21.15.114:0/3525557604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5464100cb0 msgr2=0x7f5464195790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:15.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.350+0000 7f5468ee4700 1 --2- 172.21.15.114:0/3525557604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5464100cb0 0x7f5464195790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:15.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.350+0000 7f5468ee4700 1 -- 172.21.15.114:0/3525557604 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54500095f0 con 0x7f54640ffab0 2024-02-19T19:00:15.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.350+0000 7f5468ee4700 1 --2- 172.21.15.114:0/3525557604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54640ffab0 0x7f5464195250 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f545800e8d0 tx=0x7f545800ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:15.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.350+0000 7f5461ffb700 1 -- 172.21.15.114:0/3525557604 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5458009770 con 0x7f54640ffab0 2024-02-19T19:00:15.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.350+0000 7f5461ffb700 1 -- 172.21.15.114:0/3525557604 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f545800c810 con 0x7f54640ffab0 2024-02-19T19:00:15.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.350+0000 7f546b148700 1 -- 172.21.15.114:0/3525557604 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546419a9a0 con 0x7f54640ffab0 2024-02-19T19:00:15.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.350+0000 7f5461ffb700 1 -- 172.21.15.114:0/3525557604 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5458003da0 con 0x7f54640ffab0 2024-02-19T19:00:15.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.350+0000 7f546b148700 1 -- 172.21.15.114:0/3525557604 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546419aef0 con 0x7f54640ffab0 2024-02-19T19:00:15.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.351+0000 7f546b148700 1 -- 172.21.15.114:0/3525557604 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5464108630 con 0x7f54640ffab0 2024-02-19T19:00:15.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.352+0000 7f5461ffb700 1 -- 172.21.15.114:0/3525557604 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f5458004200 con 0x7f54640ffab0 2024-02-19T19:00:15.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.352+0000 7f5461ffb700 1 --2- 172.21.15.114:0/3525557604 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f545406c4e0 0x7f545406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:15.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.352+0000 7f5461ffb700 1 -- 172.21.15.114:0/3525557604 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f5458014070 con 0x7f54640ffab0 2024-02-19T19:00:15.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.352+0000 7f5463fff700 1 --2- 172.21.15.114:0/3525557604 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f545406c4e0 0x7f545406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:15.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.352+0000 7f5463fff700 1 --2- 172.21.15.114:0/3525557604 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f545406c4e0 0x7f545406e9a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5450009fd0 tx=0x7f5450005940 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:15.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.356+0000 7f5461ffb700 1 -- 172.21.15.114:0/3525557604 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5458059d80 con 0x7f54640ffab0 2024-02-19T19:00:15.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:15 smithi174 ceph-mon[41061]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 175 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-02-19T19:00:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:15.610+0000 7f546b148700 1 -- 172.21.15.114:0/3525557604 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5464108820 con 0x7f54640ffab0 2024-02-19T19:00:16.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:16.025+0000 7f5461ffb700 1 -- 172.21.15.114:0/3525557604 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v10) v1 ==== 168+0+0 (secure 0 0 0) 0x7f545800ce60 con 0x7f54640ffab0 2024-02-19T19:00:16.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:16.027+0000 7f546b148700 1 -- 172.21.15.114:0/3525557604 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f545406c4e0 msgr2=0x7f545406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:16.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:16.027+0000 7f546b148700 1 --2- 172.21.15.114:0/3525557604 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f545406c4e0 0x7f545406e9a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5450009fd0 tx=0x7f5450005940 comp rx=0 tx=0).stop 2024-02-19T19:00:16.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:16.027+0000 7f546b148700 1 -- 172.21.15.114:0/3525557604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54640ffab0 msgr2=0x7f5464195250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:16.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:16.027+0000 7f546b148700 1 --2- 172.21.15.114:0/3525557604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54640ffab0 0x7f5464195250 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f545800e8d0 tx=0x7f545800ebe0 comp rx=0 tx=0).stop 2024-02-19T19:00:16.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:16.027+0000 7f546b148700 1 -- 172.21.15.114:0/3525557604 shutdown_connections 2024-02-19T19:00:16.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:16.027+0000 7f546b148700 1 --2- 172.21.15.114:0/3525557604 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f545406c4e0 0x7f545406e9a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:16.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:16.027+0000 7f546b148700 1 --2- 172.21.15.114:0/3525557604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54640ffab0 0x7f5464195250 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:16.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:16.027+0000 7f546b148700 1 --2- 172.21.15.114:0/3525557604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5464100cb0 0x7f5464195790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:16.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:16.027+0000 7f546b148700 1 -- 172.21.15.114:0/3525557604 >> 172.21.15.114:0/3525557604 conn(0x7f54640fb050 msgr2=0x7f5464103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:16.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:16.027+0000 7f546b148700 1 -- 172.21.15.114:0/3525557604 shutdown_connections 2024-02-19T19:00:16.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:16.027+0000 7f546b148700 1 -- 172.21.15.114:0/3525557604 wait complete. 2024-02-19T19:00:16.047 INFO:teuthology.orchestra.run.smithi114.stderr:inline data enabled 2024-02-19T19:00:16.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:16 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:16.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:16 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:16.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:16 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:16.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:16 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:16.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:16 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:16.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:16 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:16.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:16 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/3525557604' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-19T19:00:16.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:16 smithi174 ceph-mon[41061]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-19T19:00:16.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:16 smithi174 ceph-mon[41061]: mds.? [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] up:boot 2024-02-19T19:00:16.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:16 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/3525557604' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-19T19:00:16.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:16 smithi174 ceph-mon[41061]: fsmap cephfs:2 {0=cephfs.smithi114.vpoeop=up:active,1=cephfs.smithi174.znhuvd=up:active} 1 up:standby-replay 1 up:standby 2024-02-19T19:00:16.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:16 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.stnfnf"}]: dispatch 2024-02-19T19:00:16.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:16 smithi174 ceph-mon[41061]: fsmap cephfs:2 {0=cephfs.smithi114.vpoeop=up:active,1=cephfs.smithi174.znhuvd=up:active} 2 up:standby-replay 2024-02-19T19:00:16.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:16.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:16.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:16.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:16.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:16.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:16.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:16 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/3525557604' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-19T19:00:16.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:16 smithi114 ceph-mon[29441]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-19T19:00:16.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:16 smithi114 ceph-mon[29441]: mds.? [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] up:boot 2024-02-19T19:00:16.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:16 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/3525557604' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-19T19:00:16.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:16 smithi114 ceph-mon[29441]: fsmap cephfs:2 {0=cephfs.smithi114.vpoeop=up:active,1=cephfs.smithi174.znhuvd=up:active} 1 up:standby-replay 1 up:standby 2024-02-19T19:00:16.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.stnfnf"}]: dispatch 2024-02-19T19:00:16.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:16 smithi114 ceph-mon[29441]: fsmap cephfs:2 {0=cephfs.smithi114.vpoeop=up:active,1=cephfs.smithi174.znhuvd=up:active} 2 up:standby-replay 2024-02-19T19:00:16.912 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T19:00:16.922 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-02-19T19:00:16.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-19T19:00:17.493 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:17 smithi114 ceph-mon[29441]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-02-19T19:00:17.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:17 smithi174 ceph-mon[41061]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 341 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-02-19T19:00:18.416 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T19:00:18.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:18 smithi114 ceph-mon[29441]: mds.? [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] up:standby-replay 2024-02-19T19:00:18.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:18 smithi114 ceph-mon[29441]: mds.? [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] up:active 2024-02-19T19:00:18.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:18 smithi114 ceph-mon[29441]: Dropping low affinity active daemon mds.cephfs.smithi114.vpoeop in favor of higher affinity standby. 2024-02-19T19:00:18.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:18 smithi114 ceph-mon[29441]: Replacing daemon mds.cephfs.smithi114.vpoeop as rank 0 with standby daemon mds.cephfs.smithi114.tcwpod 2024-02-19T19:00:18.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:18 smithi114 ceph-mon[29441]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-19T19:00:18.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:18 smithi114 ceph-mon[29441]: fsmap cephfs:2 {0=cephfs.smithi114.vpoeop=up:active,1=cephfs.smithi174.znhuvd=up:active} 2 up:standby-replay 2024-02-19T19:00:18.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:18 smithi114 ceph-mon[29441]: osdmap e47: 6 total, 6 up, 6 in 2024-02-19T19:00:18.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:18 smithi114 ceph-mon[29441]: fsmap cephfs:2/2 {0=cephfs.smithi114.tcwpod=up:replay,1=cephfs.smithi174.znhuvd=up:active} 1 up:standby-replay 2024-02-19T19:00:18.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:18 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:18.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:18 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:18.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:18 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:18.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:18 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:18.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:18 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:18.544 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:18 smithi174 ceph-mon[41061]: mds.? [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] up:standby-replay 2024-02-19T19:00:18.544 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:18 smithi174 ceph-mon[41061]: mds.? [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] up:active 2024-02-19T19:00:18.545 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:18 smithi174 ceph-mon[41061]: Dropping low affinity active daemon mds.cephfs.smithi114.vpoeop in favor of higher affinity standby. 2024-02-19T19:00:18.545 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:18 smithi174 ceph-mon[41061]: Replacing daemon mds.cephfs.smithi114.vpoeop as rank 0 with standby daemon mds.cephfs.smithi114.tcwpod 2024-02-19T19:00:18.545 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:18 smithi174 ceph-mon[41061]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-19T19:00:18.545 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:18 smithi174 ceph-mon[41061]: fsmap cephfs:2 {0=cephfs.smithi114.vpoeop=up:active,1=cephfs.smithi174.znhuvd=up:active} 2 up:standby-replay 2024-02-19T19:00:18.545 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:18 smithi174 ceph-mon[41061]: osdmap e47: 6 total, 6 up, 6 in 2024-02-19T19:00:18.545 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:18 smithi174 ceph-mon[41061]: fsmap cephfs:2/2 {0=cephfs.smithi114.tcwpod=up:replay,1=cephfs.smithi174.znhuvd=up:active} 1 up:standby-replay 2024-02-19T19:00:18.545 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:18 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:18.545 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:18 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:18.545 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:18 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:18.545 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:18 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:18.545 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:18 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:19.388 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:19 smithi114 ceph-mon[29441]: mds.? [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] up:boot 2024-02-19T19:00:19.388 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:19 smithi114 ceph-mon[29441]: mds.? [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] up:resolve 2024-02-19T19:00:19.388 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:19 smithi114 ceph-mon[29441]: fsmap cephfs:2/2 {0=cephfs.smithi114.tcwpod=up:resolve,1=cephfs.smithi174.znhuvd=up:active} 1 up:standby-replay 1 up:standby 2024-02-19T19:00:19.388 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:19 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.vpoeop"}]: dispatch 2024-02-19T19:00:19.388 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:19 smithi114 ceph-mon[29441]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-02-19T19:00:19.555 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:19 smithi174 ceph-mon[41061]: mds.? [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] up:boot 2024-02-19T19:00:19.555 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:19 smithi174 ceph-mon[41061]: mds.? [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] up:resolve 2024-02-19T19:00:19.555 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:19 smithi174 ceph-mon[41061]: fsmap cephfs:2/2 {0=cephfs.smithi114.tcwpod=up:resolve,1=cephfs.smithi174.znhuvd=up:active} 1 up:standby-replay 1 up:standby 2024-02-19T19:00:19.555 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:19 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.vpoeop"}]: dispatch 2024-02-19T19:00:19.555 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:19 smithi174 ceph-mon[41061]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-02-19T19:00:20.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:20 smithi114 ceph-mon[29441]: mds.? [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] up:reconnect 2024-02-19T19:00:20.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:20 smithi114 ceph-mon[29441]: mds.? [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] up:standby-replay 2024-02-19T19:00:20.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:20 smithi114 ceph-mon[29441]: fsmap cephfs:2/2 {0=cephfs.smithi114.tcwpod=up:reconnect,1=cephfs.smithi174.znhuvd=up:active} 1 up:standby-replay 1 up:standby 2024-02-19T19:00:20.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:20 smithi174 ceph-mon[41061]: mds.? [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] up:reconnect 2024-02-19T19:00:20.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:20 smithi174 ceph-mon[41061]: mds.? [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] up:standby-replay 2024-02-19T19:00:20.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:20 smithi174 ceph-mon[41061]: fsmap cephfs:2/2 {0=cephfs.smithi114.tcwpod=up:reconnect,1=cephfs.smithi174.znhuvd=up:active} 1 up:standby-replay 1 up:standby 2024-02-19T19:00:21.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.155+0000 7f7cf107c700 1 -- 172.21.15.114:0/282512822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cec0ffab0 msgr2=0x7f7cec0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:21.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.155+0000 7f7cf107c700 1 --2- 172.21.15.114:0/282512822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cec0ffab0 0x7f7cec0ffed0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009910 tx=0x7f7cdc009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:21.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.155+0000 7f7cf107c700 1 -- 172.21.15.114:0/282512822 shutdown_connections 2024-02-19T19:00:21.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.155+0000 7f7cf107c700 1 --2- 172.21.15.114:0/282512822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cec100cb0 0x7f7cec101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:21.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.155+0000 7f7cf107c700 1 --2- 172.21.15.114:0/282512822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cec0ffab0 0x7f7cec0ffed0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:21.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.155+0000 7f7cf107c700 1 -- 172.21.15.114:0/282512822 >> 172.21.15.114:0/282512822 conn(0x7f7cec0fb050 msgr2=0x7f7cec0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:21.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.155+0000 7f7cf107c700 1 -- 172.21.15.114:0/282512822 shutdown_connections 2024-02-19T19:00:21.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.155+0000 7f7cf107c700 1 -- 172.21.15.114:0/282512822 wait complete. 2024-02-19T19:00:21.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.156+0000 7f7cf107c700 1 Processor -- start 2024-02-19T19:00:21.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.156+0000 7f7cf107c700 1 -- start start 2024-02-19T19:00:21.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.156+0000 7f7cf107c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cec0ffab0 0x7f7cec069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:21.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.156+0000 7f7cf107c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cec100cb0 0x7f7cec19b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:21.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.156+0000 7f7cf107c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec19b6a0 con 0x7f7cec100cb0 2024-02-19T19:00:21.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.156+0000 7f7cf107c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec19b7e0 con 0x7f7cec0ffab0 2024-02-19T19:00:21.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.157+0000 7f7cead9d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cec0ffab0 0x7f7cec069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:21.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.157+0000 7f7cea59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cec100cb0 0x7f7cec19b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:21.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.157+0000 7f7cea59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cec100cb0 0x7f7cec19b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50246/0 (socket says 172.21.15.114:50246) 2024-02-19T19:00:21.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.157+0000 7f7cea59c700 1 -- 172.21.15.114:0/195614664 learned_addr learned my addr 172.21.15.114:0/195614664 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:21.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.157+0000 7f7cea59c700 1 -- 172.21.15.114:0/195614664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cec0ffab0 msgr2=0x7f7cec069860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:21.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.157+0000 7f7cea59c700 1 --2- 172.21.15.114:0/195614664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cec0ffab0 0x7f7cec069860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:21.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.157+0000 7f7cea59c700 1 -- 172.21.15.114:0/195614664 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cdc0095f0 con 0x7f7cec100cb0 2024-02-19T19:00:21.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.157+0000 7f7cea59c700 1 --2- 172.21.15.114:0/195614664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cec100cb0 0x7f7cec19b080 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000d8d0 tx=0x7f7ce000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:21.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.157+0000 7f7cd3fff700 1 -- 172.21.15.114:0/195614664 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ce0009940 con 0x7f7cec100cb0 2024-02-19T19:00:21.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.158+0000 7f7cf107c700 1 -- 172.21.15.114:0/195614664 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cec19bac0 con 0x7f7cec100cb0 2024-02-19T19:00:21.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.158+0000 7f7cd3fff700 1 -- 172.21.15.114:0/195614664 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7ce0010460 con 0x7f7cec100cb0 2024-02-19T19:00:21.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.158+0000 7f7cf107c700 1 -- 172.21.15.114:0/195614664 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cec19c010 con 0x7f7cec100cb0 2024-02-19T19:00:21.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.158+0000 7f7cd3fff700 1 -- 172.21.15.114:0/195614664 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ce000f5f0 con 0x7f7cec100cb0 2024-02-19T19:00:21.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.159+0000 7f7cf107c700 1 -- 172.21.15.114:0/195614664 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cec108610 con 0x7f7cec100cb0 2024-02-19T19:00:21.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.159+0000 7f7cd3fff700 1 -- 172.21.15.114:0/195614664 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f7ce00105d0 con 0x7f7cec100cb0 2024-02-19T19:00:21.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.160+0000 7f7cd3fff700 1 --2- 172.21.15.114:0/195614664 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7cd406c4e0 0x7f7cd406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:21.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.160+0000 7f7cd3fff700 1 -- 172.21.15.114:0/195614664 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7ce008b870 con 0x7f7cec100cb0 2024-02-19T19:00:21.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.160+0000 7f7cead9d700 1 --2- 172.21.15.114:0/195614664 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7cd406c4e0 0x7f7cd406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:21.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.160+0000 7f7cead9d700 1 --2- 172.21.15.114:0/195614664 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7cd406c4e0 0x7f7cd406e9a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc006010 tx=0x7f7cdc00b540 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:21.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.165+0000 7f7cd3fff700 1 -- 172.21.15.114:0/195614664 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7ce0059a80 con 0x7f7cec100cb0 2024-02-19T19:00:21.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.428+0000 7f7cf107c700 1 -- 172.21.15.114:0/195614664 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7cec19c730 con 0x7f7cec100cb0 2024-02-19T19:00:21.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.428+0000 7f7cd3fff700 1 -- 172.21.15.114:0/195614664 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1850 (secure 0 0 0) 0x7f7ce0016750 con 0x7f7cec100cb0 2024-02-19T19:00:21.447 INFO:teuthology.orchestra.run.smithi114.stdout:e18 2024-02-19T19:00:21.447 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:00:21.447 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:00:21.447 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-02-19T19:00:21.448 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:00:21.448 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:00:21.448 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-02-19T19:00:21.448 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 18 2024-02-19T19:00:21.448 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:00:21.448 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:00:21.448 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-02-19T19:00:21.203922+0000 2024-02-19T19:00:21.448 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-02-19T19:00:21.448 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-02-19T19:00:21.448 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-02-19T19:00:21.448 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-02-19T19:00:21.448 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-02-19T19:00:21.448 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-02-19T19:00:21.448 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-02-19T19:00:21.448 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 47 2024-02-19T19:00:21.449 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:00:21.449 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 2 2024-02-19T19:00:21.449 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1 2024-02-19T19:00:21.449 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=14530,1=24269} 2024-02-19T19:00:21.449 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-02-19T19:00:21.449 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-02-19T19:00:21.449 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-02-19T19:00:21.449 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-02-19T19:00:21.449 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-02-19T19:00:21.449 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data enabled 2024-02-19T19:00:21.449 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-02-19T19:00:21.449 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-02-19T19:00:21.449 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 1 2024-02-19T19:00:21.449 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:00:21.450 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.vpoeop{0:14540} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:00:21.450 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.znhuvd{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:00:21.450 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.stnfnf{1:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:00:21.450 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:00:21.450 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:00:21.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.432+0000 7f7cf107c700 1 -- 172.21.15.114:0/195614664 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7cd406c4e0 msgr2=0x7f7cd406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:21.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.432+0000 7f7cf107c700 1 --2- 172.21.15.114:0/195614664 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7cd406c4e0 0x7f7cd406e9a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc006010 tx=0x7f7cdc00b540 comp rx=0 tx=0).stop 2024-02-19T19:00:21.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.432+0000 7f7cf107c700 1 -- 172.21.15.114:0/195614664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cec100cb0 msgr2=0x7f7cec19b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:21.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.432+0000 7f7cf107c700 1 --2- 172.21.15.114:0/195614664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cec100cb0 0x7f7cec19b080 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000d8d0 tx=0x7f7ce000dc90 comp rx=0 tx=0).stop 2024-02-19T19:00:21.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.432+0000 7f7cf107c700 1 -- 172.21.15.114:0/195614664 shutdown_connections 2024-02-19T19:00:21.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.432+0000 7f7cf107c700 1 --2- 172.21.15.114:0/195614664 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7cd406c4e0 0x7f7cd406e9a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:21.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.432+0000 7f7cf107c700 1 --2- 172.21.15.114:0/195614664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cec0ffab0 0x7f7cec069860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:21.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.432+0000 7f7cf107c700 1 --2- 172.21.15.114:0/195614664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7cec100cb0 0x7f7cec19b080 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:21.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.432+0000 7f7cf107c700 1 -- 172.21.15.114:0/195614664 >> 172.21.15.114:0/195614664 conn(0x7f7cec0fb050 msgr2=0x7f7cec103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:21.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.432+0000 7f7cf107c700 1 -- 172.21.15.114:0/195614664 shutdown_connections 2024-02-19T19:00:21.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:21.432+0000 7f7cf107c700 1 -- 172.21.15.114:0/195614664 wait complete. 2024-02-19T19:00:21.452 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 18 2024-02-19T19:00:21.463 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:21 smithi114 ceph-mon[29441]: mds.? [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] up:rejoin 2024-02-19T19:00:21.463 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:21 smithi114 ceph-mon[29441]: fsmap cephfs:2/2 {0=cephfs.smithi114.tcwpod=up:rejoin,1=cephfs.smithi174.znhuvd=up:active} 1 up:standby-replay 1 up:standby 2024-02-19T19:00:21.463 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:21 smithi114 ceph-mon[29441]: daemon mds.cephfs.smithi114.tcwpod is now active in filesystem cephfs as rank 0 2024-02-19T19:00:21.463 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:21 smithi114 ceph-mon[29441]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-02-19T19:00:21.642 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:21 smithi174 ceph-mon[41061]: mds.? [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] up:rejoin 2024-02-19T19:00:21.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:21 smithi174 ceph-mon[41061]: fsmap cephfs:2/2 {0=cephfs.smithi114.tcwpod=up:rejoin,1=cephfs.smithi174.znhuvd=up:active} 1 up:standby-replay 1 up:standby 2024-02-19T19:00:21.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:21 smithi174 ceph-mon[41061]: daemon mds.cephfs.smithi114.tcwpod is now active in filesystem cephfs as rank 0 2024-02-19T19:00:21.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:21 smithi174 ceph-mon[41061]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-02-19T19:00:22.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-02-19T19:00:22.322 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T19:00:22.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:22 smithi174 ceph-mon[41061]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-19T19:00:22.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:22 smithi174 ceph-mon[41061]: mds.? [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] up:active 2024-02-19T19:00:22.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:22 smithi174 ceph-mon[41061]: fsmap cephfs:2 {0=cephfs.smithi114.tcwpod=up:active,1=cephfs.smithi174.znhuvd=up:active} 1 up:standby-replay 1 up:standby 2024-02-19T19:00:22.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:22 smithi174 ceph-mon[41061]: fsmap cephfs:2 {0=cephfs.smithi114.tcwpod=up:active,1=cephfs.smithi174.znhuvd=up:active} 2 up:standby-replay 2024-02-19T19:00:22.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:22 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:22.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:22 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:22.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:22 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:22.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:22 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:22.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:22 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:22.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:22 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:22.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:22 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/195614664' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:00:22.404 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:22 smithi114 ceph-mon[29441]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-19T19:00:22.404 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:22 smithi114 ceph-mon[29441]: mds.? [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] up:active 2024-02-19T19:00:22.404 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:22 smithi114 ceph-mon[29441]: fsmap cephfs:2 {0=cephfs.smithi114.tcwpod=up:active,1=cephfs.smithi174.znhuvd=up:active} 1 up:standby-replay 1 up:standby 2024-02-19T19:00:22.404 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:22 smithi114 ceph-mon[29441]: fsmap cephfs:2 {0=cephfs.smithi114.tcwpod=up:active,1=cephfs.smithi174.znhuvd=up:active} 2 up:standby-replay 2024-02-19T19:00:22.404 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:22 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:22.404 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:22 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:22.404 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:22 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:22.404 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:22 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:22.405 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:22 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:22.405 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:22 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:22.405 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:22 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/195614664' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:00:24.260 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:23 smithi114 ceph-mon[29441]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2024-02-19T19:00:24.260 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:24.260 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:23 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:24.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:23 smithi174 ceph-mon[41061]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2024-02-19T19:00:24.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:24.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:23 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:24.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.472+0000 7f9af8dfb700 1 -- 172.21.15.114:0/2103180336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9af4101720 msgr2=0x7f9af4103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:24.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.472+0000 7f9af8dfb700 1 --2- 172.21.15.114:0/2103180336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9af4101720 0x7f9af4103b10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009910 tx=0x7f9aec009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:24.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.472+0000 7f9af8dfb700 1 -- 172.21.15.114:0/2103180336 shutdown_connections 2024-02-19T19:00:24.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.472+0000 7f9af8dfb700 1 --2- 172.21.15.114:0/2103180336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9af4104050 0x7f9af4106440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:24.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.472+0000 7f9af8dfb700 1 --2- 172.21.15.114:0/2103180336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9af4101720 0x7f9af4103b10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:24.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.472+0000 7f9af8dfb700 1 -- 172.21.15.114:0/2103180336 >> 172.21.15.114:0/2103180336 conn(0x7f9af40fb050 msgr2=0x7f9af40fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:24.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.473+0000 7f9af8dfb700 1 -- 172.21.15.114:0/2103180336 shutdown_connections 2024-02-19T19:00:24.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.473+0000 7f9af8dfb700 1 -- 172.21.15.114:0/2103180336 wait complete. 2024-02-19T19:00:24.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.473+0000 7f9af8dfb700 1 Processor -- start 2024-02-19T19:00:24.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.473+0000 7f9af8dfb700 1 -- start start 2024-02-19T19:00:24.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.473+0000 7f9af8dfb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9af4101720 0x7f9af4196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:24.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.473+0000 7f9af8dfb700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9af4104050 0x7f9af4196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:24.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.473+0000 7f9af8dfb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af4196e90 con 0x7f9af4101720 2024-02-19T19:00:24.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.473+0000 7f9af8dfb700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af4196fd0 con 0x7f9af4104050 2024-02-19T19:00:24.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.474+0000 7f9af259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9af4101720 0x7f9af4196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:24.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.474+0000 7f9af259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9af4101720 0x7f9af4196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50258/0 (socket says 172.21.15.114:50258) 2024-02-19T19:00:24.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.474+0000 7f9af259c700 1 -- 172.21.15.114:0/1898174354 learned_addr learned my addr 172.21.15.114:0/1898174354 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:24.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.474+0000 7f9af1d9b700 1 --2- 172.21.15.114:0/1898174354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9af4104050 0x7f9af4196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:24.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.474+0000 7f9af259c700 1 -- 172.21.15.114:0/1898174354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9af4104050 msgr2=0x7f9af4196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:24.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.474+0000 7f9af259c700 1 --2- 172.21.15.114:0/1898174354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9af4104050 0x7f9af4196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:24.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.474+0000 7f9af259c700 1 -- 172.21.15.114:0/1898174354 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aec0095f0 con 0x7f9af4101720 2024-02-19T19:00:24.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.474+0000 7f9af259c700 1 --2- 172.21.15.114:0/1898174354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9af4101720 0x7f9af4196330 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00bd70 tx=0x7f9aec00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:24.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.474+0000 7f9ae37fe700 1 -- 172.21.15.114:0/1898174354 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9aec01d070 con 0x7f9af4101720 2024-02-19T19:00:24.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.474+0000 7f9ae37fe700 1 -- 172.21.15.114:0/1898174354 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9aec005dc0 con 0x7f9af4101720 2024-02-19T19:00:24.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.474+0000 7f9ae37fe700 1 -- 172.21.15.114:0/1898174354 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9aec021410 con 0x7f9af4101720 2024-02-19T19:00:24.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.474+0000 7f9af8dfb700 1 -- 172.21.15.114:0/1898174354 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af419ba20 con 0x7f9af4101720 2024-02-19T19:00:24.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.474+0000 7f9af8dfb700 1 -- 172.21.15.114:0/1898174354 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af419bf10 con 0x7f9af4101720 2024-02-19T19:00:24.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.475+0000 7f9af8dfb700 1 -- 172.21.15.114:0/1898174354 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af4190540 con 0x7f9af4101720 2024-02-19T19:00:24.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.476+0000 7f9ae37fe700 1 -- 172.21.15.114:0/1898174354 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f9aec003a60 con 0x7f9af4101720 2024-02-19T19:00:24.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.476+0000 7f9ae37fe700 1 --2- 172.21.15.114:0/1898174354 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f9adc06c4e0 0x7f9adc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:24.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.476+0000 7f9ae37fe700 1 -- 172.21.15.114:0/1898174354 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f9aec08cb70 con 0x7f9af4101720 2024-02-19T19:00:24.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.476+0000 7f9af1d9b700 1 --2- 172.21.15.114:0/1898174354 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f9adc06c4e0 0x7f9adc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:24.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.476+0000 7f9af1d9b700 1 --2- 172.21.15.114:0/1898174354 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f9adc06c4e0 0x7f9adc06e9a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8005950 tx=0x7f9ae800b410 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:24.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.481+0000 7f9ae37fe700 1 -- 172.21.15.114:0/1898174354 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9aec05ad80 con 0x7f9af4101720 2024-02-19T19:00:24.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.719+0000 7f9af8dfb700 1 -- 172.21.15.114:0/1898174354 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9af402d100 con 0x7f9af4101720 2024-02-19T19:00:24.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.720+0000 7f9ae37fe700 1 -- 172.21.15.114:0/1898174354 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4776 (secure 0 0 0) 0x7f9aec05a910 con 0x7f9af4101720 2024-02-19T19:00:24.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.722+0000 7f9af8dfb700 1 -- 172.21.15.114:0/1898174354 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f9adc06c4e0 msgr2=0x7f9adc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:24.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.722+0000 7f9af8dfb700 1 --2- 172.21.15.114:0/1898174354 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f9adc06c4e0 0x7f9adc06e9a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8005950 tx=0x7f9ae800b410 comp rx=0 tx=0).stop 2024-02-19T19:00:24.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.722+0000 7f9af8dfb700 1 -- 172.21.15.114:0/1898174354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9af4101720 msgr2=0x7f9af4196330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:24.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.722+0000 7f9af8dfb700 1 --2- 172.21.15.114:0/1898174354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9af4101720 0x7f9af4196330 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00bd70 tx=0x7f9aec00bda0 comp rx=0 tx=0).stop 2024-02-19T19:00:24.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.722+0000 7f9af8dfb700 1 -- 172.21.15.114:0/1898174354 shutdown_connections 2024-02-19T19:00:24.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.722+0000 7f9af8dfb700 1 --2- 172.21.15.114:0/1898174354 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f9adc06c4e0 0x7f9adc06e9a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:24.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.722+0000 7f9af8dfb700 1 --2- 172.21.15.114:0/1898174354 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9af4101720 0x7f9af4196330 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:24.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.722+0000 7f9af8dfb700 1 --2- 172.21.15.114:0/1898174354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9af4104050 0x7f9af4196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:24.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.722+0000 7f9af8dfb700 1 -- 172.21.15.114:0/1898174354 >> 172.21.15.114:0/1898174354 conn(0x7f9af40fb050 msgr2=0x7f9af40fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:24.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.722+0000 7f9af8dfb700 1 -- 172.21.15.114:0/1898174354 shutdown_connections 2024-02-19T19:00:24.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:24.722+0000 7f9af8dfb700 1 -- 172.21.15.114:0/1898174354 wait complete. 2024-02-19T19:00:24.741 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 18 2024-02-19T19:00:24.750 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-02-19T19:00:25.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:24 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1898174354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-19T19:00:25.343 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-02-19T19:00:25.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:24 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1898174354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-19T19:00:25.808 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T19:00:26.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:25 smithi114 ceph-mon[29441]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-02-19T19:00:26.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:25 smithi174 ceph-mon[41061]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-02-19T19:00:28.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.039+0000 7f499e5a2700 1 -- 172.21.15.114:0/1269990284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4998100cb0 msgr2=0x7f4998101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:28.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.039+0000 7f499e5a2700 1 --2- 172.21.15.114:0/1269990284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4998100cb0 0x7f4998101110 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f498c009910 tx=0x7f498c009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:28.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.039+0000 7f499e5a2700 1 -- 172.21.15.114:0/1269990284 shutdown_connections 2024-02-19T19:00:28.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.039+0000 7f499e5a2700 1 --2- 172.21.15.114:0/1269990284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4998100cb0 0x7f4998101110 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:28.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.039+0000 7f499e5a2700 1 --2- 172.21.15.114:0/1269990284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49980ffab0 0x7f49980ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:28.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.039+0000 7f499e5a2700 1 -- 172.21.15.114:0/1269990284 >> 172.21.15.114:0/1269990284 conn(0x7f49980fb030 msgr2=0x7f49980fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:28.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.039+0000 7f499e5a2700 1 -- 172.21.15.114:0/1269990284 shutdown_connections 2024-02-19T19:00:28.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.039+0000 7f499e5a2700 1 -- 172.21.15.114:0/1269990284 wait complete. 2024-02-19T19:00:28.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.040+0000 7f499e5a2700 1 Processor -- start 2024-02-19T19:00:28.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.040+0000 7f499e5a2700 1 -- start start 2024-02-19T19:00:28.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.040+0000 7f499e5a2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49980ffab0 0x7f4998068fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:28.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.040+0000 7f499e5a2700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4998100cb0 0x7f499819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:28.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.040+0000 7f499e5a2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499819d9a0 con 0x7f49980ffab0 2024-02-19T19:00:28.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.040+0000 7f499e5a2700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499819dae0 con 0x7f4998100cb0 2024-02-19T19:00:28.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.040+0000 7f4997fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49980ffab0 0x7f4998068fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:28.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.040+0000 7f4997fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49980ffab0 0x7f4998068fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58466/0 (socket says 172.21.15.114:58466) 2024-02-19T19:00:28.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.040+0000 7f4997fff700 1 -- 172.21.15.114:0/842461964 learned_addr learned my addr 172.21.15.114:0/842461964 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:28.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.040+0000 7f49977fe700 1 --2- 172.21.15.114:0/842461964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4998100cb0 0x7f499819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:28.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.040+0000 7f4997fff700 1 -- 172.21.15.114:0/842461964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4998100cb0 msgr2=0x7f499819d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:28.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.040+0000 7f4997fff700 1 --2- 172.21.15.114:0/842461964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4998100cb0 0x7f499819d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:28.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.040+0000 7f4997fff700 1 -- 172.21.15.114:0/842461964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c0095f0 con 0x7f49980ffab0 2024-02-19T19:00:28.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.041+0000 7f4997fff700 1 --2- 172.21.15.114:0/842461964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49980ffab0 0x7f4998068fd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f498800d900 tx=0x7f498800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:28.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.041+0000 7f49957fa700 1 -- 172.21.15.114:0/842461964 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49880041f0 con 0x7f49980ffab0 2024-02-19T19:00:28.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.041+0000 7f49957fa700 1 -- 172.21.15.114:0/842461964 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4988004350 con 0x7f49980ffab0 2024-02-19T19:00:28.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.041+0000 7f499e5a2700 1 -- 172.21.15.114:0/842461964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499819dce0 con 0x7f49980ffab0 2024-02-19T19:00:28.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.041+0000 7f49957fa700 1 -- 172.21.15.114:0/842461964 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4988003d20 con 0x7f49980ffab0 2024-02-19T19:00:28.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.041+0000 7f499e5a2700 1 -- 172.21.15.114:0/842461964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f499819e1b0 con 0x7f49980ffab0 2024-02-19T19:00:28.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.042+0000 7f499e5a2700 1 -- 172.21.15.114:0/842461964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4998108610 con 0x7f49980ffab0 2024-02-19T19:00:28.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.042+0000 7f49957fa700 1 -- 172.21.15.114:0/842461964 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f4988009730 con 0x7f49980ffab0 2024-02-19T19:00:28.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.043+0000 7f49957fa700 1 --2- 172.21.15.114:0/842461964 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f498006c4e0 0x7f498006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:28.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.043+0000 7f49957fa700 1 -- 172.21.15.114:0/842461964 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f498808b730 con 0x7f49980ffab0 2024-02-19T19:00:28.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.043+0000 7f49977fe700 1 --2- 172.21.15.114:0/842461964 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f498006c4e0 0x7f498006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:28.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.043+0000 7f49977fe700 1 --2- 172.21.15.114:0/842461964 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f498006c4e0 0x7f498006e9a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f498c006010 tx=0x7f498c00b540 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:28.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.047+0000 7f49957fa700 1 -- 172.21.15.114:0/842461964 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4988059940 con 0x7f49980ffab0 2024-02-19T19:00:28.265 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:27 smithi114 ceph-mon[29441]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 11 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:00:28.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.329+0000 7f499e5a2700 1 -- 172.21.15.114:0/842461964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f49981087d0 con 0x7f49980ffab0 2024-02-19T19:00:28.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.329+0000 7f49957fa700 1 -- 172.21.15.114:0/842461964 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v18) v1 ==== 78+0+83 (secure 0 0 0) 0x7f4988003e80 con 0x7f49980ffab0 2024-02-19T19:00:28.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.331+0000 7f499e5a2700 1 -- 172.21.15.114:0/842461964 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f498006c4e0 msgr2=0x7f498006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:28.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.331+0000 7f499e5a2700 1 --2- 172.21.15.114:0/842461964 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f498006c4e0 0x7f498006e9a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f498c006010 tx=0x7f498c00b540 comp rx=0 tx=0).stop 2024-02-19T19:00:28.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.331+0000 7f499e5a2700 1 -- 172.21.15.114:0/842461964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49980ffab0 msgr2=0x7f4998068fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:28.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.331+0000 7f499e5a2700 1 --2- 172.21.15.114:0/842461964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49980ffab0 0x7f4998068fd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f498800d900 tx=0x7f498800dc10 comp rx=0 tx=0).stop 2024-02-19T19:00:28.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.331+0000 7f499e5a2700 1 -- 172.21.15.114:0/842461964 shutdown_connections 2024-02-19T19:00:28.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.332+0000 7f499e5a2700 1 --2- 172.21.15.114:0/842461964 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f498006c4e0 0x7f498006e9a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:28.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.332+0000 7f499e5a2700 1 --2- 172.21.15.114:0/842461964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49980ffab0 0x7f4998068fd0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:28.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.332+0000 7f499e5a2700 1 --2- 172.21.15.114:0/842461964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4998100cb0 0x7f499819d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:28.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.332+0000 7f499e5a2700 1 -- 172.21.15.114:0/842461964 >> 172.21.15.114:0/842461964 conn(0x7f49980fb030 msgr2=0x7f4998103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:28.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.332+0000 7f499e5a2700 1 -- 172.21.15.114:0/842461964 shutdown_connections 2024-02-19T19:00:28.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:28.332+0000 7f499e5a2700 1 -- 172.21.15.114:0/842461964 wait complete. 2024-02-19T19:00:28.360 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-02-19T19:00:28.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:27 smithi174 ceph-mon[41061]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 11 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:00:29.221 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:28 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/842461964' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-19T19:00:29.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:28 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/842461964' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-19T19:00:29.561 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-02-19T19:00:29.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T19:00:29.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.687+0000 7fc720c9d640 1 -- 172.21.15.114:0/3979297414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc71c0fe120 msgr2=0x7fc71c0fe5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:29.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.687+0000 7fc720c9d640 1 --2- 172.21.15.114:0/3979297414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc71c0fe120 0x7fc71c0fe5a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc710009880 tx=0x7fc71002bce0 comp rx=0 tx=0).stop 2024-02-19T19:00:29.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.687+0000 7fc720c9d640 1 -- 172.21.15.114:0/3979297414 shutdown_connections 2024-02-19T19:00:29.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.687+0000 7fc720c9d640 1 --2- 172.21.15.114:0/3979297414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc71c0fe120 0x7fc71c0fe5a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:29.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.687+0000 7fc720c9d640 1 --2- 172.21.15.114:0/3979297414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc71c1016b0 0x7fc71c101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:29.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.687+0000 7fc720c9d640 1 -- 172.21.15.114:0/3979297414 >> 172.21.15.114:0/3979297414 conn(0x7fc71c0f9c60 msgr2=0x7fc71c0fc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:29.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.687+0000 7fc720c9d640 1 -- 172.21.15.114:0/3979297414 shutdown_connections 2024-02-19T19:00:29.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.687+0000 7fc720c9d640 1 -- 172.21.15.114:0/3979297414 wait complete. 2024-02-19T19:00:29.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.688+0000 7fc720c9d640 1 Processor -- start 2024-02-19T19:00:29.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.688+0000 7fc720c9d640 1 -- start start 2024-02-19T19:00:29.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.688+0000 7fc720c9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc71c0fe120 0x7fc71c1958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:29.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.688+0000 7fc720c9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc71c1016b0 0x7fc71c195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:29.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.688+0000 7fc720c9d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c1963b0 con 0x7fc71c0fe120 2024-02-19T19:00:29.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.688+0000 7fc720c9d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c196520 con 0x7fc71c1016b0 2024-02-19T19:00:29.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.688+0000 7fc71a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc71c0fe120 0x7fc71c1958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:29.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.688+0000 7fc71a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc71c0fe120 0x7fc71c1958a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58480/0 (socket says 172.21.15.114:58480) 2024-02-19T19:00:29.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.688+0000 7fc71a59c640 1 -- 172.21.15.114:0/603985570 learned_addr learned my addr 172.21.15.114:0/603985570 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:29.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.688+0000 7fc719d9b640 1 --2- 172.21.15.114:0/603985570 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc71c1016b0 0x7fc71c195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:29.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.688+0000 7fc71a59c640 1 -- 172.21.15.114:0/603985570 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc71c1016b0 msgr2=0x7fc71c195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:29.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.688+0000 7fc71a59c640 1 --2- 172.21.15.114:0/603985570 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc71c1016b0 0x7fc71c195de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:29.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.688+0000 7fc71a59c640 1 -- 172.21.15.114:0/603985570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc710009560 con 0x7fc71c0fe120 2024-02-19T19:00:29.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.689+0000 7fc71a59c640 1 --2- 172.21.15.114:0/603985570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc71c0fe120 0x7fc71c1958a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00ea00 tx=0x7fc70c00eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:29.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.689+0000 7fc70b7fe640 1 -- 172.21.15.114:0/603985570 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc70c00cd50 con 0x7fc71c0fe120 2024-02-19T19:00:29.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.689+0000 7fc70b7fe640 1 -- 172.21.15.114:0/603985570 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc70c011920 con 0x7fc71c0fe120 2024-02-19T19:00:29.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.689+0000 7fc720c9d640 1 -- 172.21.15.114:0/603985570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71c19afb0 con 0x7fc71c0fe120 2024-02-19T19:00:29.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.689+0000 7fc70b7fe640 1 -- 172.21.15.114:0/603985570 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc70c01aa30 con 0x7fc71c0fe120 2024-02-19T19:00:29.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.689+0000 7fc720c9d640 1 -- 172.21.15.114:0/603985570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71c19b4b0 con 0x7fc71c0fe120 2024-02-19T19:00:29.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.690+0000 7fc720c9d640 1 -- 172.21.15.114:0/603985570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71c108e30 con 0x7fc71c0fe120 2024-02-19T19:00:29.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.690+0000 7fc70b7fe640 1 -- 172.21.15.114:0/603985570 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fc70c01ac60 con 0x7fc71c0fe120 2024-02-19T19:00:29.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.691+0000 7fc70b7fe640 1 --2- 172.21.15.114:0/603985570 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc70406c510 0x7fc70406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:29.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.691+0000 7fc719d9b640 1 --2- 172.21.15.114:0/603985570 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc70406c510 0x7fc70406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:29.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.691+0000 7fc70b7fe640 1 -- 172.21.15.114:0/603985570 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc70c08bf40 con 0x7fc71c0fe120 2024-02-19T19:00:29.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.691+0000 7fc719d9b640 1 --2- 172.21.15.114:0/603985570 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc70406c510 0x7fc70406e9d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc71c196db0 tx=0x7fc710000bc0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:29.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.695+0000 7fc70b7fe640 1 -- 172.21.15.114:0/603985570 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc70c0bf0e0 con 0x7fc71c0fe120 2024-02-19T19:00:29.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.870+0000 7fc720c9d640 1 -- 172.21.15.114:0/603985570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc71c19b730 con 0x7fc71c0fe120 2024-02-19T19:00:29.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.871+0000 7fc70b7fe640 1 -- 172.21.15.114:0/603985570 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+15171 (secure 0 0 0) 0x7fc70c0566c0 con 0x7fc71c0fe120 2024-02-19T19:00:29.890 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:00:29.890 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":47,"fsid":"be02a9a6-cf57-11ee-95bb-87774f69a715","created":"2024-02-19T18:51:33.283071+0000","modified":"2024-02-19T19:00:17.175713+0000","last_up_change":"2024-02-19T18:58:46.137044+0000","last_in_change":"2024-02-19T18:58:26.384214+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T18:57:30.943324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-02-19T18:59:59.211660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-02-19T19:00:00.203554+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"93a2d632-7629-4a15-b026-93b40fba59d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2204928688},{"type":"v1","addr":"172.21.15.114:6803","nonce":2204928688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2204928688},{"type":"v1","addr":"172.21.15.114:6805","nonce":2204928688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2204928688},{"type":"v1","addr":"172.21.15.114:6809","nonce":2204928688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2204928688},{"type":"v1","addr":"172.21.15.114:6807","nonce":2204928688}]},"public_addr":"172.21.15.114:6803/2204928688","cluster_addr":"172.21.15.114:6805/2204928688","heartbeat_back_addr":"172.21.15.114:6809/2204928688","heartbeat_front_addr":"172.21.15.114:6807/2204928688","state":["exists","up"]},{"osd":1,"uuid":"8334d634-9c49-4614-88e0-ca4545fa761b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":828515125},{"type":"v1","addr":"172.21.15.114:6811","nonce":828515125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":828515125},{"type":"v1","addr":"172.21.15.114:6813","nonce":828515125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":828515125},{"type":"v1","addr":"172.21.15.114:6817","nonce":828515125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":828515125},{"type":"v1","addr":"172.21.15.114:6815","nonce":828515125}]},"public_addr":"172.21.15.114:6811/828515125","cluster_addr":"172.21.15.114:6813/828515125","heartbeat_back_addr":"172.21.15.114:6817/828515125","heartbeat_front_addr":"172.21.15.114:6815/828515125","state":["exists","up"]},{"osd":2,"uuid":"2d27485a-1fb1-432d-a5a4-f46c8031a6ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":680148196},{"type":"v1","addr":"172.21.15.114:6819","nonce":680148196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":680148196},{"type":"v1","addr":"172.21.15.114:6821","nonce":680148196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":680148196},{"type":"v1","addr":"172.21.15.114:6825","nonce":680148196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":680148196},{"type":"v1","addr":"172.21.15.114:6823","nonce":680148196}]},"public_addr":"172.21.15.114:6819/680148196","cluster_addr":"172.21.15.114:6821/680148196","heartbeat_back_addr":"172.21.15.114:6825/680148196","heartbeat_front_addr":"172.21.15.114:6823/680148196","state":["exists","up"]},{"osd":3,"uuid":"fdbbfdf3-8e5c-472c-9894-97741a049a31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":2070964484},{"type":"v1","addr":"172.21.15.174:6801","nonce":2070964484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":2070964484},{"type":"v1","addr":"172.21.15.174:6803","nonce":2070964484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":2070964484},{"type":"v1","addr":"172.21.15.174:6807","nonce":2070964484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":2070964484},{"type":"v1","addr":"172.21.15.174:6805","nonce":2070964484}]},"public_addr":"172.21.15.174:6801/2070964484","cluster_addr":"172.21.15.174:6803/2070964484","heartbeat_back_addr":"172.21.15.174:6807/2070964484","heartbeat_front_addr":"172.21.15.174:6805/2070964484","state":["exists","up"]},{"osd":4,"uuid":"ebf7daeb-8e62-4a0d-ab1c-612cd6a8bf07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":2301992532},{"type":"v1","addr":"172.21.15.174:6809","nonce":2301992532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":2301992532},{"type":"v1","addr":"172.21.15.174:6811","nonce":2301992532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":2301992532},{"type":"v1","addr":"172.21.15.174:6815","nonce":2301992532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":2301992532},{"type":"v1","addr":"172.21.15.174:6813","nonce":2301992532}]},"public_addr":"172.21.15.174:6809/2301992532","cluster_addr":"172.21.15.174:6811/2301992532","heartbeat_back_addr":"172.21.15.174:6815/2301992532","heartbeat_front_addr":"172.21.15.174:6813/2301992532","state":["exists","up"]},{"osd":5,"uuid":"9a8599a6-778b-4367-bfb1-be184ee9d468","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1280361687},{"type":"v1","addr":"172.21.15.174:6817","nonce":1280361687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1280361687},{"type":"v1","addr":"172.21.15.174:6819","nonce":1280361687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1280361687},{"type":"v1","addr":"172.21.15.174:6823","nonce":1280361687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1280361687},{"type":"v1","addr":"172.21.15.174:6821","nonce":1280361687}]},"public_addr":"172.21.15.174:6817/1280361687","cluster_addr":"172.21.15.174:6819/1280361687","heartbeat_back_addr":"172.21.15.174:6823/1280361687","heartbeat_front_addr":"172.21.15.174:6821/1280361687","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:56:27.015050+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:56:56.239571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:57:26.893375+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:57:50.935953+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:16.875224+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:58:44.453528+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6827/98255571":"2024-02-20T19:00:17.175692+0000","172.21.15.114:0/2533620288":"2024-02-20T18:52:38.087813+0000","172.21.15.114:0/169338133":"2024-02-20T18:52:05.548945+0000","172.21.15.114:0/1245656641":"2024-02-20T18:52:05.548945+0000","172.21.15.114:0/945825118":"2024-02-20T18:52:38.087813+0000","172.21.15.114:0/1056805783":"2024-02-20T18:52:05.548945+0000","172.21.15.114:6800/971741305":"2024-02-20T18:52:38.087813+0000","172.21.15.114:6800/744561179":"2024-02-20T18:52:05.548945+0000","172.21.15.114:0/857878097":"2024-02-20T18:52:38.087813+0000","172.21.15.114:6800/1719431365":"2024-02-20T18:55:02.605941+0000","172.21.15.114:6801/971741305":"2024-02-20T18:52:38.087813+0000","172.21.15.114:0/110036562":"2024-02-20T18:55:02.605941+0000","172.21.15.114:6801/1719431365":"2024-02-20T18:55:02.605941+0000","172.21.15.114:0/3946301125":"2024-02-20T18:55:02.605941+0000","172.21.15.114:0/1382855124":"2024-02-20T18:55:02.605941+0000","172.21.15.114:6801/744561179":"2024-02-20T18:52:05.548945+0000","172.21.15.114:6826/98255571":"2024-02-20T19:00:17.175692+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T19:00:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.873+0000 7fc720c9d640 1 -- 172.21.15.114:0/603985570 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc70406c510 msgr2=0x7fc70406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.873+0000 7fc720c9d640 1 --2- 172.21.15.114:0/603985570 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc70406c510 0x7fc70406e9d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc71c196db0 tx=0x7fc710000bc0 comp rx=0 tx=0).stop 2024-02-19T19:00:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.873+0000 7fc720c9d640 1 -- 172.21.15.114:0/603985570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc71c0fe120 msgr2=0x7fc71c1958a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.873+0000 7fc720c9d640 1 --2- 172.21.15.114:0/603985570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc71c0fe120 0x7fc71c1958a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00ea00 tx=0x7fc70c00eec0 comp rx=0 tx=0).stop 2024-02-19T19:00:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.873+0000 7fc720c9d640 1 -- 172.21.15.114:0/603985570 shutdown_connections 2024-02-19T19:00:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.873+0000 7fc720c9d640 1 --2- 172.21.15.114:0/603985570 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc70406c510 0x7fc70406e9d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.873+0000 7fc720c9d640 1 --2- 172.21.15.114:0/603985570 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc71c1016b0 0x7fc71c195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.873+0000 7fc720c9d640 1 --2- 172.21.15.114:0/603985570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc71c0fe120 0x7fc71c1958a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:29.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.873+0000 7fc720c9d640 1 -- 172.21.15.114:0/603985570 >> 172.21.15.114:0/603985570 conn(0x7fc71c0f9c60 msgr2=0x7fc71c104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:29.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.873+0000 7fc720c9d640 1 -- 172.21.15.114:0/603985570 shutdown_connections 2024-02-19T19:00:29.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.873+0000 7fc720c9d640 1 -- 172.21.15.114:0/603985570 wait complete. 2024-02-19T19:00:29.903 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T18:57:30.943324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-02-19T18:59:59.211660+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-02-19T19:00:00.203554+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-19T19:00:29.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T19:00:30.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.990+0000 7fd2dc9c0640 1 -- 172.21.15.114:0/3671891272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2d81000c0 msgr2=0x7fd2d81004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:30.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.990+0000 7fd2dc9c0640 1 --2- 172.21.15.114:0/3671891272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2d81000c0 0x7fd2d81004e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009c20 tx=0x7fd2c802c1b0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.990+0000 7fd2dc9c0640 1 -- 172.21.15.114:0/3671891272 shutdown_connections 2024-02-19T19:00:30.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.990+0000 7fd2dc9c0640 1 --2- 172.21.15.114:0/3671891272 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2d81012b0 0x7fd2d8101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.990+0000 7fd2dc9c0640 1 --2- 172.21.15.114:0/3671891272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2d81000c0 0x7fd2d81004e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.990+0000 7fd2dc9c0640 1 -- 172.21.15.114:0/3671891272 >> 172.21.15.114:0/3671891272 conn(0x7fd2d80fb650 msgr2=0x7fd2d80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:30.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.990+0000 7fd2dc9c0640 1 -- 172.21.15.114:0/3671891272 shutdown_connections 2024-02-19T19:00:30.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.990+0000 7fd2dc9c0640 1 -- 172.21.15.114:0/3671891272 wait complete. 2024-02-19T19:00:30.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.991+0000 7fd2dc9c0640 1 Processor -- start 2024-02-19T19:00:30.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.991+0000 7fd2dc9c0640 1 -- start start 2024-02-19T19:00:30.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.991+0000 7fd2dc9c0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2d81000c0 0x7fd2d8195aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:30.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.991+0000 7fd2dc9c0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2d81012b0 0x7fd2d8195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:30.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.991+0000 7fd2dc9c0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d81965b0 con 0x7fd2d81012b0 2024-02-19T19:00:30.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.991+0000 7fd2dc9c0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d8196720 con 0x7fd2d81000c0 2024-02-19T19:00:30.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.991+0000 7fd2d5d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2d81012b0 0x7fd2d8195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:30.012 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.991+0000 7fd2d5d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2d81012b0 0x7fd2d8195fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58498/0 (socket says 172.21.15.114:58498) 2024-02-19T19:00:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.991+0000 7fd2d5d9b640 1 -- 172.21.15.114:0/2434129217 learned_addr learned my addr 172.21.15.114:0/2434129217 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.991+0000 7fd2d659c640 1 --2- 172.21.15.114:0/2434129217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2d81000c0 0x7fd2d8195aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.992+0000 7fd2d5d9b640 1 -- 172.21.15.114:0/2434129217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2d81000c0 msgr2=0x7fd2d8195aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.992+0000 7fd2d5d9b640 1 --2- 172.21.15.114:0/2434129217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2d81000c0 0x7fd2d8195aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.992+0000 7fd2d5d9b640 1 -- 172.21.15.114:0/2434129217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c8009900 con 0x7fd2d81012b0 2024-02-19T19:00:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.992+0000 7fd2d5d9b640 1 --2- 172.21.15.114:0/2434129217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2d81012b0 0x7fd2d8195fe0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000b420 tx=0x7fd2c000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.992+0000 7fd2cf7fe640 1 -- 172.21.15.114:0/2434129217 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2c0011800 con 0x7fd2d81012b0 2024-02-19T19:00:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.992+0000 7fd2cf7fe640 1 -- 172.21.15.114:0/2434129217 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd2c0011e10 con 0x7fd2d81012b0 2024-02-19T19:00:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.992+0000 7fd2dc9c0640 1 -- 172.21.15.114:0/2434129217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d819b1b0 con 0x7fd2d81012b0 2024-02-19T19:00:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.992+0000 7fd2cf7fe640 1 -- 172.21.15.114:0/2434129217 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2c000f2e0 con 0x7fd2d81012b0 2024-02-19T19:00:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.992+0000 7fd2dc9c0640 1 -- 172.21.15.114:0/2434129217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d819b790 con 0x7fd2d81012b0 2024-02-19T19:00:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.993+0000 7fd2dc9c0640 1 -- 172.21.15.114:0/2434129217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d8108e70 con 0x7fd2d81012b0 2024-02-19T19:00:30.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.993+0000 7fd2cf7fe640 1 -- 172.21.15.114:0/2434129217 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fd2c0010380 con 0x7fd2d81012b0 2024-02-19T19:00:30.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.994+0000 7fd2cf7fe640 1 --2- 172.21.15.114:0/2434129217 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd2c406c450 0x7fd2c406e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:30.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.994+0000 7fd2d659c640 1 --2- 172.21.15.114:0/2434129217 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd2c406c450 0x7fd2c406e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:30.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.994+0000 7fd2cf7fe640 1 -- 172.21.15.114:0/2434129217 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd2c008b260 con 0x7fd2d81012b0 2024-02-19T19:00:30.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.994+0000 7fd2d659c640 1 --2- 172.21.15.114:0/2434129217 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd2c406c450 0x7fd2c406e910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8006010 tx=0x7fd2c8009800 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:30.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:29.998+0000 7fd2cf7fe640 1 -- 172.21.15.114:0/2434129217 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd2c00559e0 con 0x7fd2d81012b0 2024-02-19T19:00:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.169+0000 7fd2dc9c0640 1 -- 172.21.15.114:0/2434129217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd2d8109000 con 0x7fd2d81012b0 2024-02-19T19:00:30.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.169+0000 7fd2cf7fe640 1 -- 172.21.15.114:0/2434129217 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd2d8109000 con 0x7fd2d81012b0 2024-02-19T19:00:30.189 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-02-19T19:00:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.171+0000 7fd2dc9c0640 1 -- 172.21.15.114:0/2434129217 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd2c406c450 msgr2=0x7fd2c406e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.171+0000 7fd2dc9c0640 1 --2- 172.21.15.114:0/2434129217 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd2c406c450 0x7fd2c406e910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8006010 tx=0x7fd2c8009800 comp rx=0 tx=0).stop 2024-02-19T19:00:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.171+0000 7fd2dc9c0640 1 -- 172.21.15.114:0/2434129217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2d81012b0 msgr2=0x7fd2d8195fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.171+0000 7fd2dc9c0640 1 --2- 172.21.15.114:0/2434129217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2d81012b0 0x7fd2d8195fe0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000b420 tx=0x7fd2c000b8e0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.171+0000 7fd2dc9c0640 1 -- 172.21.15.114:0/2434129217 shutdown_connections 2024-02-19T19:00:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.171+0000 7fd2dc9c0640 1 --2- 172.21.15.114:0/2434129217 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd2c406c450 0x7fd2c406e910 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.171+0000 7fd2dc9c0640 1 --2- 172.21.15.114:0/2434129217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd2d81012b0 0x7fd2d8195fe0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.171+0000 7fd2dc9c0640 1 --2- 172.21.15.114:0/2434129217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2d81000c0 0x7fd2d8195aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.171+0000 7fd2dc9c0640 1 -- 172.21.15.114:0/2434129217 >> 172.21.15.114:0/2434129217 conn(0x7fd2d80fb650 msgr2=0x7fd2d81044e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:30.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.172+0000 7fd2dc9c0640 1 -- 172.21.15.114:0/2434129217 shutdown_connections 2024-02-19T19:00:30.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.172+0000 7fd2dc9c0640 1 -- 172.21.15.114:0/2434129217 wait complete. 2024-02-19T19:00:30.200 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:29 smithi114 ceph-mon[29441]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-19T19:00:30.200 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:29 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/603985570' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T19:00:30.202 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-02-19T19:00:30.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.296+0000 7f6b43131640 1 -- 172.21.15.114:0/3220838922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b3c1000b0 msgr2=0x7f6b3c1004d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:30.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.296+0000 7f6b43131640 1 --2- 172.21.15.114:0/3220838922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b3c1000b0 0x7f6b3c1004d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6b30009880 tx=0x7f6b3002bce0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.297+0000 7f6b43131640 1 -- 172.21.15.114:0/3220838922 shutdown_connections 2024-02-19T19:00:30.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.297+0000 7f6b43131640 1 --2- 172.21.15.114:0/3220838922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b3c1012a0 0x7f6b3c101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.297+0000 7f6b43131640 1 --2- 172.21.15.114:0/3220838922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b3c1000b0 0x7f6b3c1004d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.297+0000 7f6b43131640 1 -- 172.21.15.114:0/3220838922 >> 172.21.15.114:0/3220838922 conn(0x7f6b3c0fb620 msgr2=0x7f6b3c0fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:30.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.297+0000 7f6b43131640 1 -- 172.21.15.114:0/3220838922 shutdown_connections 2024-02-19T19:00:30.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.297+0000 7f6b43131640 1 -- 172.21.15.114:0/3220838922 wait complete. 2024-02-19T19:00:30.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.297+0000 7f6b43131640 1 Processor -- start 2024-02-19T19:00:30.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.297+0000 7f6b43131640 1 -- start start 2024-02-19T19:00:30.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.297+0000 7f6b43131640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b3c1000b0 0x7f6b3c195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:30.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.297+0000 7f6b43131640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b3c1012a0 0x7f6b3c195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:30.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.297+0000 7f6b43131640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c1963f0 con 0x7f6b3c1000b0 2024-02-19T19:00:30.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.297+0000 7f6b43131640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c196560 con 0x7f6b3c1012a0 2024-02-19T19:00:30.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.297+0000 7f6b40ecd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b3c1000b0 0x7f6b3c195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:30.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.297+0000 7f6b40ecd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b3c1000b0 0x7f6b3c195970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58520/0 (socket says 172.21.15.114:58520) 2024-02-19T19:00:30.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.297+0000 7f6b40ecd640 1 -- 172.21.15.114:0/2425147982 learned_addr learned my addr 172.21.15.114:0/2425147982 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:30.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.298+0000 7f6b3bfff640 1 --2- 172.21.15.114:0/2425147982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b3c1012a0 0x7f6b3c195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:30.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.298+0000 7f6b40ecd640 1 -- 172.21.15.114:0/2425147982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b3c1012a0 msgr2=0x7f6b3c195eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:30.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.298+0000 7f6b40ecd640 1 --2- 172.21.15.114:0/2425147982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b3c1012a0 0x7f6b3c195eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.298+0000 7f6b40ecd640 1 -- 172.21.15.114:0/2425147982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b30009560 con 0x7f6b3c1000b0 2024-02-19T19:00:30.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.298+0000 7f6b40ecd640 1 --2- 172.21.15.114:0/2425147982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b3c1000b0 0x7f6b3c195970 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6b30000c00 tx=0x7f6b30004f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:30.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.298+0000 7f6b39ffb640 1 -- 172.21.15.114:0/2425147982 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b30004520 con 0x7f6b3c1000b0 2024-02-19T19:00:30.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.298+0000 7f6b43131640 1 -- 172.21.15.114:0/2425147982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3c19afd0 con 0x7f6b3c1000b0 2024-02-19T19:00:30.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.298+0000 7f6b39ffb640 1 -- 172.21.15.114:0/2425147982 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6b300054b0 con 0x7f6b3c1000b0 2024-02-19T19:00:30.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.298+0000 7f6b39ffb640 1 -- 172.21.15.114:0/2425147982 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b30034440 con 0x7f6b3c1000b0 2024-02-19T19:00:30.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.298+0000 7f6b43131640 1 -- 172.21.15.114:0/2425147982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3c19b450 con 0x7f6b3c1000b0 2024-02-19T19:00:30.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.300+0000 7f6b43131640 1 -- 172.21.15.114:0/2425147982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3c108e20 con 0x7f6b3c1000b0 2024-02-19T19:00:30.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.300+0000 7f6b39ffb640 1 -- 172.21.15.114:0/2425147982 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f6b30005000 con 0x7f6b3c1000b0 2024-02-19T19:00:30.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.300+0000 7f6b39ffb640 1 --2- 172.21.15.114:0/2425147982 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f6b2c06c450 0x7f6b2c06e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:30.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.300+0000 7f6b39ffb640 1 -- 172.21.15.114:0/2425147982 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f6b300aded0 con 0x7f6b3c1000b0 2024-02-19T19:00:30.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.300+0000 7f6b3bfff640 1 --2- 172.21.15.114:0/2425147982 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f6b2c06c450 0x7f6b2c06e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:30.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.301+0000 7f6b3bfff640 1 --2- 172.21.15.114:0/2425147982 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f6b2c06c450 0x7f6b2c06e910 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6b28005e10 tx=0x7f6b28005da0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:30.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.307+0000 7f6b39ffb640 1 -- 172.21.15.114:0/2425147982 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6b30078650 con 0x7f6b3c1000b0 2024-02-19T19:00:30.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:29 smithi174 ceph-mon[41061]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-19T19:00:30.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:29 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/603985570' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T19:00:30.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.510+0000 7f6b43131640 1 -- 172.21.15.114:0/2425147982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f6b3c19b700 con 0x7f6b3c1000b0 2024-02-19T19:00:30.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.510+0000 7f6b39ffb640 1 -- 172.21.15.114:0/2425147982 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f6b3007bc70 con 0x7f6b3c1000b0 2024-02-19T19:00:30.529 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-02-19T19:00:30.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.512+0000 7f6b43131640 1 -- 172.21.15.114:0/2425147982 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f6b2c06c450 msgr2=0x7f6b2c06e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:30.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.512+0000 7f6b43131640 1 --2- 172.21.15.114:0/2425147982 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f6b2c06c450 0x7f6b2c06e910 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6b28005e10 tx=0x7f6b28005da0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.512+0000 7f6b43131640 1 -- 172.21.15.114:0/2425147982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b3c1000b0 msgr2=0x7f6b3c195970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:30.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.512+0000 7f6b43131640 1 --2- 172.21.15.114:0/2425147982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b3c1000b0 0x7f6b3c195970 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6b30000c00 tx=0x7f6b30004f00 comp rx=0 tx=0).stop 2024-02-19T19:00:30.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.512+0000 7f6b43131640 1 -- 172.21.15.114:0/2425147982 shutdown_connections 2024-02-19T19:00:30.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.512+0000 7f6b43131640 1 --2- 172.21.15.114:0/2425147982 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f6b2c06c450 0x7f6b2c06e910 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.512+0000 7f6b43131640 1 --2- 172.21.15.114:0/2425147982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b3c1012a0 0x7f6b3c195eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.512+0000 7f6b43131640 1 --2- 172.21.15.114:0/2425147982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6b3c1000b0 0x7f6b3c195970 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.512+0000 7f6b43131640 1 -- 172.21.15.114:0/2425147982 >> 172.21.15.114:0/2425147982 conn(0x7f6b3c0fb620 msgr2=0x7f6b3c1044d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:30.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.512+0000 7f6b43131640 1 -- 172.21.15.114:0/2425147982 shutdown_connections 2024-02-19T19:00:30.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.512+0000 7f6b43131640 1 -- 172.21.15.114:0/2425147982 wait complete. 2024-02-19T19:00:30.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-02-19T19:00:30.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.632+0000 7f2270ed2640 1 -- 172.21.15.114:0/4240362273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f226c0fef90 msgr2=0x7f226c0ff3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.632+0000 7f2270ed2640 1 --2- 172.21.15.114:0/4240362273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f226c0fef90 0x7f226c0ff3b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f225c009880 tx=0x7f225c02bce0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.632+0000 7f2270ed2640 1 -- 172.21.15.114:0/4240362273 shutdown_connections 2024-02-19T19:00:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.632+0000 7f2270ed2640 1 --2- 172.21.15.114:0/4240362273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f226c100180 0x7f226c100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.632+0000 7f2270ed2640 1 --2- 172.21.15.114:0/4240362273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f226c0fef90 0x7f226c0ff3b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.632+0000 7f2270ed2640 1 -- 172.21.15.114:0/4240362273 >> 172.21.15.114:0/4240362273 conn(0x7f226c0fa4e0 msgr2=0x7f226c0fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:30.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.632+0000 7f2270ed2640 1 -- 172.21.15.114:0/4240362273 shutdown_connections 2024-02-19T19:00:30.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.632+0000 7f2270ed2640 1 -- 172.21.15.114:0/4240362273 wait complete. 2024-02-19T19:00:30.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.632+0000 7f2270ed2640 1 Processor -- start 2024-02-19T19:00:30.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.632+0000 7f2270ed2640 1 -- start start 2024-02-19T19:00:30.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.633+0000 7f2270ed2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f226c0fef90 0x7f226c197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:30.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.633+0000 7f2270ed2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f226c100180 0x7f226c198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:30.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.633+0000 7f2270ed2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c198730 con 0x7f226c100180 2024-02-19T19:00:30.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.633+0000 7f2270ed2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c1988a0 con 0x7f226c0fef90 2024-02-19T19:00:30.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.633+0000 7f2263fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f226c100180 0x7f226c198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:30.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.633+0000 7f2263fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f226c100180 0x7f226c198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58536/0 (socket says 172.21.15.114:58536) 2024-02-19T19:00:30.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.633+0000 7f2263fff640 1 -- 172.21.15.114:0/3330443525 learned_addr learned my addr 172.21.15.114:0/3330443525 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:30.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.633+0000 7f226a59c640 1 --2- 172.21.15.114:0/3330443525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f226c0fef90 0x7f226c197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:30.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.633+0000 7f2263fff640 1 -- 172.21.15.114:0/3330443525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f226c0fef90 msgr2=0x7f226c197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.633+0000 7f2263fff640 1 --2- 172.21.15.114:0/3330443525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f226c0fef90 0x7f226c197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.633+0000 7f2263fff640 1 -- 172.21.15.114:0/3330443525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225c009560 con 0x7f226c100180 2024-02-19T19:00:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.633+0000 7f2263fff640 1 --2- 172.21.15.114:0/3330443525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f226c100180 0x7f226c198160 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f225400b420 tx=0x7f225400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.633+0000 7f22637fe640 1 -- 172.21.15.114:0/3330443525 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2254011800 con 0x7f226c100180 2024-02-19T19:00:30.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.633+0000 7f2270ed2640 1 -- 172.21.15.114:0/3330443525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226c19d320 con 0x7f226c100180 2024-02-19T19:00:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.633+0000 7f22637fe640 1 -- 172.21.15.114:0/3330443525 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2254011e10 con 0x7f226c100180 2024-02-19T19:00:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.633+0000 7f22637fe640 1 -- 172.21.15.114:0/3330443525 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f225400f2e0 con 0x7f226c100180 2024-02-19T19:00:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.633+0000 7f2270ed2640 1 -- 172.21.15.114:0/3330443525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226c19d8d0 con 0x7f226c100180 2024-02-19T19:00:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.634+0000 7f2270ed2640 1 -- 172.21.15.114:0/3330443525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226c107bf0 con 0x7f226c100180 2024-02-19T19:00:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.635+0000 7f22637fe640 1 -- 172.21.15.114:0/3330443525 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2254011960 con 0x7f226c100180 2024-02-19T19:00:30.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.635+0000 7f22637fe640 1 --2- 172.21.15.114:0/3330443525 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f225806c450 0x7f225806e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:30.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.635+0000 7f22637fe640 1 -- 172.21.15.114:0/3330443525 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f225408b1b0 con 0x7f226c100180 2024-02-19T19:00:30.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.635+0000 7f226a59c640 1 --2- 172.21.15.114:0/3330443525 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f225806c450 0x7f225806e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:30.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.636+0000 7f226a59c640 1 --2- 172.21.15.114:0/3330443525 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f225806c450 0x7f225806e910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f225c000c00 tx=0x7f225c005fd0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:30.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.639+0000 7f22637fe640 1 -- 172.21.15.114:0/3330443525 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2254055930 con 0x7f226c100180 2024-02-19T19:00:30.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.808+0000 7f2270ed2640 1 -- 172.21.15.114:0/3330443525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f226c19db50 con 0x7f226c100180 2024-02-19T19:00:30.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.808+0000 7f22637fe640 1 -- 172.21.15.114:0/3330443525 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2254058f50 con 0x7f226c100180 2024-02-19T19:00:30.827 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-02-19T19:00:30.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.810+0000 7f2270ed2640 1 -- 172.21.15.114:0/3330443525 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f225806c450 msgr2=0x7f225806e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.810+0000 7f2270ed2640 1 --2- 172.21.15.114:0/3330443525 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f225806c450 0x7f225806e910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f225c000c00 tx=0x7f225c005fd0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.810+0000 7f2270ed2640 1 -- 172.21.15.114:0/3330443525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f226c100180 msgr2=0x7f226c198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.810+0000 7f2270ed2640 1 --2- 172.21.15.114:0/3330443525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f226c100180 0x7f226c198160 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f225400b420 tx=0x7f225400b8e0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.810+0000 7f2270ed2640 1 -- 172.21.15.114:0/3330443525 shutdown_connections 2024-02-19T19:00:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.810+0000 7f2270ed2640 1 --2- 172.21.15.114:0/3330443525 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f225806c450 0x7f225806e910 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.810+0000 7f2270ed2640 1 --2- 172.21.15.114:0/3330443525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f226c100180 0x7f226c198160 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.810+0000 7f2270ed2640 1 --2- 172.21.15.114:0/3330443525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f226c0fef90 0x7f226c197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.810+0000 7f2270ed2640 1 -- 172.21.15.114:0/3330443525 >> 172.21.15.114:0/3330443525 conn(0x7f226c0fa4e0 msgr2=0x7f226c067350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:30.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.810+0000 7f2270ed2640 1 -- 172.21.15.114:0/3330443525 shutdown_connections 2024-02-19T19:00:30.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.810+0000 7f2270ed2640 1 -- 172.21.15.114:0/3330443525 wait complete. 2024-02-19T19:00:30.840 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T19:00:30.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.923+0000 7f20a605f640 1 -- 172.21.15.114:0/3637538915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20a01012c0 msgr2=0x7f20a0101740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:30.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.923+0000 7f20a605f640 1 --2- 172.21.15.114:0/3637538915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20a01012c0 0x7f20a0101740 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2088009880 tx=0x7f208802bce0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.923+0000 7f20a605f640 1 -- 172.21.15.114:0/3637538915 shutdown_connections 2024-02-19T19:00:30.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.923+0000 7f20a605f640 1 --2- 172.21.15.114:0/3637538915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20a01012c0 0x7f20a0101740 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.923+0000 7f20a605f640 1 --2- 172.21.15.114:0/3637538915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20a01000d0 0x7f20a01004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.923+0000 7f20a605f640 1 -- 172.21.15.114:0/3637538915 >> 172.21.15.114:0/3637538915 conn(0x7f20a00fb620 msgr2=0x7f20a00fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:30.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.923+0000 7f20a605f640 1 -- 172.21.15.114:0/3637538915 shutdown_connections 2024-02-19T19:00:30.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.923+0000 7f20a605f640 1 -- 172.21.15.114:0/3637538915 wait complete. 2024-02-19T19:00:30.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.923+0000 7f20a605f640 1 Processor -- start 2024-02-19T19:00:30.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.923+0000 7f20a605f640 1 -- start start 2024-02-19T19:00:30.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.924+0000 7f20a605f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20a01000d0 0x7f20a0195890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:30.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.924+0000 7f20a605f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20a01012c0 0x7f20a0195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:30.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.924+0000 7f20a605f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a01963a0 con 0x7f20a01000d0 2024-02-19T19:00:30.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.924+0000 7f20a605f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a0196510 con 0x7f20a01012c0 2024-02-19T19:00:30.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.924+0000 7f209f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20a01000d0 0x7f20a0195890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:30.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.924+0000 7f209f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20a01000d0 0x7f20a0195890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58558/0 (socket says 172.21.15.114:58558) 2024-02-19T19:00:30.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.924+0000 7f209f7fe640 1 -- 172.21.15.114:0/3336949169 learned_addr learned my addr 172.21.15.114:0/3336949169 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:30.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.924+0000 7f2097fff640 1 --2- 172.21.15.114:0/3336949169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20a01012c0 0x7f20a0195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:30.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.924+0000 7f209f7fe640 1 -- 172.21.15.114:0/3336949169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20a01012c0 msgr2=0x7f20a0195dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:30.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.924+0000 7f209f7fe640 1 --2- 172.21.15.114:0/3336949169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20a01012c0 0x7f20a0195dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:30.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.924+0000 7f209f7fe640 1 -- 172.21.15.114:0/3336949169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2088009560 con 0x7f20a01000d0 2024-02-19T19:00:30.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.924+0000 7f209f7fe640 1 --2- 172.21.15.114:0/3336949169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20a01000d0 0x7f20a0195890 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f209000e940 tx=0x7f209000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:30.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.924+0000 7f209d7fa640 1 -- 172.21.15.114:0/3336949169 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20900096e0 con 0x7f20a01000d0 2024-02-19T19:00:30.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.924+0000 7f209d7fa640 1 -- 172.21.15.114:0/3336949169 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f20900044e0 con 0x7f20a01000d0 2024-02-19T19:00:30.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.924+0000 7f20a605f640 1 -- 172.21.15.114:0/3336949169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a019afa0 con 0x7f20a01000d0 2024-02-19T19:00:30.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.924+0000 7f209d7fa640 1 -- 172.21.15.114:0/3336949169 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2090010430 con 0x7f20a01000d0 2024-02-19T19:00:30.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.924+0000 7f20a605f640 1 -- 172.21.15.114:0/3336949169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a019b4a0 con 0x7f20a01000d0 2024-02-19T19:00:30.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.925+0000 7f20a605f640 1 -- 172.21.15.114:0/3336949169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a0108e60 con 0x7f20a01000d0 2024-02-19T19:00:30.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.926+0000 7f209d7fa640 1 -- 172.21.15.114:0/3336949169 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f209000cc90 con 0x7f20a01000d0 2024-02-19T19:00:30.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.926+0000 7f209d7fa640 1 --2- 172.21.15.114:0/3336949169 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f208c06c510 0x7f208c06e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:30.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.926+0000 7f209d7fa640 1 -- 172.21.15.114:0/3336949169 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2090014070 con 0x7f20a01000d0 2024-02-19T19:00:30.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.926+0000 7f2097fff640 1 --2- 172.21.15.114:0/3336949169 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f208c06c510 0x7f208c06e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:30.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.927+0000 7f2097fff640 1 --2- 172.21.15.114:0/3336949169 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f208c06c510 0x7f208c06e9d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f20a0196da0 tx=0x7f2088000bc0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:30.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:30.931+0000 7f209d7fa640 1 -- 172.21.15.114:0/3336949169 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2090056520 con 0x7f20a01000d0 2024-02-19T19:00:31.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:31.135+0000 7f20a605f640 1 -- 172.21.15.114:0/3336949169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20a019b720 con 0x7f20a01000d0 2024-02-19T19:00:31.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:31.136+0000 7f209d7fa640 1 -- 172.21.15.114:0/3336949169 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4776 (secure 0 0 0) 0x7f2090059b40 con 0x7f20a01000d0 2024-02-19T19:00:31.154 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:00:31.155 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":18,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-19T19:00:00.979838+0000","modified":"2024-02-19T19:00:21.203922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14530,"mds_1":24269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14530":{"gid":14530,"name":"cephfs.smithi114.tcwpod","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.114:6829/450622269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":450622269},{"type":"v1","addr":"172.21.15.114:6829","nonce":450622269}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14540":{"gid":14540,"name":"cephfs.smithi114.vpoeop","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6827/4219542841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":4219542841},{"type":"v1","addr":"172.21.15.114:6827","nonce":4219542841}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24269":{"gid":24269,"name":"cephfs.smithi174.znhuvd","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6825/3869337142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":3869337142},{"type":"v1","addr":"172.21.15.174:6825","nonce":3869337142}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24275":{"gid":24275,"name":"cephfs.smithi174.stnfnf","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.174:6827/963404733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":963404733},{"type":"v1","addr":"172.21.15.174:6827","nonce":963404733}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-19T19:00:31.155 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 18 2024-02-19T19:00:31.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:31.138+0000 7f20a605f640 1 -- 172.21.15.114:0/3336949169 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f208c06c510 msgr2=0x7f208c06e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:31.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:31.138+0000 7f20a605f640 1 --2- 172.21.15.114:0/3336949169 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f208c06c510 0x7f208c06e9d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f20a0196da0 tx=0x7f2088000bc0 comp rx=0 tx=0).stop 2024-02-19T19:00:31.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:31.138+0000 7f20a605f640 1 -- 172.21.15.114:0/3336949169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20a01000d0 msgr2=0x7f20a0195890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:31.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:31.138+0000 7f20a605f640 1 --2- 172.21.15.114:0/3336949169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20a01000d0 0x7f20a0195890 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f209000e940 tx=0x7f209000ee00 comp rx=0 tx=0).stop 2024-02-19T19:00:31.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:31.138+0000 7f20a605f640 1 -- 172.21.15.114:0/3336949169 shutdown_connections 2024-02-19T19:00:31.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:31.138+0000 7f20a605f640 1 --2- 172.21.15.114:0/3336949169 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f208c06c510 0x7f208c06e9d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:31.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:31.138+0000 7f20a605f640 1 --2- 172.21.15.114:0/3336949169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20a01012c0 0x7f20a0195dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:31.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:31.138+0000 7f20a605f640 1 --2- 172.21.15.114:0/3336949169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20a01000d0 0x7f20a0195890 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:31.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:31.138+0000 7f20a605f640 1 -- 172.21.15.114:0/3336949169 >> 172.21.15.114:0/3336949169 conn(0x7f20a00fb620 msgr2=0x7f20a01044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:31.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:31.138+0000 7f20a605f640 1 -- 172.21.15.114:0/3336949169 shutdown_connections 2024-02-19T19:00:31.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:31.138+0000 7f20a605f640 1 -- 172.21.15.114:0/3336949169 wait complete. 2024-02-19T19:00:31.166 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:30 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/2434129217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-19T19:00:31.166 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:30 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:31.166 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:30 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:31.166 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:30 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/2425147982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-19T19:00:31.166 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:30 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:31.166 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:30 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:31.166 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:30 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:31.166 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:30 smithi114 ceph-mon[29441]: pgmap v199: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-19T19:00:31.166 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:30 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/3330443525' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-19T19:00:31.167 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 18, 'max_mds': 2, 'flags': 50} 2024-02-19T19:00:31.168 INFO:teuthology.run_tasks:Running task kclient... 2024-02-19T19:00:31.192 INFO:tasks.kclient:Mounting kernel clients... 2024-02-19T19:00:31.192 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-02-19T19:00:31.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:31.192 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-02-19T19:00:31.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:31.223 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-02-19T19:00:31.291 INFO:teuthology.orchestra.run.smithi114.stderr:Cannot find device "ceph-brx" 2024-02-19T19:00:31.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T19:00:31.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:31.293 DEBUG:teuthology.orchestra.run.smithi174:> ip netns list 2024-02-19T19:00:31.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:31.312 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link delete ceph-brx 2024-02-19T19:00:31.380 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:30 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/2434129217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-19T19:00:31.381 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:30 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:31.381 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:30 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:31.381 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:30 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/2425147982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-19T19:00:31.381 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:30 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:31.381 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:30 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:31.381 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:30 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:31.381 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:30 smithi174 ceph-mon[41061]: pgmap v199: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-19T19:00:31.381 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:30 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/3330443525' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-19T19:00:31.388 INFO:teuthology.orchestra.run.smithi174.stderr:Cannot find device "ceph-brx" 2024-02-19T19:00:31.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T19:00:31.390 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-02-19T19:00:31.390 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-19T19:00:31.390 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T19:00:31.390 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph fs ls 2024-02-19T19:00:31.593 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T19:00:32.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:31 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/3336949169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-19T19:00:32.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:31 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/3336949169' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-19T19:00:33.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.101+0000 7fe553ebc700 1 -- 172.21.15.114:0/1205405115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54c101790 msgr2=0x7fe54c1057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:33.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.101+0000 7fe553ebc700 1 --2- 172.21.15.114:0/1205405115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54c101790 0x7fe54c1057e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009910 tx=0x7fe53c009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:33.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.101+0000 7fe553ebc700 1 -- 172.21.15.114:0/1205405115 shutdown_connections 2024-02-19T19:00:33.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.101+0000 7fe553ebc700 1 --2- 172.21.15.114:0/1205405115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54c101790 0x7fe54c1057e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:33.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.101+0000 7fe553ebc700 1 --2- 172.21.15.114:0/1205405115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe54c100de0 0x7fe54c1011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:33.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.101+0000 7fe553ebc700 1 -- 172.21.15.114:0/1205405115 >> 172.21.15.114:0/1205405115 conn(0x7fe54c0fc670 msgr2=0x7fe54c0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:33.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.101+0000 7fe553ebc700 1 -- 172.21.15.114:0/1205405115 shutdown_connections 2024-02-19T19:00:33.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.101+0000 7fe553ebc700 1 -- 172.21.15.114:0/1205405115 wait complete. 2024-02-19T19:00:33.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe553ebc700 1 Processor -- start 2024-02-19T19:00:33.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe553ebc700 1 -- start start 2024-02-19T19:00:33.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe553ebc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54c100de0 0x7fe54c198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:33.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe553ebc700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe54c101790 0x7fe54c198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:33.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe553ebc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c1994a0 con 0x7fe54c100de0 2024-02-19T19:00:33.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe553ebc700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c199610 con 0x7fe54c101790 2024-02-19T19:00:33.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe551c58700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54c100de0 0x7fe54c198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:33.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe551c58700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54c100de0 0x7fe54c198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58578/0 (socket says 172.21.15.114:58578) 2024-02-19T19:00:33.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe551c58700 1 -- 172.21.15.114:0/1683424113 learned_addr learned my addr 172.21.15.114:0/1683424113 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:33.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe551c58700 1 -- 172.21.15.114:0/1683424113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe54c101790 msgr2=0x7fe54c198ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:33.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe551c58700 1 --2- 172.21.15.114:0/1683424113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe54c101790 0x7fe54c198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:33.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe551c58700 1 -- 172.21.15.114:0/1683424113 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c0095f0 con 0x7fe54c100de0 2024-02-19T19:00:33.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe551c58700 1 --2- 172.21.15.114:0/1683424113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54c100de0 0x7fe54c198960 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe54800cfd0 tx=0x7fe54800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:33.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe542ffd700 1 -- 172.21.15.114:0/1683424113 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe54800ad70 con 0x7fe54c100de0 2024-02-19T19:00:33.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe542ffd700 1 -- 172.21.15.114:0/1683424113 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe5480128d0 con 0x7fe54c100de0 2024-02-19T19:00:33.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe553ebc700 1 -- 172.21.15.114:0/1683424113 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c19d490 con 0x7fe54c100de0 2024-02-19T19:00:33.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe542ffd700 1 -- 172.21.15.114:0/1683424113 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe548011880 con 0x7fe54c100de0 2024-02-19T19:00:33.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.102+0000 7fe553ebc700 1 -- 172.21.15.114:0/1683424113 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c19d9e0 con 0x7fe54c100de0 2024-02-19T19:00:33.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.103+0000 7fe553ebc700 1 -- 172.21.15.114:0/1683424113 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c104f60 con 0x7fe54c100de0 2024-02-19T19:00:33.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.104+0000 7fe542ffd700 1 -- 172.21.15.114:0/1683424113 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fe548012420 con 0x7fe54c100de0 2024-02-19T19:00:33.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.104+0000 7fe542ffd700 1 --2- 172.21.15.114:0/1683424113 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe53806c4e0 0x7fe53806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:33.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.105+0000 7fe551457700 1 --2- 172.21.15.114:0/1683424113 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe53806c4e0 0x7fe53806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:33.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.105+0000 7fe542ffd700 1 -- 172.21.15.114:0/1683424113 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fe54808bd40 con 0x7fe54c100de0 2024-02-19T19:00:33.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.105+0000 7fe551457700 1 --2- 172.21.15.114:0/1683424113 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe53806c4e0 0x7fe53806e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00b5c0 tx=0x7fe53c005fd0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:33.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.109+0000 7fe542ffd700 1 -- 172.21.15.114:0/1683424113 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe548059f50 con 0x7fe54c100de0 2024-02-19T19:00:33.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.342+0000 7fe553ebc700 1 -- 172.21.15.114:0/1683424113 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe54c04ea90 con 0x7fe54c100de0 2024-02-19T19:00:33.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.342+0000 7fe542ffd700 1 -- 172.21.15.114:0/1683424113 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7fe548059ae0 con 0x7fe54c100de0 2024-02-19T19:00:33.361 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-19T19:00:33.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.344+0000 7fe553ebc700 1 -- 172.21.15.114:0/1683424113 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe53806c4e0 msgr2=0x7fe53806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:33.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.344+0000 7fe553ebc700 1 --2- 172.21.15.114:0/1683424113 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe53806c4e0 0x7fe53806e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00b5c0 tx=0x7fe53c005fd0 comp rx=0 tx=0).stop 2024-02-19T19:00:33.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.344+0000 7fe553ebc700 1 -- 172.21.15.114:0/1683424113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54c100de0 msgr2=0x7fe54c198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:33.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.344+0000 7fe553ebc700 1 --2- 172.21.15.114:0/1683424113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54c100de0 0x7fe54c198960 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe54800cfd0 tx=0x7fe54800a340 comp rx=0 tx=0).stop 2024-02-19T19:00:33.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.345+0000 7fe553ebc700 1 -- 172.21.15.114:0/1683424113 shutdown_connections 2024-02-19T19:00:33.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.345+0000 7fe553ebc700 1 --2- 172.21.15.114:0/1683424113 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe53806c4e0 0x7fe53806e9a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:33.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.345+0000 7fe553ebc700 1 --2- 172.21.15.114:0/1683424113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54c100de0 0x7fe54c198960 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:33.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.345+0000 7fe553ebc700 1 --2- 172.21.15.114:0/1683424113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe54c101790 0x7fe54c198ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:33.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.345+0000 7fe553ebc700 1 -- 172.21.15.114:0/1683424113 >> 172.21.15.114:0/1683424113 conn(0x7fe54c0fc670 msgr2=0x7fe54c0fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:33.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.345+0000 7fe553ebc700 1 -- 172.21.15.114:0/1683424113 shutdown_connections 2024-02-19T19:00:33.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:33.345+0000 7fe553ebc700 1 -- 172.21.15.114:0/1683424113 wait complete. 2024-02-19T19:00:33.998 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:33 smithi114 ceph-mon[29441]: pgmap v200: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-19T19:00:33.998 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:33.998 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:33 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:33.998 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:33 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1683424113' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-19T19:00:34.021 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-19T19:00:34.021 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T19:00:34.021 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-02-19T19:00:34.022 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-19T19:00:34.022 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T19:00:34.022 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-19T19:00:34.022 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-19T19:00:34.022 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T19:00:34.022 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T19:00:34.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:34.022 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-02-19T19:00:34.038 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T19:00:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T19:00:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T19:00:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-02-19T19:00:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-02-19T19:00:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-02-19T19:00:34.040 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T19:00:34.040 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-02-19T19:00:34.040 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-02-19T19:00:34.040 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T19:00:34.041 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-02-19T19:00:34.041 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-02-19T19:00:34.041 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T19:00:34.041 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-02-19T19:00:34.041 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T19:00:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-02-19T19:00:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-19T19:00:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42074sec preferred_lft 42074sec 2024-02-19T19:00:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link 2024-02-19T19:00:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-02-19T19:00:34.043 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-19T19:00:34.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:34.043 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-02-19T19:00:34.043 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-02-19T19:00:34.043 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add name ceph-brx type bridge 2024-02-19T19:00:34.043 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr flush dev ceph-brx 2024-02-19T19:00:34.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set ceph-brx up 2024-02-19T19:00:34.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-19T19:00:34.044 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-02-19T19:00:34.052 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:33 smithi174 ceph-mon[41061]: pgmap v200: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.5 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-19T19:00:34.052 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:33 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:34.052 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:33 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:34.052 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:33 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1683424113' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-19T19:00:34.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:34.211 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T19:00:34.237 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-02-19T19:00:34.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:34.239 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-02-19T19:00:34.307 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-02-19T19:00:34.307 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T19:00:34.307 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-19T19:00:34.307 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-19T19:00:34.307 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T19:00:34.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:34.308 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-02-19T19:00:34.308 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-02-19T19:00:34.308 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-19T19:00:34.308 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-19T19:00:34.308 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-19T19:00:34.308 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-02-19T19:00:34.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:34.595 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-02-19T19:00:34.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:34.612 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list-id 2024-02-19T19:00:34.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:34.668 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-02-19T19:00:34.668 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-02-19T19:00:34.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-19T19:00:34.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-02-19T19:00:34.668 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-02-19T19:00:34.750 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-02-19T19:00:34.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:34.750 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-02-19T19:00:34.750 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-02-19T19:00:34.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-02-19T19:00:34.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-19T19:00:34.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-02-19T19:00:34.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-02-19T19:00:34.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-02-19T19:00:34.751 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-02-19T19:00:34.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:34.893 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec bash -c ' 2024-02-19T19:00:34.893 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-02-19T19:00:34.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-02-19T19:00:34.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set dev brx.0 master ceph-brx 2024-02-19T19:00:34.893 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-02-19T19:00:34.935 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T19:00:34.935 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-19T19:00:34.989 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-19T19:00:34.989 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T19:00:34.989 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-19T19:00:35.043 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T19:00:35.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:35.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T19:00:35.139 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T19:00:35.139 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T19:00:35.139 INFO:teuthology.orchestra.run.smithi114.stdout:invalid new device string format 2024-02-19T19:00:35.244 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T19:00:35.245 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T19:00:35.245 INFO:teuthology.orchestra.run.smithi114.stdout:invalid new device string format 2024-02-19T19:00:35.245 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.174:3300" 2024-02-19T19:00:35.245 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T19:00:35.250 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T19:00:35.251 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T19:00:35.251 INFO:teuthology.orchestra.run.smithi114.stdout:invalid new device string format 2024-02-19T19:00:35.251 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.174:3300" 2024-02-19T19:00:35.251 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T19:00:35.251 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:6789,172.21.15.174:6789" 2024-02-19T19:00:35.251 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.114:6789,172.21.15.174:6789:/ 2024-02-19T19:00:35.251 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=be02a9a6-cf57-11ee-95bb-87774f69a715" will pass to kernel 2024-02-19T19:00:35.252 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-02-19T19:00:35.252 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-02-19T19:00:35.252 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-19T19:00:35.252 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-02-19T19:00:35.252 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-19T19:00:35.254 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T19:00:35.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:35.254 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-19T19:00:35.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:35.282 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T19:00:35.342 INFO:teuthology.orchestra.run.smithi114.stdout:client24311 2024-02-19T19:00:35.342 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T19:00:35.342 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:00:35.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:35.400 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T19:00:35.460 INFO:teuthology.orchestra.run.smithi114.stdout:client24311 2024-02-19T19:00:35.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:35.460 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T19:00:35.521 INFO:teuthology.orchestra.run.smithi114.stdout:client24311 2024-02-19T19:00:35.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:35.521 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715.client24311/status') 2024-02-19T19:00:35.591 INFO:teuthology.orchestra.run.smithi114.stdout:instance: client.24311 (3)172.21.15.114:0/2149551086 2024-02-19T19:00:35.591 INFO:teuthology.orchestra.run.smithi114.stdout:blocklisted: false 2024-02-19T19:00:35.591 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records in 2024-02-19T19:00:35.591 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records out 2024-02-19T19:00:35.591 INFO:teuthology.orchestra.run.smithi114.stderr:72 bytes copied, 3.7824e-05 s, 1.9 MB/s 2024-02-19T19:00:35.592 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-02-19T19:00:35.592 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-19T19:00:35.593 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-19T19:00:35.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -- ceph fs ls 2024-02-19T19:00:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T19:00:35.963 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:35 smithi114 ceph-mon[29441]: pgmap v201: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.5 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-19T19:00:36.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:35 smithi174 ceph-mon[41061]: pgmap v201: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.5 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-19T19:00:37.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.451+0000 7f6e6c5f7700 1 -- 172.21.15.114:0/3994344314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e64101d00 msgr2=0x7f6e641020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:37.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.451+0000 7f6e6c5f7700 1 --2- 172.21.15.114:0/3994344314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e64101d00 0x7f6e641020e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c00b0a0 tx=0x7f6e5c00b3b0 comp rx=0 tx=0).stop 2024-02-19T19:00:37.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.451+0000 7f6e6c5f7700 1 -- 172.21.15.114:0/3994344314 shutdown_connections 2024-02-19T19:00:37.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.451+0000 7f6e6c5f7700 1 --2- 172.21.15.114:0/3994344314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e64102620 0x7f6e6410ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.451+0000 7f6e6c5f7700 1 --2- 172.21.15.114:0/3994344314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e64101d00 0x7f6e641020e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.451+0000 7f6e6c5f7700 1 -- 172.21.15.114:0/3994344314 >> 172.21.15.114:0/3994344314 conn(0x7f6e640fb570 msgr2=0x7f6e640fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.451+0000 7f6e6c5f7700 1 -- 172.21.15.114:0/3994344314 shutdown_connections 2024-02-19T19:00:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.451+0000 7f6e6c5f7700 1 -- 172.21.15.114:0/3994344314 wait complete. 2024-02-19T19:00:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.452+0000 7f6e6c5f7700 1 Processor -- start 2024-02-19T19:00:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.452+0000 7f6e6c5f7700 1 -- start start 2024-02-19T19:00:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.452+0000 7f6e6c5f7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e64101d00 0x7f6e640ff960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.452+0000 7f6e6c5f7700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e64102620 0x7f6e640ffea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.452+0000 7f6e6c5f7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e641003e0 con 0x7f6e64101d00 2024-02-19T19:00:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.452+0000 7f6e6c5f7700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e64100520 con 0x7f6e64102620 2024-02-19T19:00:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.453+0000 7f6e6a393700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e64101d00 0x7f6e640ff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.453+0000 7f6e6a393700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e64101d00 0x7f6e640ff960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60936/0 (socket says 172.21.15.114:60936) 2024-02-19T19:00:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.453+0000 7f6e6a393700 1 -- 172.21.15.114:0/152585168 learned_addr learned my addr 172.21.15.114:0/152585168 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.453+0000 7f6e69b92700 1 --2- 172.21.15.114:0/152585168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e64102620 0x7f6e640ffea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:37.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.453+0000 7f6e6a393700 1 -- 172.21.15.114:0/152585168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e64102620 msgr2=0x7f6e640ffea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.453+0000 7f6e6a393700 1 --2- 172.21.15.114:0/152585168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e64102620 0x7f6e640ffea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.453+0000 7f6e6a393700 1 -- 172.21.15.114:0/152585168 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c009d00 con 0x7f6e64101d00 2024-02-19T19:00:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.453+0000 7f6e6a393700 1 --2- 172.21.15.114:0/152585168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e64101d00 0x7f6e640ff960 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c015040 tx=0x7f6e5c0097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.453+0000 7f6e5b7fe700 1 -- 172.21.15.114:0/152585168 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e5c00e040 con 0x7f6e64101d00 2024-02-19T19:00:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.453+0000 7f6e6c5f7700 1 -- 172.21.15.114:0/152585168 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e641007a0 con 0x7f6e64101d00 2024-02-19T19:00:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.453+0000 7f6e5b7fe700 1 -- 172.21.15.114:0/152585168 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6e5c004040 con 0x7f6e64101d00 2024-02-19T19:00:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.453+0000 7f6e5b7fe700 1 -- 172.21.15.114:0/152585168 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e5c01d690 con 0x7f6e64101d00 2024-02-19T19:00:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.453+0000 7f6e6c5f7700 1 -- 172.21.15.114:0/152585168 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e641a2c80 con 0x7f6e64101d00 2024-02-19T19:00:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.454+0000 7f6e6c5f7700 1 -- 172.21.15.114:0/152585168 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e64108210 con 0x7f6e64101d00 2024-02-19T19:00:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.455+0000 7f6e5b7fe700 1 -- 172.21.15.114:0/152585168 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f6e5c0041b0 con 0x7f6e64101d00 2024-02-19T19:00:37.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.455+0000 7f6e5b7fe700 1 --2- 172.21.15.114:0/152585168 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f6e5006c4e0 0x7f6e5006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:37.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.455+0000 7f6e5b7fe700 1 -- 172.21.15.114:0/152585168 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f6e5c08d070 con 0x7f6e64101d00 2024-02-19T19:00:37.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.455+0000 7f6e69b92700 1 --2- 172.21.15.114:0/152585168 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f6e5006c4e0 0x7f6e5006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:37.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.456+0000 7f6e69b92700 1 --2- 172.21.15.114:0/152585168 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f6e5006c4e0 0x7f6e5006e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6e64101150 tx=0x7f6e54009260 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:37.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.460+0000 7f6e5b7fe700 1 -- 172.21.15.114:0/152585168 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6e5c0577f0 con 0x7f6e64101d00 2024-02-19T19:00:37.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.721+0000 7f6e6c5f7700 1 -- 172.21.15.114:0/152585168 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6e6404ea90 con 0x7f6e64101d00 2024-02-19T19:00:37.741 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:37 smithi114 ceph-mon[29441]: pgmap v202: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.2 KiB/s rd, 6 op/s 2024-02-19T19:00:37.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.721+0000 7f6e5b7fe700 1 -- 172.21.15.114:0/152585168 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7f6e5c05ae10 con 0x7f6e64101d00 2024-02-19T19:00:37.741 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-19T19:00:37.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.724+0000 7f6e6c5f7700 1 -- 172.21.15.114:0/152585168 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f6e5006c4e0 msgr2=0x7f6e5006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:37.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.724+0000 7f6e6c5f7700 1 --2- 172.21.15.114:0/152585168 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f6e5006c4e0 0x7f6e5006e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6e64101150 tx=0x7f6e54009260 comp rx=0 tx=0).stop 2024-02-19T19:00:37.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.724+0000 7f6e6c5f7700 1 -- 172.21.15.114:0/152585168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e64101d00 msgr2=0x7f6e640ff960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:37.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.724+0000 7f6e6c5f7700 1 --2- 172.21.15.114:0/152585168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e64101d00 0x7f6e640ff960 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c015040 tx=0x7f6e5c0097d0 comp rx=0 tx=0).stop 2024-02-19T19:00:37.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.725+0000 7f6e6c5f7700 1 -- 172.21.15.114:0/152585168 shutdown_connections 2024-02-19T19:00:37.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.725+0000 7f6e6c5f7700 1 --2- 172.21.15.114:0/152585168 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f6e5006c4e0 0x7f6e5006e9a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:37.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.725+0000 7f6e6c5f7700 1 --2- 172.21.15.114:0/152585168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e64101d00 0x7f6e640ff960 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:37.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.725+0000 7f6e6c5f7700 1 --2- 172.21.15.114:0/152585168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e64102620 0x7f6e640ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:37.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.725+0000 7f6e6c5f7700 1 -- 172.21.15.114:0/152585168 >> 172.21.15.114:0/152585168 conn(0x7f6e640fb570 msgr2=0x7f6e64105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:37.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.725+0000 7f6e6c5f7700 1 -- 172.21.15.114:0/152585168 shutdown_connections 2024-02-19T19:00:37.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:37.725+0000 7f6e6c5f7700 1 -- 172.21.15.114:0/152585168 wait complete. 2024-02-19T19:00:38.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:37 smithi174 ceph-mon[41061]: pgmap v202: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.2 KiB/s rd, 6 op/s 2024-02-19T19:00:38.376 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-19T19:00:38.377 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T19:00:38.377 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi174.front.sepia.ceph.com 2024-02-19T19:00:38.377 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-02-19T19:00:38.377 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-19T19:00:38.377 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-19T19:00:38.377 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-19T19:00:38.377 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T19:00:38.377 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-02-19T19:00:38.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:38.378 DEBUG:teuthology.orchestra.run.smithi174:> ip addr 2024-02-19T19:00:38.397 INFO:teuthology.orchestra.run.smithi174.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T19:00:38.398 INFO:teuthology.orchestra.run.smithi174.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T19:00:38.398 INFO:teuthology.orchestra.run.smithi174.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T19:00:38.398 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2024-02-19T19:00:38.398 INFO:teuthology.orchestra.run.smithi174.stdout: inet6 ::1/128 scope host 2024-02-19T19:00:38.398 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2024-02-19T19:00:38.398 INFO:teuthology.orchestra.run.smithi174.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T19:00:38.398 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 0c:c4:7a:d9:36:e0 brd ff:ff:ff:ff:ff:ff 2024-02-19T19:00:38.398 INFO:teuthology.orchestra.run.smithi174.stdout: altname enp7s0f0 2024-02-19T19:00:38.398 INFO:teuthology.orchestra.run.smithi174.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T19:00:38.398 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 0c:c4:7a:d9:36:e1 brd ff:ff:ff:ff:ff:ff 2024-02-19T19:00:38.398 INFO:teuthology.orchestra.run.smithi174.stdout: altname enp7s0f1 2024-02-19T19:00:38.398 INFO:teuthology.orchestra.run.smithi174.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T19:00:38.398 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 0c:c4:7a:88:77:1a brd ff:ff:ff:ff:ff:ff 2024-02-19T19:00:38.398 INFO:teuthology.orchestra.run.smithi174.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T19:00:38.399 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 0c:c4:7a:88:77:1b brd ff:ff:ff:ff:ff:ff 2024-02-19T19:00:38.399 INFO:teuthology.orchestra.run.smithi174.stdout: inet 172.21.15.174/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-19T19:00:38.399 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft 42072sec preferred_lft 42072sec 2024-02-19T19:00:38.399 INFO:teuthology.orchestra.run.smithi174.stdout: inet6 fe80::ec4:7aff:fe88:771b/64 scope link 2024-02-19T19:00:38.399 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2024-02-19T19:00:38.399 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-19T19:00:38.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:38.399 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec bash -c ' 2024-02-19T19:00:38.399 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2024-02-19T19:00:38.400 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link add name ceph-brx type bridge 2024-02-19T19:00:38.400 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip addr flush dev ceph-brx 2024-02-19T19:00:38.400 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link set ceph-brx up 2024-02-19T19:00:38.400 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-19T19:00:38.400 DEBUG:teuthology.orchestra.run.smithi174:> ') 2024-02-19T19:00:38.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:38.589 DEBUG:teuthology.orchestra.run.smithi174:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T19:00:38.615 INFO:teuthology.orchestra.run.smithi174.stdout:1 2024-02-19T19:00:38.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:38.617 DEBUG:teuthology.orchestra.run.smithi174:> route 2024-02-19T19:00:38.686 INFO:teuthology.orchestra.run.smithi174.stdout:Kernel IP routing table 2024-02-19T19:00:38.686 INFO:teuthology.orchestra.run.smithi174.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T19:00:38.687 INFO:teuthology.orchestra.run.smithi174.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-19T19:00:38.687 INFO:teuthology.orchestra.run.smithi174.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-19T19:00:38.687 INFO:teuthology.orchestra.run.smithi174.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T19:00:38.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:38.687 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec bash -c ' 2024-02-19T19:00:38.687 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2024-02-19T19:00:38.687 DEBUG:teuthology.orchestra.run.smithi174:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-19T19:00:38.687 DEBUG:teuthology.orchestra.run.smithi174:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-19T19:00:38.687 DEBUG:teuthology.orchestra.run.smithi174:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-19T19:00:38.687 DEBUG:teuthology.orchestra.run.smithi174:> ') 2024-02-19T19:00:38.751 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:38 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/152585168' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-19T19:00:38.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:38.997 DEBUG:teuthology.orchestra.run.smithi174:> ip netns list 2024-02-19T19:00:39.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:38 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/152585168' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-19T19:00:39.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:39.053 DEBUG:teuthology.orchestra.run.smithi174:> ip netns list-id 2024-02-19T19:00:39.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:39.109 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec bash -c ' 2024-02-19T19:00:39.109 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2024-02-19T19:00:39.109 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-19T19:00:39.109 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-02-19T19:00:39.109 DEBUG:teuthology.orchestra.run.smithi174:> ') 2024-02-19T19:00:39.212 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-02-19T19:00:39.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:39.212 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec bash -c ' 2024-02-19T19:00:39.212 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2024-02-19T19:00:39.212 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-02-19T19:00:39.212 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-19T19:00:39.212 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-02-19T19:00:39.212 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-02-19T19:00:39.213 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-02-19T19:00:39.213 DEBUG:teuthology.orchestra.run.smithi174:> ') 2024-02-19T19:00:39.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:39.312 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec bash -c ' 2024-02-19T19:00:39.313 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2024-02-19T19:00:39.313 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link set brx.0 up 2024-02-19T19:00:39.313 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link set dev brx.0 master ceph-brx 2024-02-19T19:00:39.313 DEBUG:teuthology.orchestra.run.smithi174:> ') 2024-02-19T19:00:39.354 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T19:00:39.354 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-02-19T19:00:39.410 INFO:teuthology.orchestra.run.smithi174.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-02-19T19:00:39.410 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T19:00:39.411 DEBUG:teuthology.orchestra.run.smithi174:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-02-19T19:00:39.465 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T19:00:39.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:39.465 DEBUG:teuthology.orchestra.run.smithi174:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-19T19:00:39.568 INFO:teuthology.orchestra.run.smithi174.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-19T19:00:39.568 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-19T19:00:39.568 INFO:teuthology.orchestra.run.smithi174.stdout:invalid new device string format 2024-02-19T19:00:39.680 INFO:teuthology.orchestra.run.smithi174.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-19T19:00:39.680 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-19T19:00:39.680 INFO:teuthology.orchestra.run.smithi174.stdout:invalid new device string format 2024-02-19T19:00:39.680 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.174:3300" 2024-02-19T19:00:39.680 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T19:00:39.691 INFO:teuthology.orchestra.run.smithi174.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-19T19:00:39.691 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-19T19:00:39.692 INFO:teuthology.orchestra.run.smithi174.stdout:invalid new device string format 2024-02-19T19:00:39.692 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.174:3300" 2024-02-19T19:00:39.692 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T19:00:39.692 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: resolved to: "172.21.15.114:6789,172.21.15.174:6789" 2024-02-19T19:00:39.692 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.114:6789,172.21.15.174:6789:/ 2024-02-19T19:00:39.692 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=be02a9a6-cf57-11ee-95bb-87774f69a715" will pass to kernel 2024-02-19T19:00:39.693 INFO:teuthology.orchestra.run.smithi174.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-02-19T19:00:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: You just mounted a file system that supports labels which does not 2024-02-19T19:00:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-19T19:00:39.693 INFO:teuthology.orchestra.run.smithi174.stdout: applications will generate AVC messages and not be allowed access to 2024-02-19T19:00:39.694 INFO:teuthology.orchestra.run.smithi174.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-19T19:00:39.694 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T19:00:39.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:39.695 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-02-19T19:00:39.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:39.721 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T19:00:39.777 INFO:teuthology.orchestra.run.smithi174.stdout:client14582 2024-02-19T19:00:39.777 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-02-19T19:00:39.777 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:00:39.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:39.833 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T19:00:39.889 INFO:teuthology.orchestra.run.smithi174.stdout:client14582 2024-02-19T19:00:39.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:39.889 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T19:00:39.945 INFO:teuthology.orchestra.run.smithi174.stdout:client14582 2024-02-19T19:00:39.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:00:39.945 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715.client14582/status') 2024-02-19T19:00:40.012 INFO:teuthology.orchestra.run.smithi174.stdout:instance: client.14582 (3)172.21.15.174:0/2538818139 2024-02-19T19:00:40.012 INFO:teuthology.orchestra.run.smithi174.stdout:blocklisted: false 2024-02-19T19:00:40.012 INFO:teuthology.orchestra.run.smithi174.stderr:0+1 records in 2024-02-19T19:00:40.012 INFO:teuthology.orchestra.run.smithi174.stderr:0+1 records out 2024-02-19T19:00:40.012 INFO:teuthology.orchestra.run.smithi174.stderr:72 bytes copied, 2.4117e-05 s, 3.0 MB/s 2024-02-19T19:00:40.013 INFO:teuthology.run_tasks:Running task print... 2024-02-19T19:00:40.024 INFO:teuthology.task.print:**** done client 2024-02-19T19:00:40.024 INFO:teuthology.run_tasks:Running task parallel... 2024-02-19T19:00:40.034 INFO:teuthology.task.parallel:starting parallel... 2024-02-19T19:00:40.035 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-19T19:00:40.035 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-19T19:00:40.035 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-02-19T19:00:40.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2024-02-19T19:00:40.036 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-19T19:00:40.036 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-02-19T19:00:40.040 INFO:tasks.workunit:Pulling workunits from ref 1ce7ea8877c709fa862eac608550374bd96e166e 2024-02-19T19:00:40.040 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-19T19:00:40.040 INFO:tasks.workunit:timeout=3h 2024-02-19T19:00:40.040 INFO:tasks.workunit:cleanup=True 2024-02-19T19:00:40.041 DEBUG:teuthology.orchestra.run.smithi114:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-19T19:00:40.060 INFO:teuthology.orchestra.run.smithi114.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-02-19T19:00:40.060 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-02-19T19:00:40.060 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 85h/133d Inode: 1 Links: 2 2024-02-19T19:00:40.061 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T19:00:40.061 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-19T19:00:40.061 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-19T19:00:40.061 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-02-19 19:00:05.986338813 +0000 2024-02-19T19:00:40.061 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-02-19 19:00:35.261851308 +0000 2024-02-19T19:00:40.061 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-02-19 19:00:05.986338813 +0000 2024-02-19T19:00:40.061 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-02-19T19:00:40.061 DEBUG:teuthology.orchestra.run.smithi114:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-02-19T19:00:40.134 DEBUG:teuthology.orchestra.run.smithi174:> stat -- /home/ubuntu/cephtest/mnt.1 2024-02-19T19:00:40.150 INFO:teuthology.orchestra.run.smithi174.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-02-19T19:00:40.151 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-02-19T19:00:40.151 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 7eh/126d Inode: 1 Links: 3 2024-02-19T19:00:40.151 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T19:00:40.151 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-19T19:00:40.151 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-19T19:00:40.151 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-02-19 19:00:40.110746921 +0000 2024-02-19T19:00:40.151 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-02-19 19:00:40.110746921 +0000 2024-02-19T19:00:40.151 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-02-19 19:00:05.986338813 +0000 2024-02-19T19:00:40.151 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-02-19T19:00:40.151 DEBUG:teuthology.orchestra.run.smithi174:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-02-19T19:00:40.225 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1ce7ea8877c709fa862eac608550374bd96e166e 2024-02-19T19:00:40.225 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/batrick/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 1ce7ea8877c709fa862eac608550374bd96e166e 2024-02-19T19:00:40.241 INFO:tasks.workunit.client.0.smithi114.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-19T19:00:40.266 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T19:00:40.281 INFO:tasks.workunit.client.1.smithi174.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-02-19T19:00:40.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:39 smithi114 ceph-mon[29441]: pgmap v203: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-02-19T19:00:40.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:39 smithi174 ceph-mon[41061]: pgmap v203: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-02-19T19:00:41.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.869+0000 7f5097064700 1 -- 172.21.15.114:0/3691550837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50900ffab0 msgr2=0x7f50900ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:41.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.869+0000 7f5097064700 1 --2- 172.21.15.114:0/3691550837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50900ffab0 0x7f50900ffed0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5084009910 tx=0x7f5084009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:41.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.869+0000 7f5097064700 1 -- 172.21.15.114:0/3691550837 shutdown_connections 2024-02-19T19:00:41.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.869+0000 7f5097064700 1 --2- 172.21.15.114:0/3691550837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5090100cb0 0x7f5090101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:41.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.869+0000 7f5097064700 1 --2- 172.21.15.114:0/3691550837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50900ffab0 0x7f50900ffed0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:41.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.869+0000 7f5097064700 1 -- 172.21.15.114:0/3691550837 >> 172.21.15.114:0/3691550837 conn(0x7f50900fb030 msgr2=0x7f50900fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:41.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.869+0000 7f5097064700 1 -- 172.21.15.114:0/3691550837 shutdown_connections 2024-02-19T19:00:41.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.869+0000 7f5097064700 1 -- 172.21.15.114:0/3691550837 wait complete. 2024-02-19T19:00:41.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.870+0000 7f5097064700 1 Processor -- start 2024-02-19T19:00:41.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.870+0000 7f5097064700 1 -- start start 2024-02-19T19:00:41.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.870+0000 7f5097064700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50900ffab0 0x7f5090068f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:41.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.870+0000 7f5097064700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5090100cb0 0x7f509019b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:41.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.870+0000 7f5097064700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509019b5e0 con 0x7f50900ffab0 2024-02-19T19:00:41.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.870+0000 7f5097064700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509019b720 con 0x7f5090100cb0 2024-02-19T19:00:41.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.870+0000 7f5094e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50900ffab0 0x7f5090068f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:41.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.870+0000 7f5094e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50900ffab0 0x7f5090068f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60950/0 (socket says 172.21.15.114:60950) 2024-02-19T19:00:41.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.870+0000 7f5094e00700 1 -- 172.21.15.114:0/1172283432 learned_addr learned my addr 172.21.15.114:0/1172283432 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:41.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.870+0000 7f508ffff700 1 --2- 172.21.15.114:0/1172283432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5090100cb0 0x7f509019b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:41.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.870+0000 7f5094e00700 1 -- 172.21.15.114:0/1172283432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5090100cb0 msgr2=0x7f509019b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:41.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.870+0000 7f5094e00700 1 --2- 172.21.15.114:0/1172283432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5090100cb0 0x7f509019b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:41.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.870+0000 7f5094e00700 1 -- 172.21.15.114:0/1172283432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50840095f0 con 0x7f50900ffab0 2024-02-19T19:00:41.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.870+0000 7f508ffff700 1 --2- 172.21.15.114:0/1172283432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5090100cb0 0x7f509019b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:00:41.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.871+0000 7f5094e00700 1 --2- 172.21.15.114:0/1172283432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50900ffab0 0x7f5090068f70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f508400bda0 tx=0x7f508400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:41.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.871+0000 7f508dffb700 1 -- 172.21.15.114:0/1172283432 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f508401d070 con 0x7f50900ffab0 2024-02-19T19:00:41.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.871+0000 7f508dffb700 1 -- 172.21.15.114:0/1172283432 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5084004c50 con 0x7f50900ffab0 2024-02-19T19:00:41.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.871+0000 7f508dffb700 1 -- 172.21.15.114:0/1172283432 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f508400f690 con 0x7f50900ffab0 2024-02-19T19:00:41.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.871+0000 7f5097064700 1 -- 172.21.15.114:0/1172283432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509019b9a0 con 0x7f50900ffab0 2024-02-19T19:00:41.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.871+0000 7f5097064700 1 -- 172.21.15.114:0/1172283432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509019be90 con 0x7f50900ffab0 2024-02-19T19:00:41.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.872+0000 7f5097064700 1 -- 172.21.15.114:0/1172283432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5090108610 con 0x7f50900ffab0 2024-02-19T19:00:41.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.872+0000 7f508dffb700 1 -- 172.21.15.114:0/1172283432 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f5084022a20 con 0x7f50900ffab0 2024-02-19T19:00:41.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.873+0000 7f508dffb700 1 --2- 172.21.15.114:0/1172283432 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f508006c4e0 0x7f508006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:41.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.873+0000 7f508dffb700 1 -- 172.21.15.114:0/1172283432 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f508408cf70 con 0x7f50900ffab0 2024-02-19T19:00:41.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.873+0000 7f508ffff700 1 --2- 172.21.15.114:0/1172283432 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f508006c4e0 0x7f508006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:41.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.873+0000 7f508ffff700 1 --2- 172.21.15.114:0/1172283432 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f508006c4e0 0x7f508006e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f507c005fd0 tx=0x7f507c005e30 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:41.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:41.877+0000 7f508dffb700 1 -- 172.21.15.114:0/1172283432 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f50840c00e0 con 0x7f50900ffab0 2024-02-19T19:00:42.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:42.071+0000 7f5097064700 1 -- 172.21.15.114:0/1172283432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f5090108830 con 0x7f50900ffab0 2024-02-19T19:00:42.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:42.076+0000 7f508dffb700 1 -- 172.21.15.114:0/1172283432 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v22)=0 v22) v1 ==== 126+0+0 (secure 0 0 0) 0x7f5090108830 con 0x7f50900ffab0 2024-02-19T19:00:42.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:42.078+0000 7f5097064700 1 -- 172.21.15.114:0/1172283432 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f508006c4e0 msgr2=0x7f508006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:42.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:42.078+0000 7f5097064700 1 --2- 172.21.15.114:0/1172283432 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f508006c4e0 0x7f508006e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f507c005fd0 tx=0x7f507c005e30 comp rx=0 tx=0).stop 2024-02-19T19:00:42.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:42.078+0000 7f5097064700 1 -- 172.21.15.114:0/1172283432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50900ffab0 msgr2=0x7f5090068f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:42.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:42.078+0000 7f5097064700 1 --2- 172.21.15.114:0/1172283432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50900ffab0 0x7f5090068f70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f508400bda0 tx=0x7f508400be80 comp rx=0 tx=0).stop 2024-02-19T19:00:42.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:42.078+0000 7f5097064700 1 -- 172.21.15.114:0/1172283432 shutdown_connections 2024-02-19T19:00:42.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:42.078+0000 7f5097064700 1 --2- 172.21.15.114:0/1172283432 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f508006c4e0 0x7f508006e9a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:42.078+0000 7f5097064700 1 --2- 172.21.15.114:0/1172283432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50900ffab0 0x7f5090068f70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:42.078+0000 7f5097064700 1 --2- 172.21.15.114:0/1172283432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5090100cb0 0x7f509019b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:42.078+0000 7f5097064700 1 -- 172.21.15.114:0/1172283432 >> 172.21.15.114:0/1172283432 conn(0x7f50900fb030 msgr2=0x7f5090103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:42.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:42.078+0000 7f5097064700 1 -- 172.21.15.114:0/1172283432 shutdown_connections 2024-02-19T19:00:42.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:42.078+0000 7f5097064700 1 -- 172.21.15.114:0/1172283432 wait complete. 2024-02-19T19:00:42.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:41 smithi114 ceph-mon[29441]: pgmap v204: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 4 op/s 2024-02-19T19:00:42.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:41 smithi174 ceph-mon[41061]: pgmap v204: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 4 op/s 2024-02-19T19:00:42.850 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-19T19:00:42.851 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-02-19T19:00:42.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-19T19:00:43.102 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T19:00:43.369 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:43 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1172283432' entity='client.admin' 2024-02-19T19:00:43.369 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:43 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:43.369 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:43 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:43.369 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:43 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:43.369 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:43 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:43.369 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:43 smithi114 ceph-mon[29441]: pgmap v205: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-02-19T19:00:43.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:43 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1172283432' entity='client.admin' 2024-02-19T19:00:43.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:43 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:43.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:43 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:43.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:43 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:43.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:43 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:43.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:43 smithi174 ceph-mon[41061]: pgmap v205: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-02-19T19:00:44.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.910+0000 7f9abdf80700 1 -- 172.21.15.114:0/4027533168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab8100bc0 msgr2=0x7f9ab8100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:44.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.910+0000 7f9abdf80700 1 --2- 172.21.15.114:0/4027533168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab8100bc0 0x7f9ab8100fe0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0009910 tx=0x7f9ab0009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:44.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.910+0000 7f9abdf80700 1 -- 172.21.15.114:0/4027533168 shutdown_connections 2024-02-19T19:00:44.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.910+0000 7f9abdf80700 1 --2- 172.21.15.114:0/4027533168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ab8101dc0 0x7f9ab8102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:44.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.910+0000 7f9abdf80700 1 --2- 172.21.15.114:0/4027533168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab8100bc0 0x7f9ab8100fe0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:44.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.910+0000 7f9abdf80700 1 -- 172.21.15.114:0/4027533168 >> 172.21.15.114:0/4027533168 conn(0x7f9ab80fc140 msgr2=0x7f9ab80fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:44.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.910+0000 7f9abdf80700 1 -- 172.21.15.114:0/4027533168 shutdown_connections 2024-02-19T19:00:44.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.910+0000 7f9abdf80700 1 -- 172.21.15.114:0/4027533168 wait complete. 2024-02-19T19:00:44.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.910+0000 7f9abdf80700 1 Processor -- start 2024-02-19T19:00:44.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.911+0000 7f9abdf80700 1 -- start start 2024-02-19T19:00:44.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.911+0000 7f9abdf80700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ab8100bc0 0x7f9ab8196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:44.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.911+0000 7f9abdf80700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab8101dc0 0x7f9ab81968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:44.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.911+0000 7f9abdf80700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab8196ef0 con 0x7f9ab8101dc0 2024-02-19T19:00:44.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.911+0000 7f9abdf80700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab8197030 con 0x7f9ab8100bc0 2024-02-19T19:00:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.911+0000 7f9ab6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab8101dc0 0x7f9ab81968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.911+0000 7f9ab6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab8101dc0 0x7f9ab81968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60966/0 (socket says 172.21.15.114:60966) 2024-02-19T19:00:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.911+0000 7f9ab6ffd700 1 -- 172.21.15.114:0/2660955849 learned_addr learned my addr 172.21.15.114:0/2660955849 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.911+0000 7f9ab77fe700 1 --2- 172.21.15.114:0/2660955849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ab8100bc0 0x7f9ab8196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:44.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.911+0000 7f9ab6ffd700 1 -- 172.21.15.114:0/2660955849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ab8100bc0 msgr2=0x7f9ab8196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:44.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.911+0000 7f9ab6ffd700 1 --2- 172.21.15.114:0/2660955849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ab8100bc0 0x7f9ab8196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:44.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.911+0000 7f9ab6ffd700 1 -- 172.21.15.114:0/2660955849 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab00095f0 con 0x7f9ab8101dc0 2024-02-19T19:00:44.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.911+0000 7f9ab6ffd700 1 --2- 172.21.15.114:0/2660955849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab8101dc0 0x7f9ab81968d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9aa000d8d0 tx=0x7f9aa000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:44.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.912+0000 7f9ab4ff9700 1 -- 172.21.15.114:0/2660955849 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9aa0009940 con 0x7f9ab8101dc0 2024-02-19T19:00:44.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.912+0000 7f9ab4ff9700 1 -- 172.21.15.114:0/2660955849 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9aa0010460 con 0x7f9ab8101dc0 2024-02-19T19:00:44.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.912+0000 7f9abdf80700 1 -- 172.21.15.114:0/2660955849 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab819bae0 con 0x7f9ab8101dc0 2024-02-19T19:00:44.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.912+0000 7f9ab4ff9700 1 -- 172.21.15.114:0/2660955849 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9aa000f5f0 con 0x7f9ab8101dc0 2024-02-19T19:00:44.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.912+0000 7f9abdf80700 1 -- 172.21.15.114:0/2660955849 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab819c030 con 0x7f9ab8101dc0 2024-02-19T19:00:44.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.913+0000 7f9abdf80700 1 -- 172.21.15.114:0/2660955849 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab8109720 con 0x7f9ab8101dc0 2024-02-19T19:00:44.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.913+0000 7f9ab4ff9700 1 -- 172.21.15.114:0/2660955849 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f9aa000f830 con 0x7f9ab8101dc0 2024-02-19T19:00:44.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.913+0000 7f9ab4ff9700 1 --2- 172.21.15.114:0/2660955849 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f9aa406c4e0 0x7f9aa406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:44.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.914+0000 7f9ab4ff9700 1 -- 172.21.15.114:0/2660955849 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f9aa008b860 con 0x7f9ab8101dc0 2024-02-19T19:00:44.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.914+0000 7f9ab77fe700 1 --2- 172.21.15.114:0/2660955849 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f9aa406c4e0 0x7f9aa406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:44.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.914+0000 7f9ab77fe700 1 --2- 172.21.15.114:0/2660955849 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f9aa406c4e0 0x7f9aa406e9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0006010 tx=0x7f9ab000b580 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:44.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:44.918+0000 7f9ab4ff9700 1 -- 172.21.15.114:0/2660955849 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9aa0059a70 con 0x7f9ab8101dc0 2024-02-19T19:00:45.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:45.119+0000 7f9abdf80700 1 -- 172.21.15.114:0/2660955849 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f9ab819c640 con 0x7f9ab8101dc0 2024-02-19T19:00:45.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:45.119+0000 7f9ab4ff9700 1 -- 172.21.15.114:0/2660955849 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v22)=0 v22) v1 ==== 155+0+0 (secure 0 0 0) 0x7f9aa0059600 con 0x7f9ab8101dc0 2024-02-19T19:00:45.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:45.122+0000 7f9abdf80700 1 -- 172.21.15.114:0/2660955849 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f9aa406c4e0 msgr2=0x7f9aa406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:45.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:45.122+0000 7f9abdf80700 1 --2- 172.21.15.114:0/2660955849 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f9aa406c4e0 0x7f9aa406e9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0006010 tx=0x7f9ab000b580 comp rx=0 tx=0).stop 2024-02-19T19:00:45.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:45.122+0000 7f9abdf80700 1 -- 172.21.15.114:0/2660955849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab8101dc0 msgr2=0x7f9ab81968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:45.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:45.122+0000 7f9abdf80700 1 --2- 172.21.15.114:0/2660955849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab8101dc0 0x7f9ab81968d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9aa000d8d0 tx=0x7f9aa000dc90 comp rx=0 tx=0).stop 2024-02-19T19:00:45.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:45.122+0000 7f9abdf80700 1 -- 172.21.15.114:0/2660955849 shutdown_connections 2024-02-19T19:00:45.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:45.122+0000 7f9abdf80700 1 --2- 172.21.15.114:0/2660955849 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f9aa406c4e0 0x7f9aa406e9a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:45.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:45.122+0000 7f9abdf80700 1 --2- 172.21.15.114:0/2660955849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ab8100bc0 0x7f9ab8196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:45.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:45.122+0000 7f9abdf80700 1 --2- 172.21.15.114:0/2660955849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab8101dc0 0x7f9ab81968d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:45.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:45.122+0000 7f9abdf80700 1 -- 172.21.15.114:0/2660955849 >> 172.21.15.114:0/2660955849 conn(0x7f9ab80fc140 msgr2=0x7f9ab8104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:45.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:45.122+0000 7f9abdf80700 1 -- 172.21.15.114:0/2660955849 shutdown_connections 2024-02-19T19:00:45.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:45.122+0000 7f9abdf80700 1 -- 172.21.15.114:0/2660955849 wait complete. 2024-02-19T19:00:45.803 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:45 smithi114 ceph-mon[29441]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-02-19T19:00:45.825 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-19T19:00:46.035 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T19:00:46.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:45 smithi174 ceph-mon[41061]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-02-19T19:00:47.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.828+0000 7fe17bdc2700 1 -- 172.21.15.114:0/1774852560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe174100cb0 msgr2=0x7fe174101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.828+0000 7fe17bdc2700 1 --2- 172.21.15.114:0/1774852560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe174100cb0 0x7fe174101110 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe164009910 tx=0x7fe164009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.828+0000 7fe17bdc2700 1 -- 172.21.15.114:0/1774852560 shutdown_connections 2024-02-19T19:00:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.828+0000 7fe17bdc2700 1 --2- 172.21.15.114:0/1774852560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe174100cb0 0x7fe174101110 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.828+0000 7fe17bdc2700 1 --2- 172.21.15.114:0/1774852560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1740ffab0 0x7fe1740ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.828+0000 7fe17bdc2700 1 -- 172.21.15.114:0/1774852560 >> 172.21.15.114:0/1774852560 conn(0x7fe1740fb050 msgr2=0x7fe1740fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.828+0000 7fe17bdc2700 1 -- 172.21.15.114:0/1774852560 shutdown_connections 2024-02-19T19:00:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.828+0000 7fe17bdc2700 1 -- 172.21.15.114:0/1774852560 wait complete. 2024-02-19T19:00:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.828+0000 7fe17bdc2700 1 Processor -- start 2024-02-19T19:00:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.828+0000 7fe17bdc2700 1 -- start start 2024-02-19T19:00:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.828+0000 7fe17bdc2700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1740ffab0 0x7fe174196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.828+0000 7fe17bdc2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe174100cb0 0x7fe174196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:47.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.828+0000 7fe17bdc2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe174196ea0 con 0x7fe174100cb0 2024-02-19T19:00:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.828+0000 7fe17bdc2700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe174196fe0 con 0x7fe1740ffab0 2024-02-19T19:00:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.829+0000 7fe17935d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe174100cb0 0x7fe174196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.829+0000 7fe17935d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe174100cb0 0x7fe174196880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49834/0 (socket says 172.21.15.114:49834) 2024-02-19T19:00:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.829+0000 7fe179b5e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1740ffab0 0x7fe174196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.829+0000 7fe17935d700 1 -- 172.21.15.114:0/2010622154 learned_addr learned my addr 172.21.15.114:0/2010622154 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.829+0000 7fe17935d700 1 -- 172.21.15.114:0/2010622154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1740ffab0 msgr2=0x7fe174196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.829+0000 7fe17935d700 1 --2- 172.21.15.114:0/2010622154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1740ffab0 0x7fe174196340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.829+0000 7fe17935d700 1 -- 172.21.15.114:0/2010622154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1640095f0 con 0x7fe174100cb0 2024-02-19T19:00:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.829+0000 7fe179b5e700 1 --2- 172.21.15.114:0/2010622154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1740ffab0 0x7fe174196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:00:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.829+0000 7fe17935d700 1 --2- 172.21.15.114:0/2010622154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe174100cb0 0x7fe174196880 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe164000c00 tx=0x7fe16400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.829+0000 7fe16affd700 1 -- 172.21.15.114:0/2010622154 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe16401d070 con 0x7fe174100cb0 2024-02-19T19:00:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.829+0000 7fe16affd700 1 -- 172.21.15.114:0/2010622154 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe164005dc0 con 0x7fe174100cb0 2024-02-19T19:00:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.829+0000 7fe16affd700 1 -- 172.21.15.114:0/2010622154 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1640226a0 con 0x7fe174100cb0 2024-02-19T19:00:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.829+0000 7fe17bdc2700 1 -- 172.21.15.114:0/2010622154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe17419ba30 con 0x7fe174100cb0 2024-02-19T19:00:47.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.829+0000 7fe17bdc2700 1 -- 172.21.15.114:0/2010622154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe17419bf20 con 0x7fe174100cb0 2024-02-19T19:00:47.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.830+0000 7fe17bdc2700 1 -- 172.21.15.114:0/2010622154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe174108610 con 0x7fe174100cb0 2024-02-19T19:00:47.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.831+0000 7fe16affd700 1 -- 172.21.15.114:0/2010622154 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fe1640038f0 con 0x7fe174100cb0 2024-02-19T19:00:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.831+0000 7fe16affd700 1 --2- 172.21.15.114:0/2010622154 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe16006c4e0 0x7fe16006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.831+0000 7fe16affd700 1 -- 172.21.15.114:0/2010622154 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fe16408cdc0 con 0x7fe174100cb0 2024-02-19T19:00:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.831+0000 7fe179b5e700 1 --2- 172.21.15.114:0/2010622154 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe16006c4e0 0x7fe16006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.832+0000 7fe179b5e700 1 --2- 172.21.15.114:0/2010622154 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe16006c4e0 0x7fe16006e9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe174100b10 tx=0x7fe170009260 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:47.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:47.836+0000 7fe16affd700 1 -- 172.21.15.114:0/2010622154 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe16405afd0 con 0x7fe174100cb0 2024-02-19T19:00:48.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:47 smithi114 ceph-mon[29441]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-19T19:00:48.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:47 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:48.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:48.028+0000 7fe17bdc2700 1 -- 172.21.15.114:0/2010622154 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7fe17419c590 con 0x7fe174100cb0 2024-02-19T19:00:48.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:48.028+0000 7fe16affd700 1 -- 172.21.15.114:0/2010622154 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v22)=0 v22) v1 ==== 163+0+0 (secure 0 0 0) 0x7fe164003680 con 0x7fe174100cb0 2024-02-19T19:00:48.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:48.030+0000 7fe17bdc2700 1 -- 172.21.15.114:0/2010622154 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe16006c4e0 msgr2=0x7fe16006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:48.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:48.030+0000 7fe17bdc2700 1 --2- 172.21.15.114:0/2010622154 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe16006c4e0 0x7fe16006e9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe174100b10 tx=0x7fe170009260 comp rx=0 tx=0).stop 2024-02-19T19:00:48.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:48.030+0000 7fe17bdc2700 1 -- 172.21.15.114:0/2010622154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe174100cb0 msgr2=0x7fe174196880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:48.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:48.030+0000 7fe17bdc2700 1 --2- 172.21.15.114:0/2010622154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe174100cb0 0x7fe174196880 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe164000c00 tx=0x7fe16400f690 comp rx=0 tx=0).stop 2024-02-19T19:00:48.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:48.030+0000 7fe17bdc2700 1 -- 172.21.15.114:0/2010622154 shutdown_connections 2024-02-19T19:00:48.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:48.030+0000 7fe17bdc2700 1 --2- 172.21.15.114:0/2010622154 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe16006c4e0 0x7fe16006e9a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:48.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:48.030+0000 7fe17bdc2700 1 --2- 172.21.15.114:0/2010622154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1740ffab0 0x7fe174196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:48.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:48.030+0000 7fe17bdc2700 1 --2- 172.21.15.114:0/2010622154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe174100cb0 0x7fe174196880 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:48.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:48.030+0000 7fe17bdc2700 1 -- 172.21.15.114:0/2010622154 >> 172.21.15.114:0/2010622154 conn(0x7fe1740fb050 msgr2=0x7fe174068ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:48.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:48.030+0000 7fe17bdc2700 1 -- 172.21.15.114:0/2010622154 shutdown_connections 2024-02-19T19:00:48.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:48.030+0000 7fe17bdc2700 1 -- 172.21.15.114:0/2010622154 wait complete. 2024-02-19T19:00:48.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:47 smithi174 ceph-mon[41061]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-19T19:00:48.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:47 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:48.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-19T19:00:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T19:00:50.245 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:49 smithi114 ceph-mon[29441]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-02-19T19:00:50.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:49 smithi174 ceph-mon[41061]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-02-19T19:00:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.795+0000 7fcc51869700 1 -- 172.21.15.114:0/2931592478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc4c101d80 msgr2=0x7fcc4c1021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.795+0000 7fcc51869700 1 --2- 172.21.15.114:0/2931592478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc4c101d80 0x7fcc4c1021e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fcc40009910 tx=0x7fcc40009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.795+0000 7fcc51869700 1 -- 172.21.15.114:0/2931592478 shutdown_connections 2024-02-19T19:00:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.795+0000 7fcc51869700 1 --2- 172.21.15.114:0/2931592478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc4c101d80 0x7fcc4c1021e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.795+0000 7fcc51869700 1 --2- 172.21.15.114:0/2931592478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc4c100b80 0x7fcc4c100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:50.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.795+0000 7fcc51869700 1 -- 172.21.15.114:0/2931592478 >> 172.21.15.114:0/2931592478 conn(0x7fcc4c0fc0e0 msgr2=0x7fcc4c0fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:50.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.795+0000 7fcc51869700 1 -- 172.21.15.114:0/2931592478 shutdown_connections 2024-02-19T19:00:50.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.795+0000 7fcc51869700 1 -- 172.21.15.114:0/2931592478 wait complete. 2024-02-19T19:00:50.815 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.796+0000 7fcc51869700 1 Processor -- start 2024-02-19T19:00:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.796+0000 7fcc51869700 1 -- start start 2024-02-19T19:00:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.796+0000 7fcc51869700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc4c100b80 0x7fcc4c196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.796+0000 7fcc51869700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc4c101d80 0x7fcc4c196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.796+0000 7fcc51869700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c196e60 con 0x7fcc4c101d80 2024-02-19T19:00:50.816 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.796+0000 7fcc51869700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c196fa0 con 0x7fcc4c100b80 2024-02-19T19:00:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.796+0000 7fcc4a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc4c101d80 0x7fcc4c196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.796+0000 7fcc4affd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc4c100b80 0x7fcc4c196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.796+0000 7fcc4a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc4c101d80 0x7fcc4c196840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49858/0 (socket says 172.21.15.114:49858) 2024-02-19T19:00:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.796+0000 7fcc4a7fc700 1 -- 172.21.15.114:0/1324803904 learned_addr learned my addr 172.21.15.114:0/1324803904 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.797+0000 7fcc4affd700 1 -- 172.21.15.114:0/1324803904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc4c101d80 msgr2=0x7fcc4c196840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.797+0000 7fcc4affd700 1 --2- 172.21.15.114:0/1324803904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc4c101d80 0x7fcc4c196840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.797+0000 7fcc4affd700 1 -- 172.21.15.114:0/1324803904 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc400095f0 con 0x7fcc4c100b80 2024-02-19T19:00:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.797+0000 7fcc4a7fc700 1 --2- 172.21.15.114:0/1324803904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc4c101d80 0x7fcc4c196840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:00:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.797+0000 7fcc4affd700 1 --2- 172.21.15.114:0/1324803904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc4c100b80 0x7fcc4c196300 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c00d6c0 tx=0x7fcc3c00d9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.797+0000 7fcc50867700 1 -- 172.21.15.114:0/1324803904 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc3c0041f0 con 0x7fcc4c100b80 2024-02-19T19:00:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.797+0000 7fcc50867700 1 -- 172.21.15.114:0/1324803904 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcc3c004350 con 0x7fcc4c100b80 2024-02-19T19:00:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.797+0000 7fcc51869700 1 -- 172.21.15.114:0/1324803904 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4c19ba50 con 0x7fcc4c100b80 2024-02-19T19:00:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.797+0000 7fcc50867700 1 -- 172.21.15.114:0/1324803904 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc3c003d20 con 0x7fcc4c100b80 2024-02-19T19:00:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.797+0000 7fcc51869700 1 -- 172.21.15.114:0/1324803904 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4c19bfa0 con 0x7fcc4c100b80 2024-02-19T19:00:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.798+0000 7fcc51869700 1 -- 172.21.15.114:0/1324803904 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc4c1096e0 con 0x7fcc4c100b80 2024-02-19T19:00:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.799+0000 7fcc50867700 1 -- 172.21.15.114:0/1324803904 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fcc3c00f690 con 0x7fcc4c100b80 2024-02-19T19:00:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.799+0000 7fcc50867700 1 --2- 172.21.15.114:0/1324803904 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fcc3406c4e0 0x7fcc3406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.799+0000 7fcc50867700 1 -- 172.21.15.114:0/1324803904 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fcc3c08b390 con 0x7fcc4c100b80 2024-02-19T19:00:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.799+0000 7fcc4a7fc700 1 --2- 172.21.15.114:0/1324803904 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fcc3406c4e0 0x7fcc3406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:50.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.799+0000 7fcc4a7fc700 1 --2- 172.21.15.114:0/1324803904 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fcc3406c4e0 0x7fcc3406e9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcc40006010 tx=0x7fcc4000b540 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:50.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.804+0000 7fcc50867700 1 -- 172.21.15.114:0/1324803904 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fcc3c059540 con 0x7fcc4c100b80 2024-02-19T19:00:51.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.995+0000 7fcc51869700 1 -- 172.21.15.114:0/1324803904 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fcc4c1098a0 con 0x7fcc4c100b80 2024-02-19T19:00:51.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.996+0000 7fcc50867700 1 -- 172.21.15.114:0/1324803904 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v22)=0 v22) v1 ==== 135+0+0 (secure 0 0 0) 0x7fcc3c003e80 con 0x7fcc4c100b80 2024-02-19T19:00:51.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.998+0000 7fcc51869700 1 -- 172.21.15.114:0/1324803904 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fcc3406c4e0 msgr2=0x7fcc3406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:51.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.998+0000 7fcc51869700 1 --2- 172.21.15.114:0/1324803904 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fcc3406c4e0 0x7fcc3406e9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcc40006010 tx=0x7fcc4000b540 comp rx=0 tx=0).stop 2024-02-19T19:00:51.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.998+0000 7fcc51869700 1 -- 172.21.15.114:0/1324803904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc4c100b80 msgr2=0x7fcc4c196300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:51.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.998+0000 7fcc51869700 1 --2- 172.21.15.114:0/1324803904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc4c100b80 0x7fcc4c196300 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c00d6c0 tx=0x7fcc3c00d9d0 comp rx=0 tx=0).stop 2024-02-19T19:00:51.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.998+0000 7fcc51869700 1 -- 172.21.15.114:0/1324803904 shutdown_connections 2024-02-19T19:00:51.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.998+0000 7fcc51869700 1 --2- 172.21.15.114:0/1324803904 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fcc3406c4e0 0x7fcc3406e9a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:51.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.998+0000 7fcc51869700 1 --2- 172.21.15.114:0/1324803904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc4c100b80 0x7fcc4c196300 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:51.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.998+0000 7fcc51869700 1 --2- 172.21.15.114:0/1324803904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc4c101d80 0x7fcc4c196840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:51.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.998+0000 7fcc51869700 1 -- 172.21.15.114:0/1324803904 >> 172.21.15.114:0/1324803904 conn(0x7fcc4c0fc0e0 msgr2=0x7fcc4c104fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:51.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.998+0000 7fcc51869700 1 -- 172.21.15.114:0/1324803904 shutdown_connections 2024-02-19T19:00:51.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:50.998+0000 7fcc51869700 1 -- 172.21.15.114:0/1324803904 wait complete. 2024-02-19T19:00:51.791 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-19T19:00:51.999 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T19:00:52.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:51 smithi114 ceph-mon[29441]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 938 B/s wr, 4 op/s 2024-02-19T19:00:52.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:51 smithi174 ceph-mon[41061]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 938 B/s wr, 4 op/s 2024-02-19T19:00:53.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.709+0000 7f36d8763700 1 -- 172.21.15.114:0/1884720657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d0100cb0 msgr2=0x7f36d0101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:53.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.709+0000 7f36d8763700 1 --2- 172.21.15.114:0/1884720657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d0100cb0 0x7f36d0101110 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f36c0009910 tx=0x7f36c0009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:53.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.709+0000 7f36d8763700 1 -- 172.21.15.114:0/1884720657 shutdown_connections 2024-02-19T19:00:53.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.709+0000 7f36d8763700 1 --2- 172.21.15.114:0/1884720657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d0100cb0 0x7f36d0101110 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.709+0000 7f36d8763700 1 --2- 172.21.15.114:0/1884720657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36d00ffab0 0x7f36d00ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.709+0000 7f36d8763700 1 -- 172.21.15.114:0/1884720657 >> 172.21.15.114:0/1884720657 conn(0x7f36d00fb030 msgr2=0x7f36d00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.709+0000 7f36d8763700 1 -- 172.21.15.114:0/1884720657 shutdown_connections 2024-02-19T19:00:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.709+0000 7f36d8763700 1 -- 172.21.15.114:0/1884720657 wait complete. 2024-02-19T19:00:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.710+0000 7f36d8763700 1 Processor -- start 2024-02-19T19:00:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.710+0000 7f36d8763700 1 -- start start 2024-02-19T19:00:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.710+0000 7f36d8763700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36d00ffab0 0x7f36d01963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.710+0000 7f36d8763700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d0100cb0 0x7f36d0196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.710+0000 7f36d8763700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d0196f40 con 0x7f36d0100cb0 2024-02-19T19:00:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.710+0000 7f36d8763700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d0197080 con 0x7f36d00ffab0 2024-02-19T19:00:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.710+0000 7f36d5cfe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d0100cb0 0x7f36d0196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.710+0000 7f36d5cfe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d0100cb0 0x7f36d0196920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49880/0 (socket says 172.21.15.114:49880) 2024-02-19T19:00:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.710+0000 7f36d5cfe700 1 -- 172.21.15.114:0/1506771593 learned_addr learned my addr 172.21.15.114:0/1506771593 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.710+0000 7f36d5cfe700 1 -- 172.21.15.114:0/1506771593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36d00ffab0 msgr2=0x7f36d01963e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.710+0000 7f36d64ff700 1 --2- 172.21.15.114:0/1506771593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36d00ffab0 0x7f36d01963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:53.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.710+0000 7f36d5cfe700 1 --2- 172.21.15.114:0/1506771593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36d00ffab0 0x7f36d01963e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.710+0000 7f36d5cfe700 1 -- 172.21.15.114:0/1506771593 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c00095f0 con 0x7f36d0100cb0 2024-02-19T19:00:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.710+0000 7f36d64ff700 1 --2- 172.21.15.114:0/1506771593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36d00ffab0 0x7f36d01963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:00:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.710+0000 7f36d5cfe700 1 --2- 172.21.15.114:0/1506771593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d0100cb0 0x7f36d0196920 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f36c0006010 tx=0x7f36c000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.711+0000 7f36c77fe700 1 -- 172.21.15.114:0/1506771593 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36c001d070 con 0x7f36d0100cb0 2024-02-19T19:00:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.711+0000 7f36c77fe700 1 -- 172.21.15.114:0/1506771593 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f36c000f460 con 0x7f36d0100cb0 2024-02-19T19:00:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.711+0000 7f36c77fe700 1 -- 172.21.15.114:0/1506771593 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36c0021640 con 0x7f36d0100cb0 2024-02-19T19:00:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.711+0000 7f36d8763700 1 -- 172.21.15.114:0/1506771593 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d019bad0 con 0x7f36d0100cb0 2024-02-19T19:00:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.711+0000 7f36d8763700 1 -- 172.21.15.114:0/1506771593 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d019bff0 con 0x7f36d0100cb0 2024-02-19T19:00:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.712+0000 7f36d8763700 1 -- 172.21.15.114:0/1506771593 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d01086b0 con 0x7f36d0100cb0 2024-02-19T19:00:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.712+0000 7f36c77fe700 1 -- 172.21.15.114:0/1506771593 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f36c000fab0 con 0x7f36d0100cb0 2024-02-19T19:00:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.713+0000 7f36c77fe700 1 --2- 172.21.15.114:0/1506771593 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f36bc06c4e0 0x7f36bc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.713+0000 7f36d64ff700 1 --2- 172.21.15.114:0/1506771593 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f36bc06c4e0 0x7f36bc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.713+0000 7f36c77fe700 1 -- 172.21.15.114:0/1506771593 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f36c008cd10 con 0x7f36d0100cb0 2024-02-19T19:00:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.713+0000 7f36d64ff700 1 --2- 172.21.15.114:0/1506771593 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f36bc06c4e0 0x7f36bc06e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f36d0100b10 tx=0x7f36cc008040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:53.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.717+0000 7f36c77fe700 1 -- 172.21.15.114:0/1506771593 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f36c005b040 con 0x7f36d0100cb0 2024-02-19T19:00:53.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.929+0000 7f36d8763700 1 -- 172.21.15.114:0/1506771593 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", "target": ["mon-mgr", ""]}) v1 -- 0x7f36d0068bc0 con 0x7f36bc06c4e0 2024-02-19T19:00:53.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.934+0000 7f36c77fe700 1 -- 172.21.15.114:0/1506771593 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f36d0068bc0 con 0x7f36bc06c4e0 2024-02-19T19:00:53.953 INFO:teuthology.orchestra.run.smithi114.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T19:00:53.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.936+0000 7f36d8763700 1 -- 172.21.15.114:0/1506771593 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f36bc06c4e0 msgr2=0x7f36bc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:53.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.936+0000 7f36d8763700 1 --2- 172.21.15.114:0/1506771593 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f36bc06c4e0 0x7f36bc06e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f36d0100b10 tx=0x7f36cc008040 comp rx=0 tx=0).stop 2024-02-19T19:00:53.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.936+0000 7f36d8763700 1 -- 172.21.15.114:0/1506771593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d0100cb0 msgr2=0x7f36d0196920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:53.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.936+0000 7f36d8763700 1 --2- 172.21.15.114:0/1506771593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d0100cb0 0x7f36d0196920 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f36c0006010 tx=0x7f36c000be20 comp rx=0 tx=0).stop 2024-02-19T19:00:53.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.936+0000 7f36d8763700 1 -- 172.21.15.114:0/1506771593 shutdown_connections 2024-02-19T19:00:53.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.936+0000 7f36d8763700 1 --2- 172.21.15.114:0/1506771593 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f36bc06c4e0 0x7f36bc06e9a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:53.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.936+0000 7f36d8763700 1 --2- 172.21.15.114:0/1506771593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36d00ffab0 0x7f36d01963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:53.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.936+0000 7f36d8763700 1 --2- 172.21.15.114:0/1506771593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36d0100cb0 0x7f36d0196920 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:53.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.936+0000 7f36d8763700 1 -- 172.21.15.114:0/1506771593 >> 172.21.15.114:0/1506771593 conn(0x7f36d00fb030 msgr2=0x7f36d0068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:53.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.936+0000 7f36d8763700 1 -- 172.21.15.114:0/1506771593 shutdown_connections 2024-02-19T19:00:53.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:53.936+0000 7f36d8763700 1 -- 172.21.15.114:0/1506771593 wait complete. 2024-02-19T19:00:54.270 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:53 smithi174 ceph-mon[41061]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 938 B/s wr, 4 op/s 2024-02-19T19:00:54.270 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:53 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:54.270 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:53 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:54.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:53 smithi114 ceph-mon[29441]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 938 B/s wr, 4 op/s 2024-02-19T19:00:54.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:53 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:54.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:53 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:54.732 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-19T19:00:54.733 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi114.front.sepia.ceph.com 2024-02-19T19:00:54.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-02-19T19:00:54.956 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T19:00:55.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:54 smithi114 ceph-mon[29441]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:55.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:54 smithi114 ceph-mon[29441]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T19:00:55.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:54 smithi114 ceph-mon[29441]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 5 op/s 2024-02-19T19:00:55.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:54 smithi174 ceph-mon[41061]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:55.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:54 smithi174 ceph-mon[41061]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T19:00:55.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:54 smithi174 ceph-mon[41061]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 5 op/s 2024-02-19T19:00:56.196 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:55 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:56.196 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:55 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:56.196 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:55 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:56.196 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:55 smithi114 ceph-mon[29441]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T19:00:56.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:55 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:56.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:55 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:56.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:55 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:00:56.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:55 smithi174 ceph-mon[41061]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T19:00:56.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.736+0000 7f100a09c700 1 -- 172.21.15.114:0/3153322594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10040ffac0 msgr2=0x7f10040ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:56.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.736+0000 7f100a09c700 1 --2- 172.21.15.114:0/3153322594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10040ffac0 0x7f10040ffee0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0ff400b0a0 tx=0x7f0ff400b3b0 comp rx=0 tx=0).stop 2024-02-19T19:00:56.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.736+0000 7f100a09c700 1 -- 172.21.15.114:0/3153322594 shutdown_connections 2024-02-19T19:00:56.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.736+0000 7f100a09c700 1 --2- 172.21.15.114:0/3153322594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1004100cc0 0x7f1004101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:56.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.736+0000 7f100a09c700 1 --2- 172.21.15.114:0/3153322594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10040ffac0 0x7f10040ffee0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:56.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.736+0000 7f100a09c700 1 -- 172.21.15.114:0/3153322594 >> 172.21.15.114:0/3153322594 conn(0x7f10040fb060 msgr2=0x7f10040fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:56.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.736+0000 7f100a09c700 1 -- 172.21.15.114:0/3153322594 shutdown_connections 2024-02-19T19:00:56.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.736+0000 7f100a09c700 1 -- 172.21.15.114:0/3153322594 wait complete. 2024-02-19T19:00:56.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.737+0000 7f100a09c700 1 Processor -- start 2024-02-19T19:00:56.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.737+0000 7f100a09c700 1 -- start start 2024-02-19T19:00:56.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.737+0000 7f100a09c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10040ffac0 0x7f10041964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:56.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.737+0000 7f100a09c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1004100cc0 0x7f10041969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:56.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.737+0000 7f100a09c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1004197000 con 0x7f1004100cc0 2024-02-19T19:00:56.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.737+0000 7f100a09c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1004197140 con 0x7f10040ffac0 2024-02-19T19:00:56.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.737+0000 7f1002ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1004100cc0 0x7f10041969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:56.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.737+0000 7f1002ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1004100cc0 0x7f10041969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35222/0 (socket says 172.21.15.114:35222) 2024-02-19T19:00:56.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.737+0000 7f1002ffd700 1 -- 172.21.15.114:0/1147964870 learned_addr learned my addr 172.21.15.114:0/1147964870 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:56.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.737+0000 7f1002ffd700 1 -- 172.21.15.114:0/1147964870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10040ffac0 msgr2=0x7f10041964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.737+0000 7f1002ffd700 1 --2- 172.21.15.114:0/1147964870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10040ffac0 0x7f10041964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.737+0000 7f1002ffd700 1 -- 172.21.15.114:0/1147964870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff4009d00 con 0x7f1004100cc0 2024-02-19T19:00:56.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.738+0000 7f1002ffd700 1 --2- 172.21.15.114:0/1147964870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1004100cc0 0x7f10041969e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0ff800b4b0 tx=0x7f0ff800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.738+0000 7f1000ff9700 1 -- 172.21.15.114:0/1147964870 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ff8011800 con 0x7f1004100cc0 2024-02-19T19:00:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.738+0000 7f1000ff9700 1 -- 172.21.15.114:0/1147964870 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0ff8011e10 con 0x7f1004100cc0 2024-02-19T19:00:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.738+0000 7f100a09c700 1 -- 172.21.15.114:0/1147964870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f100419bbf0 con 0x7f1004100cc0 2024-02-19T19:00:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.738+0000 7f1000ff9700 1 -- 172.21.15.114:0/1147964870 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ff800f2e0 con 0x7f1004100cc0 2024-02-19T19:00:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.738+0000 7f100a09c700 1 -- 172.21.15.114:0/1147964870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f100419c1c0 con 0x7f1004100cc0 2024-02-19T19:00:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.739+0000 7f100a09c700 1 -- 172.21.15.114:0/1147964870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10041086a0 con 0x7f1004100cc0 2024-02-19T19:00:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.739+0000 7f1000ff9700 1 -- 172.21.15.114:0/1147964870 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f0ff8010380 con 0x7f1004100cc0 2024-02-19T19:00:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.740+0000 7f1000ff9700 1 --2- 172.21.15.114:0/1147964870 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0fec06c4e0 0x7f0fec06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:56.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.740+0000 7f10037fe700 1 --2- 172.21.15.114:0/1147964870 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0fec06c4e0 0x7f0fec06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:56.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.740+0000 7f1000ff9700 1 -- 172.21.15.114:0/1147964870 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f0ff808b3f0 con 0x7f1004100cc0 2024-02-19T19:00:56.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.740+0000 7f10037fe700 1 --2- 172.21.15.114:0/1147964870 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0fec06c4e0 0x7f0fec06e9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0ff40077b0 tx=0x7f0ff4006000 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:56.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.745+0000 7f1000ff9700 1 -- 172.21.15.114:0/1147964870 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0ff8059680 con 0x7f1004100cc0 2024-02-19T19:00:56.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.976+0000 7f100a09c700 1 -- 172.21.15.114:0/1147964870 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f10040694c0 con 0x7f0fec06c4e0 2024-02-19T19:00:56.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.977+0000 7f1000ff9700 1 -- 172.21.15.114:0/1147964870 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f10040694c0 con 0x7f0fec06c4e0 2024-02-19T19:00:56.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.979+0000 7f100a09c700 1 -- 172.21.15.114:0/1147964870 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0fec06c4e0 msgr2=0x7f0fec06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:56.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.979+0000 7f100a09c700 1 --2- 172.21.15.114:0/1147964870 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0fec06c4e0 0x7f0fec06e9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0ff40077b0 tx=0x7f0ff4006000 comp rx=0 tx=0).stop 2024-02-19T19:00:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.979+0000 7f100a09c700 1 -- 172.21.15.114:0/1147964870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1004100cc0 msgr2=0x7f10041969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.979+0000 7f100a09c700 1 --2- 172.21.15.114:0/1147964870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1004100cc0 0x7f10041969e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0ff800b4b0 tx=0x7f0ff800b870 comp rx=0 tx=0).stop 2024-02-19T19:00:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.979+0000 7f100a09c700 1 -- 172.21.15.114:0/1147964870 shutdown_connections 2024-02-19T19:00:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.979+0000 7f100a09c700 1 --2- 172.21.15.114:0/1147964870 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f0fec06c4e0 0x7f0fec06e9a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.979+0000 7f100a09c700 1 --2- 172.21.15.114:0/1147964870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10040ffac0 0x7f10041964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.979+0000 7f100a09c700 1 --2- 172.21.15.114:0/1147964870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1004100cc0 0x7f10041969e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.979+0000 7f100a09c700 1 -- 172.21.15.114:0/1147964870 >> 172.21.15.114:0/1147964870 conn(0x7f10040fb060 msgr2=0x7f1004068db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:57.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.979+0000 7f100a09c700 1 -- 172.21.15.114:0/1147964870 shutdown_connections 2024-02-19T19:00:57.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:56.979+0000 7f100a09c700 1 -- 172.21.15.114:0/1147964870 wait complete. 2024-02-19T19:00:57.010 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-02-19T19:00:57.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.089+0000 7ff64f4fe700 1 -- 172.21.15.114:0/351530877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff648100ce0 msgr2=0x7ff648101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:57.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.089+0000 7ff64f4fe700 1 --2- 172.21.15.114:0/351530877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff648100ce0 0x7ff648101140 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff638009910 tx=0x7ff638009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:57.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.089+0000 7ff64f4fe700 1 -- 172.21.15.114:0/351530877 shutdown_connections 2024-02-19T19:00:57.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.089+0000 7ff64f4fe700 1 --2- 172.21.15.114:0/351530877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff648100ce0 0x7ff648101140 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.089+0000 7ff64f4fe700 1 --2- 172.21.15.114:0/351530877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6480ffae0 0x7ff6480fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.089+0000 7ff64f4fe700 1 -- 172.21.15.114:0/351530877 >> 172.21.15.114:0/351530877 conn(0x7ff6480fb080 msgr2=0x7ff6480fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:57.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.090+0000 7ff64f4fe700 1 -- 172.21.15.114:0/351530877 shutdown_connections 2024-02-19T19:00:57.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.090+0000 7ff64f4fe700 1 -- 172.21.15.114:0/351530877 wait complete. 2024-02-19T19:00:57.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.090+0000 7ff64f4fe700 1 Processor -- start 2024-02-19T19:00:57.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.090+0000 7ff64f4fe700 1 -- start start 2024-02-19T19:00:57.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.090+0000 7ff64f4fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6480ffae0 0x7ff648069930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:57.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.090+0000 7ff64f4fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff648100ce0 0x7ff64819b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:57.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.090+0000 7ff64f4fe700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64819b730 con 0x7ff6480ffae0 2024-02-19T19:00:57.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.090+0000 7ff64f4fe700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64819b870 con 0x7ff648100ce0 2024-02-19T19:00:57.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.090+0000 7ff64d29a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6480ffae0 0x7ff648069930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:57.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.090+0000 7ff64d29a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6480ffae0 0x7ff648069930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35238/0 (socket says 172.21.15.114:35238) 2024-02-19T19:00:57.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.090+0000 7ff64d29a700 1 -- 172.21.15.114:0/4030171148 learned_addr learned my addr 172.21.15.114:0/4030171148 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:57.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.091+0000 7ff64d29a700 1 -- 172.21.15.114:0/4030171148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff648100ce0 msgr2=0x7ff64819b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:57.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.091+0000 7ff64d29a700 1 --2- 172.21.15.114:0/4030171148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff648100ce0 0x7ff64819b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.091+0000 7ff64d29a700 1 -- 172.21.15.114:0/4030171148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6380095f0 con 0x7ff6480ffae0 2024-02-19T19:00:57.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.091+0000 7ff64d29a700 1 --2- 172.21.15.114:0/4030171148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6480ffae0 0x7ff648069930 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff64400b480 tx=0x7ff64400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:57.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.091+0000 7ff63e7fc700 1 -- 172.21.15.114:0/4030171148 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6440118c0 con 0x7ff6480ffae0 2024-02-19T19:00:57.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.091+0000 7ff64f4fe700 1 -- 172.21.15.114:0/4030171148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64819bb50 con 0x7ff6480ffae0 2024-02-19T19:00:57.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.091+0000 7ff63e7fc700 1 -- 172.21.15.114:0/4030171148 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff644010380 con 0x7ff6480ffae0 2024-02-19T19:00:57.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.091+0000 7ff63e7fc700 1 -- 172.21.15.114:0/4030171148 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff64400f4f0 con 0x7ff6480ffae0 2024-02-19T19:00:57.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.091+0000 7ff64f4fe700 1 -- 172.21.15.114:0/4030171148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64819c120 con 0x7ff6480ffae0 2024-02-19T19:00:57.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.092+0000 7ff64f4fe700 1 -- 172.21.15.114:0/4030171148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6481086e0 con 0x7ff6480ffae0 2024-02-19T19:00:57.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.093+0000 7ff63e7fc700 1 -- 172.21.15.114:0/4030171148 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ff6440104f0 con 0x7ff6480ffae0 2024-02-19T19:00:57.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.093+0000 7ff63e7fc700 1 --2- 172.21.15.114:0/4030171148 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff63406c4e0 0x7ff63406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:57.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.093+0000 7ff63e7fc700 1 -- 172.21.15.114:0/4030171148 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff64408b540 con 0x7ff6480ffae0 2024-02-19T19:00:57.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.093+0000 7ff64ca99700 1 --2- 172.21.15.114:0/4030171148 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff63406c4e0 0x7ff63406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:57.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.093+0000 7ff64ca99700 1 --2- 172.21.15.114:0/4030171148 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff63406c4e0 0x7ff63406e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff63800b5c0 tx=0x7ff638005fd0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:57.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.097+0000 7ff63e7fc700 1 -- 172.21.15.114:0/4030171148 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff644059750 con 0x7ff6480ffae0 2024-02-19T19:00:57.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:56 smithi114 ceph-mon[29441]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:00:57.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.342+0000 7ff64f4fe700 1 -- 172.21.15.114:0/4030171148 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff648105630 con 0x7ff63406c4e0 2024-02-19T19:00:57.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.344+0000 7ff63e7fc700 1 -- 172.21.15.114:0/4030171148 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7ff648105630 con 0x7ff63406c4e0 2024-02-19T19:00:57.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.345+0000 7ff64f4fe700 1 -- 172.21.15.114:0/4030171148 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff63406c4e0 msgr2=0x7ff63406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:57.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.345+0000 7ff64f4fe700 1 --2- 172.21.15.114:0/4030171148 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff63406c4e0 0x7ff63406e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff63800b5c0 tx=0x7ff638005fd0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.345+0000 7ff64f4fe700 1 -- 172.21.15.114:0/4030171148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6480ffae0 msgr2=0x7ff648069930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:57.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.345+0000 7ff64f4fe700 1 --2- 172.21.15.114:0/4030171148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6480ffae0 0x7ff648069930 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff64400b480 tx=0x7ff64400b840 comp rx=0 tx=0).stop 2024-02-19T19:00:57.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.346+0000 7ff64f4fe700 1 -- 172.21.15.114:0/4030171148 shutdown_connections 2024-02-19T19:00:57.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.346+0000 7ff64f4fe700 1 --2- 172.21.15.114:0/4030171148 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff63406c4e0 0x7ff63406e9a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.346+0000 7ff64f4fe700 1 --2- 172.21.15.114:0/4030171148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6480ffae0 0x7ff648069930 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.346+0000 7ff64f4fe700 1 --2- 172.21.15.114:0/4030171148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff648100ce0 0x7ff64819b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.346+0000 7ff64f4fe700 1 -- 172.21.15.114:0/4030171148 >> 172.21.15.114:0/4030171148 conn(0x7ff6480fb080 msgr2=0x7ff648103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:57.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.346+0000 7ff64f4fe700 1 -- 172.21.15.114:0/4030171148 shutdown_connections 2024-02-19T19:00:57.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.346+0000 7ff64f4fe700 1 -- 172.21.15.114:0/4030171148 wait complete. 2024-02-19T19:00:57.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:56 smithi174 ceph-mon[41061]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:00:57.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.457+0000 7f01d4845700 1 -- 172.21.15.114:0/3713351598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d00ffab0 msgr2=0x7f01d00ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:57.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.457+0000 7f01d4845700 1 --2- 172.21.15.114:0/3713351598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d00ffab0 0x7f01d00ffed0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f01c0009910 tx=0x7f01c0009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:57.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.457+0000 7f01d4845700 1 -- 172.21.15.114:0/3713351598 shutdown_connections 2024-02-19T19:00:57.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.457+0000 7f01d4845700 1 --2- 172.21.15.114:0/3713351598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d0100cb0 0x7f01d0101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.457+0000 7f01d4845700 1 --2- 172.21.15.114:0/3713351598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d00ffab0 0x7f01d00ffed0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.457+0000 7f01d4845700 1 -- 172.21.15.114:0/3713351598 >> 172.21.15.114:0/3713351598 conn(0x7f01d00fb030 msgr2=0x7f01d00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:57.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.457+0000 7f01d4845700 1 -- 172.21.15.114:0/3713351598 shutdown_connections 2024-02-19T19:00:57.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.457+0000 7f01d4845700 1 -- 172.21.15.114:0/3713351598 wait complete. 2024-02-19T19:00:57.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.458+0000 7f01d4845700 1 Processor -- start 2024-02-19T19:00:57.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.458+0000 7f01d4845700 1 -- start start 2024-02-19T19:00:57.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.458+0000 7f01d4845700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d00ffab0 0x7f01d0196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:57.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.458+0000 7f01d4845700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d0100cb0 0x7f01d0196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:57.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.458+0000 7f01d4845700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0196e80 con 0x7f01d00ffab0 2024-02-19T19:00:57.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.458+0000 7f01d4845700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0196fc0 con 0x7f01d0100cb0 2024-02-19T19:00:57.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.458+0000 7f01ce59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d00ffab0 0x7f01d0196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:57.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.458+0000 7f01ce59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d00ffab0 0x7f01d0196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35256/0 (socket says 172.21.15.114:35256) 2024-02-19T19:00:57.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.458+0000 7f01ce59c700 1 -- 172.21.15.114:0/2354929655 learned_addr learned my addr 172.21.15.114:0/2354929655 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:57.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.459+0000 7f01cdd9b700 1 --2- 172.21.15.114:0/2354929655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d0100cb0 0x7f01d0196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:57.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.459+0000 7f01ce59c700 1 -- 172.21.15.114:0/2354929655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d0100cb0 msgr2=0x7f01d0196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:57.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.459+0000 7f01ce59c700 1 --2- 172.21.15.114:0/2354929655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d0100cb0 0x7f01d0196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.459+0000 7f01ce59c700 1 -- 172.21.15.114:0/2354929655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c00095f0 con 0x7f01d00ffab0 2024-02-19T19:00:57.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.459+0000 7f01ce59c700 1 --2- 172.21.15.114:0/2354929655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d00ffab0 0x7f01d0196320 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f01c000bda0 tx=0x7f01c000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:57.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.459+0000 7f01bf7fe700 1 -- 172.21.15.114:0/2354929655 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01c001d070 con 0x7f01d00ffab0 2024-02-19T19:00:57.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.459+0000 7f01d4845700 1 -- 172.21.15.114:0/2354929655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d019ba10 con 0x7f01d00ffab0 2024-02-19T19:00:57.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.459+0000 7f01bf7fe700 1 -- 172.21.15.114:0/2354929655 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f01c0004c50 con 0x7f01d00ffab0 2024-02-19T19:00:57.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.459+0000 7f01bf7fe700 1 -- 172.21.15.114:0/2354929655 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01c000f690 con 0x7f01d00ffab0 2024-02-19T19:00:57.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.459+0000 7f01d4845700 1 -- 172.21.15.114:0/2354929655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d019bf00 con 0x7f01d00ffab0 2024-02-19T19:00:57.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.461+0000 7f01d4845700 1 -- 172.21.15.114:0/2354929655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d0108610 con 0x7f01d00ffab0 2024-02-19T19:00:57.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.461+0000 7f01bf7fe700 1 -- 172.21.15.114:0/2354929655 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f01c00229b0 con 0x7f01d00ffab0 2024-02-19T19:00:57.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.462+0000 7f01bf7fe700 1 --2- 172.21.15.114:0/2354929655 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f01b806c4e0 0x7f01b806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:57.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.462+0000 7f01bf7fe700 1 -- 172.21.15.114:0/2354929655 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f01c008cf70 con 0x7f01d00ffab0 2024-02-19T19:00:57.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.462+0000 7f01cdd9b700 1 --2- 172.21.15.114:0/2354929655 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f01b806c4e0 0x7f01b806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:57.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.462+0000 7f01cdd9b700 1 --2- 172.21.15.114:0/2354929655 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f01b806c4e0 0x7f01b806e9a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f01c4005fd0 tx=0x7f01c400a560 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:57.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.466+0000 7f01bf7fe700 1 -- 172.21.15.114:0/2354929655 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f01c00c00e0 con 0x7f01d00ffab0 2024-02-19T19:00:57.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.708+0000 7f01d4845700 1 -- 172.21.15.114:0/2354929655 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f01d0068ae0 con 0x7f01b806c4e0 2024-02-19T19:00:57.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.725+0000 7f01bf7fe700 1 -- 172.21.15.114:0/2354929655 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7f01d0068ae0 con 0x7f01b806c4e0 2024-02-19T19:00:57.744 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:00:57.744 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (4m) 27s ago 7m 26.5M - 0.25.0 c8568f914cd2 30444b676ca4 2024-02-19T19:00:57.744 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (8m) 27s ago 8m 7470k - 18.2.1 94eadd990e0a 4918cd87e312 2024-02-19T19:00:57.744 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi174 smithi174 running (2m) 34s ago 5m 7394k - 18.2.1 94eadd990e0a 516c34431b93 2024-02-19T19:00:57.744 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (7m) 27s ago 7m 6660k - 18.2.1 94eadd990e0a ecfcd98907ba 2024-02-19T19:00:57.744 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi174 smithi174 running (5m) 34s ago 5m 6660k - 18.2.1 94eadd990e0a de88985f5a07 2024-02-19T19:00:57.744 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (4m) 27s ago 6m 84.7M - 9.4.7 954c08fa6188 478bd533e827 2024-02-19T19:00:57.744 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tcwpod smithi114 running (45s) 27s ago 45s 15.1M - 18.2.1 94eadd990e0a 420175777cf4 2024-02-19T19:00:57.744 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.vpoeop smithi114 running (52s) 27s ago 52s 11.8M - 18.2.1 94eadd990e0a f81a79e53ead 2024-02-19T19:00:57.744 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.stnfnf smithi174 running (43s) 34s ago 42s 14.6M - 18.2.1 94eadd990e0a 3df35954a492 2024-02-19T19:00:57.744 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.znhuvd smithi174 running (50s) 34s ago 49s 15.1M - 18.2.1 94eadd990e0a 2d2267494ae1 2024-02-19T19:00:57.745 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.pcvtrk smithi114 *:9283,8765,8443 running (9m) 27s ago 9m 489M - 18.2.1 94eadd990e0a 9d81af39d736 2024-02-19T19:00:57.745 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi174.lydvnj smithi174 *:8443,9283,8765 running (5m) 34s ago 5m 443M - 18.2.1 94eadd990e0a 9b31a7d679b9 2024-02-19T19:00:57.745 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (9m) 27s ago 9m 46.8M 2048M 18.2.1 94eadd990e0a 41468f811fc0 2024-02-19T19:00:57.745 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi174 smithi174 running (5m) 34s ago 5m 39.9M 2048M 18.2.1 94eadd990e0a 70dd7bf69fd9 2024-02-19T19:00:57.745 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (7m) 27s ago 7m 14.2M - 1.5.0 0da6a335fe13 b6b258074cce 2024-02-19T19:00:57.745 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi174 smithi174 *:9100 running (5m) 34s ago 5m 15.2M - 1.5.0 0da6a335fe13 d0b4fdf67b22 2024-02-19T19:00:57.745 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (4m) 27s ago 4m 46.8M 3946M 18.2.1 94eadd990e0a 990afc982b6c 2024-02-19T19:00:57.745 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (4m) 27s ago 4m 47.2M 3946M 18.2.1 94eadd990e0a 932e3ad37db5 2024-02-19T19:00:57.745 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (3m) 27s ago 3m 48.7M 3946M 18.2.1 94eadd990e0a 12953988af86 2024-02-19T19:00:57.745 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi174 running (3m) 34s ago 3m 46.1M 4970M 18.2.1 94eadd990e0a 979f45cabbce 2024-02-19T19:00:57.745 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi174 running (2m) 34s ago 2m 46.4M 4970M 18.2.1 94eadd990e0a 8b8c3745b8c5 2024-02-19T19:00:57.745 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi174 running (2m) 34s ago 2m 48.6M 4970M 18.2.1 94eadd990e0a 5771a995ed75 2024-02-19T19:00:57.745 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (4m) 27s ago 6m 55.2M - 2.43.0 a07b618ecd1d 52087f218c92 2024-02-19T19:00:57.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.728+0000 7f01d4845700 1 -- 172.21.15.114:0/2354929655 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f01b806c4e0 msgr2=0x7f01b806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:57.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.728+0000 7f01d4845700 1 --2- 172.21.15.114:0/2354929655 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f01b806c4e0 0x7f01b806e9a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f01c4005fd0 tx=0x7f01c400a560 comp rx=0 tx=0).stop 2024-02-19T19:00:57.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.728+0000 7f01d4845700 1 -- 172.21.15.114:0/2354929655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d00ffab0 msgr2=0x7f01d0196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:57.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.728+0000 7f01d4845700 1 --2- 172.21.15.114:0/2354929655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d00ffab0 0x7f01d0196320 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f01c000bda0 tx=0x7f01c000be80 comp rx=0 tx=0).stop 2024-02-19T19:00:57.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.728+0000 7f01d4845700 1 -- 172.21.15.114:0/2354929655 shutdown_connections 2024-02-19T19:00:57.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.728+0000 7f01d4845700 1 --2- 172.21.15.114:0/2354929655 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f01b806c4e0 0x7f01b806e9a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.728+0000 7f01d4845700 1 --2- 172.21.15.114:0/2354929655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d00ffab0 0x7f01d0196320 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.728+0000 7f01d4845700 1 --2- 172.21.15.114:0/2354929655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d0100cb0 0x7f01d0196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.728+0000 7f01d4845700 1 -- 172.21.15.114:0/2354929655 >> 172.21.15.114:0/2354929655 conn(0x7f01d00fb030 msgr2=0x7f01d00683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:57.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.728+0000 7f01d4845700 1 -- 172.21.15.114:0/2354929655 shutdown_connections 2024-02-19T19:00:57.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.728+0000 7f01d4845700 1 -- 172.21.15.114:0/2354929655 wait complete. 2024-02-19T19:00:57.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.847+0000 7fe5aca82700 1 -- 172.21.15.114:0/1414404129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a8100bc0 msgr2=0x7fe5a8100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:57.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.848+0000 7fe5aca82700 1 --2- 172.21.15.114:0/1414404129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a8100bc0 0x7fe5a8100fe0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe598009910 tx=0x7fe598009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:57.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.848+0000 7fe5aca82700 1 -- 172.21.15.114:0/1414404129 shutdown_connections 2024-02-19T19:00:57.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.848+0000 7fe5aca82700 1 --2- 172.21.15.114:0/1414404129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5a8101dc0 0x7fe5a8102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.848+0000 7fe5aca82700 1 --2- 172.21.15.114:0/1414404129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a8100bc0 0x7fe5a8100fe0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.848+0000 7fe5aca82700 1 -- 172.21.15.114:0/1414404129 >> 172.21.15.114:0/1414404129 conn(0x7fe5a80fc140 msgr2=0x7fe5a80fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:57.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.848+0000 7fe5aca82700 1 -- 172.21.15.114:0/1414404129 shutdown_connections 2024-02-19T19:00:57.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.848+0000 7fe5aca82700 1 -- 172.21.15.114:0/1414404129 wait complete. 2024-02-19T19:00:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.848+0000 7fe5aca82700 1 Processor -- start 2024-02-19T19:00:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.848+0000 7fe5aca82700 1 -- start start 2024-02-19T19:00:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.849+0000 7fe5aca82700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a8100bc0 0x7fe5a8074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.849+0000 7fe5aca82700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5a8101dc0 0x7fe5a8073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.849+0000 7fe5aca82700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a80736c0 con 0x7fe5a8100bc0 2024-02-19T19:00:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.849+0000 7fe5aca82700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a8073800 con 0x7fe5a8101dc0 2024-02-19T19:00:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.849+0000 7fe5a659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a8100bc0 0x7fe5a8074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.849+0000 7fe5a659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a8100bc0 0x7fe5a8074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35282/0 (socket says 172.21.15.114:35282) 2024-02-19T19:00:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.849+0000 7fe5a659c700 1 -- 172.21.15.114:0/3006878386 learned_addr learned my addr 172.21.15.114:0/3006878386 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.849+0000 7fe5a5d9b700 1 --2- 172.21.15.114:0/3006878386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5a8101dc0 0x7fe5a8073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.849+0000 7fe5a5d9b700 1 -- 172.21.15.114:0/3006878386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a8100bc0 msgr2=0x7fe5a8074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:57.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.849+0000 7fe5a5d9b700 1 --2- 172.21.15.114:0/3006878386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a8100bc0 0x7fe5a8074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:57.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.849+0000 7fe5a5d9b700 1 -- 172.21.15.114:0/3006878386 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5980095f0 con 0x7fe5a8101dc0 2024-02-19T19:00:57.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.849+0000 7fe5a659c700 1 --2- 172.21.15.114:0/3006878386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a8100bc0 0x7fe5a8074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:00:57.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.849+0000 7fe5a5d9b700 1 --2- 172.21.15.114:0/3006878386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5a8101dc0 0x7fe5a8073180 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00e970 tx=0x7fe59c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:57.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.850+0000 7fe5977fe700 1 -- 172.21.15.114:0/3006878386 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe59c00c9d0 con 0x7fe5a8101dc0 2024-02-19T19:00:57.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.850+0000 7fe5977fe700 1 -- 172.21.15.114:0/3006878386 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe59c010430 con 0x7fe5a8101dc0 2024-02-19T19:00:57.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.850+0000 7fe5977fe700 1 -- 172.21.15.114:0/3006878386 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe59c018630 con 0x7fe5a8101dc0 2024-02-19T19:00:57.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.850+0000 7fe5aca82700 1 -- 172.21.15.114:0/3006878386 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a8073ae0 con 0x7fe5a8101dc0 2024-02-19T19:00:57.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.850+0000 7fe5aca82700 1 -- 172.21.15.114:0/3006878386 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a819e3f0 con 0x7fe5a8101dc0 2024-02-19T19:00:57.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.851+0000 7fe5aca82700 1 -- 172.21.15.114:0/3006878386 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a8109720 con 0x7fe5a8101dc0 2024-02-19T19:00:57.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.851+0000 7fe5977fe700 1 -- 172.21.15.114:0/3006878386 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fe59c0105a0 con 0x7fe5a8101dc0 2024-02-19T19:00:57.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.852+0000 7fe5977fe700 1 --2- 172.21.15.114:0/3006878386 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe59006c4e0 0x7fe59006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:57.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.852+0000 7fe5977fe700 1 -- 172.21.15.114:0/3006878386 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fe59c014070 con 0x7fe5a8101dc0 2024-02-19T19:00:57.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.852+0000 7fe5a659c700 1 --2- 172.21.15.114:0/3006878386 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe59006c4e0 0x7fe59006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:57.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.852+0000 7fe5a659c700 1 --2- 172.21.15.114:0/3006878386 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe59006c4e0 0x7fe59006e9a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe598009fd0 tx=0x7fe598005940 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:57.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:57.856+0000 7fe5977fe700 1 -- 172.21.15.114:0/3006878386 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe59c0520e0 con 0x7fe5a8101dc0 2024-02-19T19:00:58.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.171+0000 7fe5aca82700 1 -- 172.21.15.114:0/3006878386 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe5a8109910 con 0x7fe5a8101dc0 2024-02-19T19:00:58.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.171+0000 7fe5977fe700 1 -- 172.21.15.114:0/3006878386 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7fe59c059c50 con 0x7fe5a8101dc0 2024-02-19T19:00:58.190 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:00:58.190 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-02-19T19:00:58.190 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:00:58.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:00:58.190 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-02-19T19:00:58.190 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:00:58.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:00:58.191 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-02-19T19:00:58.191 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-02-19T19:00:58.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:00:58.191 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-02-19T19:00:58.191 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:00:58.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:00:58.191 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-02-19T19:00:58.191 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-02-19T19:00:58.191 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-02-19T19:00:58.191 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:00:58.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.173+0000 7fe5aca82700 1 -- 172.21.15.114:0/3006878386 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe59006c4e0 msgr2=0x7fe59006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:58.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.173+0000 7fe5aca82700 1 --2- 172.21.15.114:0/3006878386 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe59006c4e0 0x7fe59006e9a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe598009fd0 tx=0x7fe598005940 comp rx=0 tx=0).stop 2024-02-19T19:00:58.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.173+0000 7fe5aca82700 1 -- 172.21.15.114:0/3006878386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5a8101dc0 msgr2=0x7fe5a8073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:58.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.173+0000 7fe5aca82700 1 --2- 172.21.15.114:0/3006878386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5a8101dc0 0x7fe5a8073180 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00e970 tx=0x7fe59c00ec80 comp rx=0 tx=0).stop 2024-02-19T19:00:58.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.173+0000 7fe5aca82700 1 -- 172.21.15.114:0/3006878386 shutdown_connections 2024-02-19T19:00:58.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.173+0000 7fe5aca82700 1 --2- 172.21.15.114:0/3006878386 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fe59006c4e0 0x7fe59006e9a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:58.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.173+0000 7fe5aca82700 1 --2- 172.21.15.114:0/3006878386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5a8100bc0 0x7fe5a8074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:58.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.173+0000 7fe5aca82700 1 --2- 172.21.15.114:0/3006878386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5a8101dc0 0x7fe5a8073180 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:58.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.173+0000 7fe5aca82700 1 -- 172.21.15.114:0/3006878386 >> 172.21.15.114:0/3006878386 conn(0x7fe5a80fc140 msgr2=0x7fe5a8104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:58.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.173+0000 7fe5aca82700 1 -- 172.21.15.114:0/3006878386 shutdown_connections 2024-02-19T19:00:58.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.173+0000 7fe5aca82700 1 -- 172.21.15.114:0/3006878386 wait complete. 2024-02-19T19:00:58.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.286+0000 7f9114b31700 1 -- 172.21.15.114:0/4226710195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9110104010 msgr2=0x7f9110106400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:58.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.286+0000 7f9114b31700 1 --2- 172.21.15.114:0/4226710195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9110104010 0x7f9110106400 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9104009910 tx=0x7f9104009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:58.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.286+0000 7f9114b31700 1 -- 172.21.15.114:0/4226710195 shutdown_connections 2024-02-19T19:00:58.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.286+0000 7f9114b31700 1 --2- 172.21.15.114:0/4226710195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9110104010 0x7f9110106400 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:58.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.286+0000 7f9114b31700 1 --2- 172.21.15.114:0/4226710195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91101016e0 0x7f9110103ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:58.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.286+0000 7f9114b31700 1 -- 172.21.15.114:0/4226710195 >> 172.21.15.114:0/4226710195 conn(0x7f91100fb030 msgr2=0x7f91100fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:58.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.286+0000 7f9114b31700 1 -- 172.21.15.114:0/4226710195 shutdown_connections 2024-02-19T19:00:58.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.286+0000 7f9114b31700 1 -- 172.21.15.114:0/4226710195 wait complete. 2024-02-19T19:00:58.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.287+0000 7f9114b31700 1 Processor -- start 2024-02-19T19:00:58.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.287+0000 7f9114b31700 1 -- start start 2024-02-19T19:00:58.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.287+0000 7f9114b31700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91101016e0 0x7f91101963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:58.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.287+0000 7f9114b31700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9110104010 0x7f91101968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:58.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.287+0000 7f9114b31700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9110196f10 con 0x7f9110104010 2024-02-19T19:00:58.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.287+0000 7f9114b31700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9110197050 con 0x7f91101016e0 2024-02-19T19:00:58.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.287+0000 7f910dd9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9110104010 0x7f91101968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:58.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.287+0000 7f910dd9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9110104010 0x7f91101968f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35308/0 (socket says 172.21.15.114:35308) 2024-02-19T19:00:58.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.287+0000 7f910dd9b700 1 -- 172.21.15.114:0/3116408413 learned_addr learned my addr 172.21.15.114:0/3116408413 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:58.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.287+0000 7f910e59c700 1 --2- 172.21.15.114:0/3116408413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91101016e0 0x7f91101963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:58.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.287+0000 7f910dd9b700 1 -- 172.21.15.114:0/3116408413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91101016e0 msgr2=0x7f91101963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:58.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.287+0000 7f910dd9b700 1 --2- 172.21.15.114:0/3116408413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91101016e0 0x7f91101963b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:58.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.287+0000 7f910dd9b700 1 -- 172.21.15.114:0/3116408413 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91040095f0 con 0x7f9110104010 2024-02-19T19:00:58.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.287+0000 7f910e59c700 1 --2- 172.21.15.114:0/3116408413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91101016e0 0x7f91101963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:00:58.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.288+0000 7f910dd9b700 1 --2- 172.21.15.114:0/3116408413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9110104010 0x7f91101968f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f910400bd70 tx=0x7f910400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:58.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.288+0000 7f90ff7fe700 1 -- 172.21.15.114:0/3116408413 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f910401d070 con 0x7f9110104010 2024-02-19T19:00:58.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.288+0000 7f9114b31700 1 -- 172.21.15.114:0/3116408413 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f911019baa0 con 0x7f9110104010 2024-02-19T19:00:58.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.288+0000 7f90ff7fe700 1 -- 172.21.15.114:0/3116408413 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9104022470 con 0x7f9110104010 2024-02-19T19:00:58.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.288+0000 7f90ff7fe700 1 -- 172.21.15.114:0/3116408413 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f910400f670 con 0x7f9110104010 2024-02-19T19:00:58.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.288+0000 7f9114b31700 1 -- 172.21.15.114:0/3116408413 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f911019c010 con 0x7f9110104010 2024-02-19T19:00:58.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.289+0000 7f9114b31700 1 -- 172.21.15.114:0/3116408413 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91101905b0 con 0x7f9110104010 2024-02-19T19:00:58.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.289+0000 7f90ff7fe700 1 -- 172.21.15.114:0/3116408413 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f9104022a50 con 0x7f9110104010 2024-02-19T19:00:58.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.290+0000 7f90ff7fe700 1 --2- 172.21.15.114:0/3116408413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f90f806c4e0 0x7f90f806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:58.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.290+0000 7f90ff7fe700 1 -- 172.21.15.114:0/3116408413 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f910408cf30 con 0x7f9110104010 2024-02-19T19:00:58.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.290+0000 7f910e59c700 1 --2- 172.21.15.114:0/3116408413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f90f806c4e0 0x7f90f806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:58.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.290+0000 7f910e59c700 1 --2- 172.21.15.114:0/3116408413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f90f806c4e0 0x7f90f806e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f91000099d0 tx=0x7f9100008040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:58.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.297+0000 7f90ff7fe700 1 -- 172.21.15.114:0/3116408413 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f91040c00e0 con 0x7f9110104010 2024-02-19T19:00:58.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.561+0000 7f9114b31700 1 -- 172.21.15.114:0/3116408413 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f911002cc70 con 0x7f9110104010 2024-02-19T19:00:58.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.562+0000 7f90ff7fe700 1 -- 172.21.15.114:0/3116408413 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1850 (secure 0 0 0) 0x7f9104027750 con 0x7f9110104010 2024-02-19T19:00:58.581 INFO:teuthology.orchestra.run.smithi114.stdout:e18 2024-02-19T19:00:58.581 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:00:58.581 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:00:58.581 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-02-19T19:00:58.581 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:00:58.581 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:00:58.581 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-02-19T19:00:58.581 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 18 2024-02-19T19:00:58.581 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:00:58.582 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:00:58.582 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-02-19T19:00:21.203922+0000 2024-02-19T19:00:58.582 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-02-19T19:00:58.582 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-02-19T19:00:58.582 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-02-19T19:00:58.582 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-02-19T19:00:58.582 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-02-19T19:00:58.582 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-02-19T19:00:58.582 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-02-19T19:00:58.582 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 47 2024-02-19T19:00:58.582 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:00:58.582 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 2 2024-02-19T19:00:58.582 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1 2024-02-19T19:00:58.582 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=14530,1=24269} 2024-02-19T19:00:58.582 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-02-19T19:00:58.583 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-02-19T19:00:58.583 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-02-19T19:00:58.583 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-02-19T19:00:58.583 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-02-19T19:00:58.583 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data enabled 2024-02-19T19:00:58.583 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-02-19T19:00:58.583 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-02-19T19:00:58.583 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 1 2024-02-19T19:00:58.583 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:00:58.583 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.vpoeop{0:14540} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:00:58.583 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.znhuvd{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:00:58.583 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.stnfnf{1:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:00:58.583 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:00:58.583 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:00:58.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.565+0000 7f9114b31700 1 -- 172.21.15.114:0/3116408413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f90f806c4e0 msgr2=0x7f90f806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:58.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.565+0000 7f9114b31700 1 --2- 172.21.15.114:0/3116408413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f90f806c4e0 0x7f90f806e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f91000099d0 tx=0x7f9100008040 comp rx=0 tx=0).stop 2024-02-19T19:00:58.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.565+0000 7f9114b31700 1 -- 172.21.15.114:0/3116408413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9110104010 msgr2=0x7f91101968f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:58.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.565+0000 7f9114b31700 1 --2- 172.21.15.114:0/3116408413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9110104010 0x7f91101968f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f910400bd70 tx=0x7f910400be50 comp rx=0 tx=0).stop 2024-02-19T19:00:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.565+0000 7f9114b31700 1 -- 172.21.15.114:0/3116408413 shutdown_connections 2024-02-19T19:00:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.565+0000 7f9114b31700 1 --2- 172.21.15.114:0/3116408413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f90f806c4e0 0x7f90f806e9a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.565+0000 7f9114b31700 1 --2- 172.21.15.114:0/3116408413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91101016e0 0x7f91101963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.565+0000 7f9114b31700 1 --2- 172.21.15.114:0/3116408413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9110104010 0x7f91101968f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.565+0000 7f9114b31700 1 -- 172.21.15.114:0/3116408413 >> 172.21.15.114:0/3116408413 conn(0x7f91100fb030 msgr2=0x7f91100fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.565+0000 7f9114b31700 1 -- 172.21.15.114:0/3116408413 shutdown_connections 2024-02-19T19:00:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.565+0000 7f9114b31700 1 -- 172.21.15.114:0/3116408413 wait complete. 2024-02-19T19:00:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 18 2024-02-19T19:00:58.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.686+0000 7f85919cf700 1 -- 172.21.15.114:0/605720347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f858c103fd0 msgr2=0x7f858c104430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:58.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.686+0000 7f85919cf700 1 --2- 172.21.15.114:0/605720347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f858c103fd0 0x7f858c104430 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8580009910 tx=0x7f8580009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:58.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.686+0000 7f85919cf700 1 -- 172.21.15.114:0/605720347 shutdown_connections 2024-02-19T19:00:58.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.686+0000 7f85919cf700 1 --2- 172.21.15.114:0/605720347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f858c103fd0 0x7f858c104430 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:58.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.686+0000 7f85919cf700 1 --2- 172.21.15.114:0/605720347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f858c102dd0 0x7f858c1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:58.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.687+0000 7f85919cf700 1 -- 172.21.15.114:0/605720347 >> 172.21.15.114:0/605720347 conn(0x7f858c0fe390 msgr2=0x7f858c1007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:58.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.687+0000 7f85919cf700 1 -- 172.21.15.114:0/605720347 shutdown_connections 2024-02-19T19:00:58.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.687+0000 7f85919cf700 1 -- 172.21.15.114:0/605720347 wait complete. 2024-02-19T19:00:58.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.687+0000 7f85919cf700 1 Processor -- start 2024-02-19T19:00:58.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.687+0000 7f85919cf700 1 -- start start 2024-02-19T19:00:58.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.688+0000 7f85919cf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f858c102dd0 0x7f858c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:58.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.688+0000 7f85919cf700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f858c103fd0 0x7f858c198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:58.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.688+0000 7f85919cf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858c1990c0 con 0x7f858c102dd0 2024-02-19T19:00:58.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.688+0000 7f85919cf700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858c199200 con 0x7f858c103fd0 2024-02-19T19:00:58.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.688+0000 7f858affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f858c102dd0 0x7f858c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:58.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.688+0000 7f858affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f858c102dd0 0x7f858c198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35314/0 (socket says 172.21.15.114:35314) 2024-02-19T19:00:58.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.688+0000 7f858affd700 1 -- 172.21.15.114:0/540509470 learned_addr learned my addr 172.21.15.114:0/540509470 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:58.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.688+0000 7f858a7fc700 1 --2- 172.21.15.114:0/540509470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f858c103fd0 0x7f858c198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:58.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.688+0000 7f858affd700 1 -- 172.21.15.114:0/540509470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f858c103fd0 msgr2=0x7f858c198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:58.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.688+0000 7f858affd700 1 --2- 172.21.15.114:0/540509470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f858c103fd0 0x7f858c198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:58.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.688+0000 7f858affd700 1 -- 172.21.15.114:0/540509470 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85800095f0 con 0x7f858c102dd0 2024-02-19T19:00:58.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.688+0000 7f858affd700 1 --2- 172.21.15.114:0/540509470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f858c102dd0 0x7f858c198560 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f857c00d900 tx=0x7f857c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:58.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.688+0000 7f85909cd700 1 -- 172.21.15.114:0/540509470 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f857c0041f0 con 0x7f858c102dd0 2024-02-19T19:00:58.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.689+0000 7f85909cd700 1 -- 172.21.15.114:0/540509470 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f857c004350 con 0x7f858c102dd0 2024-02-19T19:00:58.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.689+0000 7f85909cd700 1 -- 172.21.15.114:0/540509470 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f857c003da0 con 0x7f858c102dd0 2024-02-19T19:00:58.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.689+0000 7f85919cf700 1 -- 172.21.15.114:0/540509470 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f858c19dcb0 con 0x7f858c102dd0 2024-02-19T19:00:58.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.689+0000 7f85919cf700 1 -- 172.21.15.114:0/540509470 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f858c19e150 con 0x7f858c102dd0 2024-02-19T19:00:58.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.690+0000 7f85919cf700 1 -- 172.21.15.114:0/540509470 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f858c10b9a0 con 0x7f858c102dd0 2024-02-19T19:00:58.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.690+0000 7f85909cd700 1 -- 172.21.15.114:0/540509470 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f857c009730 con 0x7f858c102dd0 2024-02-19T19:00:58.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.691+0000 7f85909cd700 1 --2- 172.21.15.114:0/540509470 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f857406c4e0 0x7f857406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:58.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.691+0000 7f85909cd700 1 -- 172.21.15.114:0/540509470 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f857c08b9f0 con 0x7f858c102dd0 2024-02-19T19:00:58.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.691+0000 7f858a7fc700 1 --2- 172.21.15.114:0/540509470 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f857406c4e0 0x7f857406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:58.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.691+0000 7f858a7fc700 1 --2- 172.21.15.114:0/540509470 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f857406c4e0 0x7f857406e9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f858000b5c0 tx=0x7f8580005960 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:58.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.695+0000 7f85909cd700 1 -- 172.21.15.114:0/540509470 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f857c08de60 con 0x7f858c102dd0 2024-02-19T19:00:58.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.925+0000 7f85919cf700 1 -- 172.21.15.114:0/540509470 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f858c108920 con 0x7f857406c4e0 2024-02-19T19:00:58.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.927+0000 7f85909cd700 1 -- 172.21.15.114:0/540509470 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f858c108920 con 0x7f857406c4e0 2024-02-19T19:00:58.946 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:00:58.946 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", 2024-02-19T19:00:58.946 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-02-19T19:00:58.946 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:00:58.946 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [], 2024-02-19T19:00:58.946 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "", 2024-02-19T19:00:58.946 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 image", 2024-02-19T19:00:58.946 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-02-19T19:00:58.946 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:00:58.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.930+0000 7f85919cf700 1 -- 172.21.15.114:0/540509470 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f857406c4e0 msgr2=0x7f857406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:58.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.930+0000 7f85919cf700 1 --2- 172.21.15.114:0/540509470 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f857406c4e0 0x7f857406e9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f858000b5c0 tx=0x7f8580005960 comp rx=0 tx=0).stop 2024-02-19T19:00:58.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.930+0000 7f85919cf700 1 -- 172.21.15.114:0/540509470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f858c102dd0 msgr2=0x7f858c198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:58.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.930+0000 7f85919cf700 1 --2- 172.21.15.114:0/540509470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f858c102dd0 0x7f858c198560 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f857c00d900 tx=0x7f857c00dcc0 comp rx=0 tx=0).stop 2024-02-19T19:00:58.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.930+0000 7f85919cf700 1 -- 172.21.15.114:0/540509470 shutdown_connections 2024-02-19T19:00:58.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.930+0000 7f85919cf700 1 --2- 172.21.15.114:0/540509470 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f857406c4e0 0x7f857406e9a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:58.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.930+0000 7f85919cf700 1 --2- 172.21.15.114:0/540509470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f858c102dd0 0x7f858c198560 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:58.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.930+0000 7f85919cf700 1 --2- 172.21.15.114:0/540509470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f858c103fd0 0x7f858c198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:58.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.930+0000 7f85919cf700 1 -- 172.21.15.114:0/540509470 >> 172.21.15.114:0/540509470 conn(0x7f858c0fe390 msgr2=0x7f858c107200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:58.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.930+0000 7f85919cf700 1 -- 172.21.15.114:0/540509470 shutdown_connections 2024-02-19T19:00:58.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:58.930+0000 7f85919cf700 1 -- 172.21.15.114:0/540509470 wait complete. 2024-02-19T19:00:59.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.042+0000 7fc91a870700 1 -- 172.21.15.114:0/1593003618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc914100cb0 msgr2=0x7fc914101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:59.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.042+0000 7fc91a870700 1 --2- 172.21.15.114:0/1593003618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc914100cb0 0x7fc914101110 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc908009910 tx=0x7fc908009c20 comp rx=0 tx=0).stop 2024-02-19T19:00:59.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.042+0000 7fc91a870700 1 -- 172.21.15.114:0/1593003618 shutdown_connections 2024-02-19T19:00:59.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.042+0000 7fc91a870700 1 --2- 172.21.15.114:0/1593003618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc914100cb0 0x7fc914101110 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:59.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.042+0000 7fc91a870700 1 --2- 172.21.15.114:0/1593003618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9140ffab0 0x7fc9140ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:59.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.042+0000 7fc91a870700 1 -- 172.21.15.114:0/1593003618 >> 172.21.15.114:0/1593003618 conn(0x7fc9140fb030 msgr2=0x7fc9140fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:59.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.042+0000 7fc91a870700 1 -- 172.21.15.114:0/1593003618 shutdown_connections 2024-02-19T19:00:59.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.042+0000 7fc91a870700 1 -- 172.21.15.114:0/1593003618 wait complete. 2024-02-19T19:00:59.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.042+0000 7fc91a870700 1 Processor -- start 2024-02-19T19:00:59.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.042+0000 7fc91a870700 1 -- start start 2024-02-19T19:00:59.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.043+0000 7fc91a870700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9140ffab0 0x7fc9141985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:59.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.043+0000 7fc91a870700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc914100cb0 0x7fc914198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:59.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.043+0000 7fc91a870700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc914199130 con 0x7fc9140ffab0 2024-02-19T19:00:59.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.043+0000 7fc91a870700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc914199270 con 0x7fc914100cb0 2024-02-19T19:00:59.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.043+0000 7fc913fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9140ffab0 0x7fc9141985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:59.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.043+0000 7fc913fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9140ffab0 0x7fc9141985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35328/0 (socket says 172.21.15.114:35328) 2024-02-19T19:00:59.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.043+0000 7fc913fff700 1 -- 172.21.15.114:0/1080424150 learned_addr learned my addr 172.21.15.114:0/1080424150 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:00:59.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.043+0000 7fc9137fe700 1 --2- 172.21.15.114:0/1080424150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc914100cb0 0x7fc914198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:59.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.043+0000 7fc913fff700 1 -- 172.21.15.114:0/1080424150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc914100cb0 msgr2=0x7fc914198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:59.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.043+0000 7fc913fff700 1 --2- 172.21.15.114:0/1080424150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc914100cb0 0x7fc914198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:59.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.043+0000 7fc913fff700 1 -- 172.21.15.114:0/1080424150 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9080095f0 con 0x7fc9140ffab0 2024-02-19T19:00:59.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.044+0000 7fc913fff700 1 --2- 172.21.15.114:0/1080424150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9140ffab0 0x7fc9141985d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc90400b480 tx=0x7fc90400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:59.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.044+0000 7fc9117fa700 1 -- 172.21.15.114:0/1080424150 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc904011800 con 0x7fc9140ffab0 2024-02-19T19:00:59.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.044+0000 7fc91a870700 1 -- 172.21.15.114:0/1080424150 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc91419dd20 con 0x7fc9140ffab0 2024-02-19T19:00:59.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.044+0000 7fc91a870700 1 -- 172.21.15.114:0/1080424150 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc91419e290 con 0x7fc9140ffab0 2024-02-19T19:00:59.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.044+0000 7fc9117fa700 1 -- 172.21.15.114:0/1080424150 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc904009330 con 0x7fc9140ffab0 2024-02-19T19:00:59.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.044+0000 7fc9117fa700 1 -- 172.21.15.114:0/1080424150 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc90401ae60 con 0x7fc9140ffab0 2024-02-19T19:00:59.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.045+0000 7fc91a870700 1 -- 172.21.15.114:0/1080424150 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc900005320 con 0x7fc9140ffab0 2024-02-19T19:00:59.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.046+0000 7fc9117fa700 1 -- 172.21.15.114:0/1080424150 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fc904019460 con 0x7fc9140ffab0 2024-02-19T19:00:59.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.046+0000 7fc9117fa700 1 --2- 172.21.15.114:0/1080424150 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc8fc06c4e0 0x7fc8fc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:59.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.046+0000 7fc9117fa700 1 -- 172.21.15.114:0/1080424150 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc90408b2c0 con 0x7fc9140ffab0 2024-02-19T19:00:59.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.046+0000 7fc9137fe700 1 --2- 172.21.15.114:0/1080424150 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc8fc06c4e0 0x7fc8fc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:59.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.047+0000 7fc9137fe700 1 --2- 172.21.15.114:0/1080424150 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc8fc06c4e0 0x7fc8fc06e9a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc90800b5c0 tx=0x7fc9080052a0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:59.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.052+0000 7fc9117fa700 1 -- 172.21.15.114:0/1080424150 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc9040594d0 con 0x7fc9140ffab0 2024-02-19T19:00:59.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:58 smithi114 ceph-mon[29441]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:59.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:58 smithi114 ceph-mon[29441]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:59.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:58 smithi114 ceph-mon[29441]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:59.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:58 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/3006878386' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:00:59.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:58 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/3116408413' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:00:59.326 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:58 smithi174 ceph-mon[41061]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:59.326 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:58 smithi174 ceph-mon[41061]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:59.327 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:58 smithi174 ceph-mon[41061]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:59.327 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:58 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/3006878386' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:00:59.327 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:58 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/3116408413' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:00:59.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.353+0000 7fc91a870700 1 -- 172.21.15.114:0/1080424150 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc900005190 con 0x7fc9140ffab0 2024-02-19T19:00:59.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.354+0000 7fc9117fa700 1 -- 172.21.15.114:0/1080424150 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fc904059060 con 0x7fc9140ffab0 2024-02-19T19:00:59.372 INFO:teuthology.orchestra.run.smithi114.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:00:59.372 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:00:59.373 INFO:teuthology.orchestra.run.smithi114.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:00:59.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.356+0000 7fc91a870700 1 -- 172.21.15.114:0/1080424150 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc8fc06c4e0 msgr2=0x7fc8fc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:59.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.356+0000 7fc91a870700 1 --2- 172.21.15.114:0/1080424150 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc8fc06c4e0 0x7fc8fc06e9a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc90800b5c0 tx=0x7fc9080052a0 comp rx=0 tx=0).stop 2024-02-19T19:00:59.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.356+0000 7fc91a870700 1 -- 172.21.15.114:0/1080424150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9140ffab0 msgr2=0x7fc9141985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:59.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.356+0000 7fc91a870700 1 --2- 172.21.15.114:0/1080424150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9140ffab0 0x7fc9141985d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc90400b480 tx=0x7fc90400b790 comp rx=0 tx=0).stop 2024-02-19T19:00:59.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.356+0000 7fc91a870700 1 -- 172.21.15.114:0/1080424150 shutdown_connections 2024-02-19T19:00:59.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.356+0000 7fc91a870700 1 --2- 172.21.15.114:0/1080424150 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc8fc06c4e0 0x7fc8fc06e9a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:59.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.356+0000 7fc91a870700 1 --2- 172.21.15.114:0/1080424150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9140ffab0 0x7fc9141985d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:59.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.356+0000 7fc91a870700 1 --2- 172.21.15.114:0/1080424150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc914100cb0 0x7fc914198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:59.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.356+0000 7fc91a870700 1 -- 172.21.15.114:0/1080424150 >> 172.21.15.114:0/1080424150 conn(0x7fc9140fb030 msgr2=0x7fc914068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:59.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.356+0000 7fc91a870700 1 -- 172.21.15.114:0/1080424150 shutdown_connections 2024-02-19T19:00:59.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:00:59.356+0000 7fc91a870700 1 -- 172.21.15.114:0/1080424150 wait complete. 2024-02-19T19:01:00.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:59 smithi114 ceph-mon[29441]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:01:00.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:59 smithi114 ceph-mon[29441]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:00.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:00:59 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1080424150' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:01:00.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:59 smithi174 ceph-mon[41061]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:01:00.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:59 smithi174 ceph-mon[41061]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:00.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:00:59 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1080424150' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:01:02.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:01 smithi174 ceph-mon[41061]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:01:02.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:01 smithi114 ceph-mon[29441]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:01:03.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:02 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:03.392 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:02 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:04.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:03 smithi114 ceph-mon[29441]: pgmap v215: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-19T19:01:04.376 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:03 smithi174 ceph-mon[41061]: pgmap v215: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-19T19:01:05.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:04 smithi114 ceph-mon[29441]: pgmap v216: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-19T19:01:05.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:04 smithi174 ceph-mon[41061]: pgmap v216: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-19T19:01:08.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:07 smithi114 ceph-mon[29441]: pgmap v217: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:01:08.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:07 smithi174 ceph-mon[41061]: pgmap v217: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:01:10.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:09 smithi114 ceph-mon[29441]: pgmap v218: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:01:10.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:09 smithi174 ceph-mon[41061]: pgmap v218: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:01:12.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:11 smithi114 ceph-mon[29441]: pgmap v219: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:01:12.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:11 smithi174 ceph-mon[41061]: pgmap v219: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:01:14.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:13 smithi114 ceph-mon[29441]: pgmap v220: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-19T19:01:14.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:13 smithi174 ceph-mon[41061]: pgmap v220: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-19T19:01:15.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:15 smithi174 ceph-mon[41061]: pgmap v221: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-19T19:01:15.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:15 smithi114 ceph-mon[29441]: pgmap v221: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-19T19:01:18.016 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:17 smithi114 ceph-mon[29441]: pgmap v222: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:01:18.016 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:17 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:18.096 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:17 smithi174 ceph-mon[41061]: pgmap v222: 65 pgs: 65 active+clean; 586 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:01:18.096 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:17 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:20.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:19 smithi114 ceph-mon[29441]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 426 B/s wr, 4 op/s 2024-02-19T19:01:20.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:19 smithi174 ceph-mon[41061]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 426 B/s wr, 4 op/s 2024-02-19T19:01:22.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:21 smithi114 ceph-mon[29441]: pgmap v224: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-02-19T19:01:22.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:21 smithi174 ceph-mon[41061]: pgmap v224: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-02-19T19:01:24.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:23 smithi114 ceph-mon[29441]: pgmap v225: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-02-19T19:01:24.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:23 smithi174 ceph-mon[41061]: pgmap v225: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-02-19T19:01:26.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:25 smithi114 ceph-mon[29441]: pgmap v226: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-02-19T19:01:26.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:25 smithi174 ceph-mon[41061]: pgmap v226: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-02-19T19:01:27.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:27 smithi174 ceph-mon[41061]: pgmap v227: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 4 op/s 2024-02-19T19:01:27.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:27 smithi114 ceph-mon[29441]: pgmap v227: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 4 op/s 2024-02-19T19:01:29.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.495+0000 7f3659985700 1 -- 172.21.15.114:0/507250848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3654101640 msgr2=0x7f3654103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:29.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.495+0000 7f3659985700 1 --2- 172.21.15.114:0/507250848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3654101640 0x7f3654103a30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3644009910 tx=0x7f3644009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:29.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.495+0000 7f3659985700 1 -- 172.21.15.114:0/507250848 shutdown_connections 2024-02-19T19:01:29.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.495+0000 7f3659985700 1 --2- 172.21.15.114:0/507250848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3654103f70 0x7f3654106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.495+0000 7f3659985700 1 --2- 172.21.15.114:0/507250848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3654101640 0x7f3654103a30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.495+0000 7f3659985700 1 -- 172.21.15.114:0/507250848 >> 172.21.15.114:0/507250848 conn(0x7f36540fb030 msgr2=0x7f36540fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:29.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.495+0000 7f3659985700 1 -- 172.21.15.114:0/507250848 shutdown_connections 2024-02-19T19:01:29.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.495+0000 7f3659985700 1 -- 172.21.15.114:0/507250848 wait complete. 2024-02-19T19:01:29.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.495+0000 7f3659985700 1 Processor -- start 2024-02-19T19:01:29.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.496+0000 7f3659985700 1 -- start start 2024-02-19T19:01:29.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.496+0000 7f3659985700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3654101640 0x7f3654196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:29.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.496+0000 7f3659985700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3654103f70 0x7f36541968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:29.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.496+0000 7f3659985700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3654196ee0 con 0x7f3654103f70 2024-02-19T19:01:29.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.496+0000 7f3659985700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3654197020 con 0x7f3654101640 2024-02-19T19:01:29.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.496+0000 7f36527fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3654103f70 0x7f36541968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:29.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.496+0000 7f36527fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3654103f70 0x7f36541968c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54308/0 (socket says 172.21.15.114:54308) 2024-02-19T19:01:29.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.496+0000 7f36527fc700 1 -- 172.21.15.114:0/3562068104 learned_addr learned my addr 172.21.15.114:0/3562068104 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:01:29.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.496+0000 7f3652ffd700 1 --2- 172.21.15.114:0/3562068104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3654101640 0x7f3654196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:29.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.497+0000 7f36527fc700 1 -- 172.21.15.114:0/3562068104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3654101640 msgr2=0x7f3654196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:29.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.497+0000 7f36527fc700 1 --2- 172.21.15.114:0/3562068104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3654101640 0x7f3654196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.497+0000 7f36527fc700 1 -- 172.21.15.114:0/3562068104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36440095f0 con 0x7f3654103f70 2024-02-19T19:01:29.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.497+0000 7f36527fc700 1 --2- 172.21.15.114:0/3562068104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3654103f70 0x7f36541968c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f364800b4b0 tx=0x7f364800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:29.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.497+0000 7f3658983700 1 -- 172.21.15.114:0/3562068104 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3648011800 con 0x7f3654103f70 2024-02-19T19:01:29.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.497+0000 7f3659985700 1 -- 172.21.15.114:0/3562068104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f365419bad0 con 0x7f3654103f70 2024-02-19T19:01:29.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.497+0000 7f3658983700 1 -- 172.21.15.114:0/3562068104 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3648011e10 con 0x7f3654103f70 2024-02-19T19:01:29.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.497+0000 7f3658983700 1 -- 172.21.15.114:0/3562068104 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f364800f2e0 con 0x7f3654103f70 2024-02-19T19:01:29.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.497+0000 7f3659985700 1 -- 172.21.15.114:0/3562068104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f365419c0a0 con 0x7f3654103f70 2024-02-19T19:01:29.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.498+0000 7f3659985700 1 -- 172.21.15.114:0/3562068104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36541905a0 con 0x7f3654103f70 2024-02-19T19:01:29.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.499+0000 7f3658983700 1 -- 172.21.15.114:0/3562068104 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3648010380 con 0x7f3654103f70 2024-02-19T19:01:29.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.500+0000 7f3658983700 1 --2- 172.21.15.114:0/3562068104 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f363c06c4e0 0x7f363c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:29.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.500+0000 7f3658983700 1 -- 172.21.15.114:0/3562068104 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f364808b360 con 0x7f3654103f70 2024-02-19T19:01:29.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.500+0000 7f3652ffd700 1 --2- 172.21.15.114:0/3562068104 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f363c06c4e0 0x7f363c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:29.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.500+0000 7f3652ffd700 1 --2- 172.21.15.114:0/3562068104 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f363c06c4e0 0x7f363c06e9a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3644006010 tx=0x7f364400b560 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:29.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.506+0000 7f3658983700 1 -- 172.21.15.114:0/3562068104 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f36480595f0 con 0x7f3654103f70 2024-02-19T19:01:29.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.754+0000 7f3659985700 1 -- 172.21.15.114:0/3562068104 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f36540611d0 con 0x7f363c06c4e0 2024-02-19T19:01:29.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.756+0000 7f3658983700 1 -- 172.21.15.114:0/3562068104 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f36540611d0 con 0x7f363c06c4e0 2024-02-19T19:01:29.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.759+0000 7f3659985700 1 -- 172.21.15.114:0/3562068104 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f363c06c4e0 msgr2=0x7f363c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:29.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.759+0000 7f3659985700 1 --2- 172.21.15.114:0/3562068104 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f363c06c4e0 0x7f363c06e9a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3644006010 tx=0x7f364400b560 comp rx=0 tx=0).stop 2024-02-19T19:01:29.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.759+0000 7f3659985700 1 -- 172.21.15.114:0/3562068104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3654103f70 msgr2=0x7f36541968c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:29.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.759+0000 7f3659985700 1 --2- 172.21.15.114:0/3562068104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3654103f70 0x7f36541968c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f364800b4b0 tx=0x7f364800b7c0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.759+0000 7f3659985700 1 -- 172.21.15.114:0/3562068104 shutdown_connections 2024-02-19T19:01:29.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.759+0000 7f3659985700 1 --2- 172.21.15.114:0/3562068104 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f363c06c4e0 0x7f363c06e9a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.759+0000 7f3659985700 1 --2- 172.21.15.114:0/3562068104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3654101640 0x7f3654196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.759+0000 7f3659985700 1 --2- 172.21.15.114:0/3562068104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3654103f70 0x7f36541968c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.759+0000 7f3659985700 1 -- 172.21.15.114:0/3562068104 >> 172.21.15.114:0/3562068104 conn(0x7f36540fb030 msgr2=0x7f36540fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:29.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.759+0000 7f3659985700 1 -- 172.21.15.114:0/3562068104 shutdown_connections 2024-02-19T19:01:29.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.759+0000 7f3659985700 1 -- 172.21.15.114:0/3562068104 wait complete. 2024-02-19T19:01:29.789 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-02-19T19:01:29.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.883+0000 7f92aeb6f700 1 -- 172.21.15.114:0/4204190618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92a8101dc0 msgr2=0x7f92a8102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:29.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.883+0000 7f92aeb6f700 1 --2- 172.21.15.114:0/4204190618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92a8101dc0 0x7f92a8102220 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9294009910 tx=0x7f9294009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:29.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.883+0000 7f92aeb6f700 1 -- 172.21.15.114:0/4204190618 shutdown_connections 2024-02-19T19:01:29.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.883+0000 7f92aeb6f700 1 --2- 172.21.15.114:0/4204190618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92a8101dc0 0x7f92a8102220 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.883+0000 7f92aeb6f700 1 --2- 172.21.15.114:0/4204190618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92a8100bc0 0x7f92a8100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.883+0000 7f92aeb6f700 1 -- 172.21.15.114:0/4204190618 >> 172.21.15.114:0/4204190618 conn(0x7f92a80fc140 msgr2=0x7f92a80fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:29.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.883+0000 7f92aeb6f700 1 -- 172.21.15.114:0/4204190618 shutdown_connections 2024-02-19T19:01:29.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.883+0000 7f92aeb6f700 1 -- 172.21.15.114:0/4204190618 wait complete. 2024-02-19T19:01:29.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.883+0000 7f92aeb6f700 1 Processor -- start 2024-02-19T19:01:29.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.883+0000 7f92aeb6f700 1 -- start start 2024-02-19T19:01:29.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.884+0000 7f92aeb6f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92a8100bc0 0x7f92a8198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:29.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.884+0000 7f92aeb6f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92a8101dc0 0x7f92a8198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:29.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.884+0000 7f92aeb6f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a81990e0 con 0x7f92a8101dc0 2024-02-19T19:01:29.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.884+0000 7f92aeb6f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a8199220 con 0x7f92a8100bc0 2024-02-19T19:01:29.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.884+0000 7f92a7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92a8101dc0 0x7f92a8198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:29.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.884+0000 7f92a7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92a8101dc0 0x7f92a8198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54336/0 (socket says 172.21.15.114:54336) 2024-02-19T19:01:29.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.884+0000 7f92a7fff700 1 -- 172.21.15.114:0/2612220357 learned_addr learned my addr 172.21.15.114:0/2612220357 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:01:29.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.884+0000 7f92ac90b700 1 --2- 172.21.15.114:0/2612220357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92a8100bc0 0x7f92a8198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:29.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.884+0000 7f92a7fff700 1 -- 172.21.15.114:0/2612220357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92a8100bc0 msgr2=0x7f92a8198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:29.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.884+0000 7f92a7fff700 1 --2- 172.21.15.114:0/2612220357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92a8100bc0 0x7f92a8198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:29.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.884+0000 7f92a7fff700 1 -- 172.21.15.114:0/2612220357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92940095f0 con 0x7f92a8101dc0 2024-02-19T19:01:29.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.884+0000 7f92a7fff700 1 --2- 172.21.15.114:0/2612220357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92a8101dc0 0x7f92a8198ac0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9294005b40 tx=0x7f929400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:29.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.884+0000 7f92a5ffb700 1 -- 172.21.15.114:0/2612220357 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f929401d070 con 0x7f92a8101dc0 2024-02-19T19:01:29.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.884+0000 7f92a5ffb700 1 -- 172.21.15.114:0/2612220357 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f929400f460 con 0x7f92a8101dc0 2024-02-19T19:01:29.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.884+0000 7f92a5ffb700 1 -- 172.21.15.114:0/2612220357 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9294021620 con 0x7f92a8101dc0 2024-02-19T19:01:29.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.884+0000 7f92aeb6f700 1 -- 172.21.15.114:0/2612220357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a819dc70 con 0x7f92a8101dc0 2024-02-19T19:01:29.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.885+0000 7f92aeb6f700 1 -- 172.21.15.114:0/2612220357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a819e160 con 0x7f92a8101dc0 2024-02-19T19:01:29.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.886+0000 7f92aeb6f700 1 -- 172.21.15.114:0/2612220357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a8105ea0 con 0x7f92a8101dc0 2024-02-19T19:01:29.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.886+0000 7f92a5ffb700 1 -- 172.21.15.114:0/2612220357 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f929400fa40 con 0x7f92a8101dc0 2024-02-19T19:01:29.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.887+0000 7f92a5ffb700 1 --2- 172.21.15.114:0/2612220357 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f929806c4e0 0x7f929806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:29.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.887+0000 7f92a5ffb700 1 -- 172.21.15.114:0/2612220357 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f929408c9c0 con 0x7f92a8101dc0 2024-02-19T19:01:29.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.887+0000 7f92ac90b700 1 --2- 172.21.15.114:0/2612220357 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f929806c4e0 0x7f929806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:29.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.887+0000 7f92ac90b700 1 --2- 172.21.15.114:0/2612220357 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f929806c4e0 0x7f929806e9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f92a8101c20 tx=0x7f929c00b410 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:29.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:29.896+0000 7f92a5ffb700 1 -- 172.21.15.114:0/2612220357 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f929405ac50 con 0x7f92a8101dc0 2024-02-19T19:01:30.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:29 smithi114 ceph-mon[29441]: pgmap v228: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-02-19T19:01:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.128+0000 7f92aeb6f700 1 -- 172.21.15.114:0/2612220357 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f92a80611d0 con 0x7f929806c4e0 2024-02-19T19:01:30.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.130+0000 7f92a5ffb700 1 -- 172.21.15.114:0/2612220357 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f92a80611d0 con 0x7f929806c4e0 2024-02-19T19:01:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.132+0000 7f92aeb6f700 1 -- 172.21.15.114:0/2612220357 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f929806c4e0 msgr2=0x7f929806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.132+0000 7f92aeb6f700 1 --2- 172.21.15.114:0/2612220357 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f929806c4e0 0x7f929806e9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f92a8101c20 tx=0x7f929c00b410 comp rx=0 tx=0).stop 2024-02-19T19:01:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.132+0000 7f92aeb6f700 1 -- 172.21.15.114:0/2612220357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92a8101dc0 msgr2=0x7f92a8198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.132+0000 7f92aeb6f700 1 --2- 172.21.15.114:0/2612220357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92a8101dc0 0x7f92a8198ac0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9294005b40 tx=0x7f929400bd70 comp rx=0 tx=0).stop 2024-02-19T19:01:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.132+0000 7f92aeb6f700 1 -- 172.21.15.114:0/2612220357 shutdown_connections 2024-02-19T19:01:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.132+0000 7f92aeb6f700 1 --2- 172.21.15.114:0/2612220357 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f929806c4e0 0x7f929806e9a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.132+0000 7f92aeb6f700 1 --2- 172.21.15.114:0/2612220357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92a8100bc0 0x7f92a8198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.132+0000 7f92aeb6f700 1 --2- 172.21.15.114:0/2612220357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92a8101dc0 0x7f92a8198ac0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.132+0000 7f92aeb6f700 1 -- 172.21.15.114:0/2612220357 >> 172.21.15.114:0/2612220357 conn(0x7f92a80fc140 msgr2=0x7f92a8102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.132+0000 7f92aeb6f700 1 -- 172.21.15.114:0/2612220357 shutdown_connections 2024-02-19T19:01:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.132+0000 7f92aeb6f700 1 -- 172.21.15.114:0/2612220357 wait complete. 2024-02-19T19:01:30.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.244+0000 7fc651dad700 1 -- 172.21.15.114:0/2335572958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc64c0ffae0 msgr2=0x7fc64c0fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:30.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.244+0000 7fc651dad700 1 --2- 172.21.15.114:0/2335572958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc64c0ffae0 0x7fc64c0fff00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc63c009910 tx=0x7fc63c009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:30.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.245+0000 7fc651dad700 1 -- 172.21.15.114:0/2335572958 shutdown_connections 2024-02-19T19:01:30.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.245+0000 7fc651dad700 1 --2- 172.21.15.114:0/2335572958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc64c100ce0 0x7fc64c101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:30.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.245+0000 7fc651dad700 1 --2- 172.21.15.114:0/2335572958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc64c0ffae0 0x7fc64c0fff00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:30.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.245+0000 7fc651dad700 1 -- 172.21.15.114:0/2335572958 >> 172.21.15.114:0/2335572958 conn(0x7fc64c0fb080 msgr2=0x7fc64c0fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:30.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.245+0000 7fc651dad700 1 -- 172.21.15.114:0/2335572958 shutdown_connections 2024-02-19T19:01:30.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.245+0000 7fc651dad700 1 -- 172.21.15.114:0/2335572958 wait complete. 2024-02-19T19:01:30.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.245+0000 7fc651dad700 1 Processor -- start 2024-02-19T19:01:30.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.245+0000 7fc651dad700 1 -- start start 2024-02-19T19:01:30.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.246+0000 7fc651dad700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc64c0ffae0 0x7fc64c069890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:30.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.246+0000 7fc651dad700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc64c100ce0 0x7fc64c19b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:30.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.246+0000 7fc651dad700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c19b6a0 con 0x7fc64c100ce0 2024-02-19T19:01:30.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.246+0000 7fc651dad700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c19b7e0 con 0x7fc64c0ffae0 2024-02-19T19:01:30.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.246+0000 7fc64affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc64c100ce0 0x7fc64c19b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:30.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.246+0000 7fc64affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc64c100ce0 0x7fc64c19b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54354/0 (socket says 172.21.15.114:54354) 2024-02-19T19:01:30.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.246+0000 7fc64affd700 1 -- 172.21.15.114:0/2422228990 learned_addr learned my addr 172.21.15.114:0/2422228990 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:01:30.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.246+0000 7fc64b7fe700 1 --2- 172.21.15.114:0/2422228990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc64c0ffae0 0x7fc64c069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:30.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.246+0000 7fc64affd700 1 -- 172.21.15.114:0/2422228990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc64c0ffae0 msgr2=0x7fc64c069890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:30.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.246+0000 7fc64affd700 1 --2- 172.21.15.114:0/2422228990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc64c0ffae0 0x7fc64c069890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:30.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.246+0000 7fc64affd700 1 -- 172.21.15.114:0/2422228990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63c0095f0 con 0x7fc64c100ce0 2024-02-19T19:01:30.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.246+0000 7fc64affd700 1 --2- 172.21.15.114:0/2422228990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc64c100ce0 0x7fc64c19b0d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc64000d900 tx=0x7fc64000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:30.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.246+0000 7fc648ff9700 1 -- 172.21.15.114:0/2422228990 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6400041f0 con 0x7fc64c100ce0 2024-02-19T19:01:30.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.246+0000 7fc648ff9700 1 -- 172.21.15.114:0/2422228990 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc640004ce0 con 0x7fc64c100ce0 2024-02-19T19:01:30.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.246+0000 7fc651dad700 1 -- 172.21.15.114:0/2422228990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc64c19bac0 con 0x7fc64c100ce0 2024-02-19T19:01:30.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.247+0000 7fc648ff9700 1 -- 172.21.15.114:0/2422228990 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc64000b750 con 0x7fc64c100ce0 2024-02-19T19:01:30.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.247+0000 7fc651dad700 1 -- 172.21.15.114:0/2422228990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc64c19c010 con 0x7fc64c100ce0 2024-02-19T19:01:30.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.248+0000 7fc651dad700 1 -- 172.21.15.114:0/2422228990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc64c108640 con 0x7fc64c100ce0 2024-02-19T19:01:30.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.248+0000 7fc648ff9700 1 -- 172.21.15.114:0/2422228990 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fc640004750 con 0x7fc64c100ce0 2024-02-19T19:01:30.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.248+0000 7fc648ff9700 1 --2- 172.21.15.114:0/2422228990 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc63406c4e0 0x7fc63406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:30.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.249+0000 7fc648ff9700 1 -- 172.21.15.114:0/2422228990 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc64008b770 con 0x7fc64c100ce0 2024-02-19T19:01:30.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.249+0000 7fc64b7fe700 1 --2- 172.21.15.114:0/2422228990 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc63406c4e0 0x7fc63406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:30.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.249+0000 7fc64b7fe700 1 --2- 172.21.15.114:0/2422228990 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc63406c4e0 0x7fc63406e9a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc63c006010 tx=0x7fc63c005ec0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:30.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.257+0000 7fc648ff9700 1 -- 172.21.15.114:0/2422228990 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc640059980 con 0x7fc64c100ce0 2024-02-19T19:01:30.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:29 smithi174 ceph-mon[41061]: pgmap v228: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-02-19T19:01:30.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.491+0000 7fc651dad700 1 -- 172.21.15.114:0/2422228990 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc64c105630 con 0x7fc63406c4e0 2024-02-19T19:01:30.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.504+0000 7fc648ff9700 1 -- 172.21.15.114:0/2422228990 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7fc64c105630 con 0x7fc63406c4e0 2024-02-19T19:01:30.523 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:01:30.523 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (5m) 60s ago 8m 26.5M - 0.25.0 c8568f914cd2 30444b676ca4 2024-02-19T19:01:30.523 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (8m) 60s ago 8m 7470k - 18.2.1 94eadd990e0a 4918cd87e312 2024-02-19T19:01:30.523 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi174 smithi174 running (3m) 66s ago 6m 7394k - 18.2.1 94eadd990e0a 516c34431b93 2024-02-19T19:01:30.523 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (8m) 60s ago 8m 6660k - 18.2.1 94eadd990e0a ecfcd98907ba 2024-02-19T19:01:30.523 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi174 smithi174 running (6m) 66s ago 6m 6660k - 18.2.1 94eadd990e0a de88985f5a07 2024-02-19T19:01:30.524 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (5m) 60s ago 7m 84.7M - 9.4.7 954c08fa6188 478bd533e827 2024-02-19T19:01:30.524 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tcwpod smithi114 running (78s) 60s ago 78s 15.1M - 18.2.1 94eadd990e0a 420175777cf4 2024-02-19T19:01:30.524 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.vpoeop smithi114 running (85s) 60s ago 85s 11.8M - 18.2.1 94eadd990e0a f81a79e53ead 2024-02-19T19:01:30.524 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.stnfnf smithi174 running (75s) 66s ago 75s 14.6M - 18.2.1 94eadd990e0a 3df35954a492 2024-02-19T19:01:30.524 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.znhuvd smithi174 running (82s) 66s ago 82s 15.1M - 18.2.1 94eadd990e0a 2d2267494ae1 2024-02-19T19:01:30.524 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.pcvtrk smithi114 *:9283,8765,8443 running (9m) 60s ago 9m 489M - 18.2.1 94eadd990e0a 9d81af39d736 2024-02-19T19:01:30.524 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi174.lydvnj smithi174 *:8443,9283,8765 running (5m) 66s ago 5m 443M - 18.2.1 94eadd990e0a 9b31a7d679b9 2024-02-19T19:01:30.524 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (9m) 60s ago 9m 46.8M 2048M 18.2.1 94eadd990e0a 41468f811fc0 2024-02-19T19:01:30.524 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi174 smithi174 running (5m) 66s ago 5m 39.9M 2048M 18.2.1 94eadd990e0a 70dd7bf69fd9 2024-02-19T19:01:30.524 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (8m) 60s ago 8m 14.2M - 1.5.0 0da6a335fe13 b6b258074cce 2024-02-19T19:01:30.524 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi174 smithi174 *:9100 running (6m) 66s ago 6m 15.2M - 1.5.0 0da6a335fe13 d0b4fdf67b22 2024-02-19T19:01:30.524 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (5m) 60s ago 5m 46.8M 3946M 18.2.1 94eadd990e0a 990afc982b6c 2024-02-19T19:01:30.525 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (4m) 60s ago 4m 47.2M 3946M 18.2.1 94eadd990e0a 932e3ad37db5 2024-02-19T19:01:30.525 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (4m) 60s ago 4m 48.7M 3946M 18.2.1 94eadd990e0a 12953988af86 2024-02-19T19:01:30.525 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi174 running (3m) 66s ago 3m 46.1M 4970M 18.2.1 94eadd990e0a 979f45cabbce 2024-02-19T19:01:30.525 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi174 running (3m) 66s ago 3m 46.4M 4970M 18.2.1 94eadd990e0a 8b8c3745b8c5 2024-02-19T19:01:30.525 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi174 running (2m) 66s ago 2m 48.6M 4970M 18.2.1 94eadd990e0a 5771a995ed75 2024-02-19T19:01:30.525 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (5m) 60s ago 6m 55.2M - 2.43.0 a07b618ecd1d 52087f218c92 2024-02-19T19:01:30.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.506+0000 7fc651dad700 1 -- 172.21.15.114:0/2422228990 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc63406c4e0 msgr2=0x7fc63406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:30.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.506+0000 7fc651dad700 1 --2- 172.21.15.114:0/2422228990 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc63406c4e0 0x7fc63406e9a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc63c006010 tx=0x7fc63c005ec0 comp rx=0 tx=0).stop 2024-02-19T19:01:30.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.506+0000 7fc651dad700 1 -- 172.21.15.114:0/2422228990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc64c100ce0 msgr2=0x7fc64c19b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:30.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.506+0000 7fc651dad700 1 --2- 172.21.15.114:0/2422228990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc64c100ce0 0x7fc64c19b0d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc64000d900 tx=0x7fc64000dc10 comp rx=0 tx=0).stop 2024-02-19T19:01:30.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.507+0000 7fc651dad700 1 -- 172.21.15.114:0/2422228990 shutdown_connections 2024-02-19T19:01:30.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.507+0000 7fc651dad700 1 --2- 172.21.15.114:0/2422228990 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fc63406c4e0 0x7fc63406e9a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:30.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.507+0000 7fc651dad700 1 --2- 172.21.15.114:0/2422228990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc64c0ffae0 0x7fc64c069890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:30.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.507+0000 7fc651dad700 1 --2- 172.21.15.114:0/2422228990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc64c100ce0 0x7fc64c19b0d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:30.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.507+0000 7fc651dad700 1 -- 172.21.15.114:0/2422228990 >> 172.21.15.114:0/2422228990 conn(0x7fc64c0fb080 msgr2=0x7fc64c103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:30.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.507+0000 7fc651dad700 1 -- 172.21.15.114:0/2422228990 shutdown_connections 2024-02-19T19:01:30.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.507+0000 7fc651dad700 1 -- 172.21.15.114:0/2422228990 wait complete. 2024-02-19T19:01:30.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.645+0000 7fd920077700 1 -- 172.21.15.114:0/3427382130 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9180ffad0 msgr2=0x7fd9180ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.645+0000 7fd920077700 1 --2- 172.21.15.114:0/3427382130 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9180ffad0 0x7fd9180ffef0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd914009910 tx=0x7fd914009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.645+0000 7fd920077700 1 -- 172.21.15.114:0/3427382130 shutdown_connections 2024-02-19T19:01:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.645+0000 7fd920077700 1 --2- 172.21.15.114:0/3427382130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd918100cd0 0x7fd918101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.645+0000 7fd920077700 1 --2- 172.21.15.114:0/3427382130 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9180ffad0 0x7fd9180ffef0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.645+0000 7fd920077700 1 -- 172.21.15.114:0/3427382130 >> 172.21.15.114:0/3427382130 conn(0x7fd9180fb050 msgr2=0x7fd9180fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.646+0000 7fd920077700 1 -- 172.21.15.114:0/3427382130 shutdown_connections 2024-02-19T19:01:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.646+0000 7fd920077700 1 -- 172.21.15.114:0/3427382130 wait complete. 2024-02-19T19:01:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.646+0000 7fd920077700 1 Processor -- start 2024-02-19T19:01:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.646+0000 7fd920077700 1 -- start start 2024-02-19T19:01:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.646+0000 7fd920077700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9180ffad0 0x7fd918195280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:30.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.646+0000 7fd920077700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd918100cd0 0x7fd9181957c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:30.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.646+0000 7fd920077700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd918195de0 con 0x7fd918100cd0 2024-02-19T19:01:30.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.647+0000 7fd920077700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd918195f20 con 0x7fd9180ffad0 2024-02-19T19:01:30.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.647+0000 7fd91de13700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9180ffad0 0x7fd918195280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:30.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.647+0000 7fd91de13700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9180ffad0 0x7fd918195280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.114:48080/0 (socket says 172.21.15.114:48080) 2024-02-19T19:01:30.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.647+0000 7fd91de13700 1 -- 172.21.15.114:0/767367413 learned_addr learned my addr 172.21.15.114:0/767367413 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:01:30.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.647+0000 7fd91d612700 1 --2- 172.21.15.114:0/767367413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd918100cd0 0x7fd9181957c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:30.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.647+0000 7fd91de13700 1 -- 172.21.15.114:0/767367413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd918100cd0 msgr2=0x7fd9181957c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:30.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.647+0000 7fd91de13700 1 --2- 172.21.15.114:0/767367413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd918100cd0 0x7fd9181957c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:30.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.647+0000 7fd91de13700 1 -- 172.21.15.114:0/767367413 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9140095f0 con 0x7fd9180ffad0 2024-02-19T19:01:30.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.647+0000 7fd91de13700 1 --2- 172.21.15.114:0/767367413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9180ffad0 0x7fd918195280 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd91400fa00 tx=0x7fd91400fa30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:30.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.647+0000 7fd90effd700 1 -- 172.21.15.114:0/767367413 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd91401d070 con 0x7fd9180ffad0 2024-02-19T19:01:30.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.648+0000 7fd920077700 1 -- 172.21.15.114:0/767367413 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd918068d30 con 0x7fd9180ffad0 2024-02-19T19:01:30.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.648+0000 7fd90effd700 1 -- 172.21.15.114:0/767367413 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd91400fe90 con 0x7fd9180ffad0 2024-02-19T19:01:30.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.648+0000 7fd90effd700 1 -- 172.21.15.114:0/767367413 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd914017630 con 0x7fd9180ffad0 2024-02-19T19:01:30.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.648+0000 7fd920077700 1 -- 172.21.15.114:0/767367413 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd918069220 con 0x7fd9180ffad0 2024-02-19T19:01:30.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.648+0000 7fd920077700 1 -- 172.21.15.114:0/767367413 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9181087c0 con 0x7fd9180ffad0 2024-02-19T19:01:30.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.649+0000 7fd90effd700 1 -- 172.21.15.114:0/767367413 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fd914003680 con 0x7fd9180ffad0 2024-02-19T19:01:30.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.650+0000 7fd90effd700 1 --2- 172.21.15.114:0/767367413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd90406c4e0 0x7fd90406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:30.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.650+0000 7fd90effd700 1 -- 172.21.15.114:0/767367413 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd91408ccc0 con 0x7fd9180ffad0 2024-02-19T19:01:30.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.650+0000 7fd91d612700 1 --2- 172.21.15.114:0/767367413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd90406c4e0 0x7fd90406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:30.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.651+0000 7fd91d612700 1 --2- 172.21.15.114:0/767367413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd90406c4e0 0x7fd90406e9a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd908009c60 tx=0x7fd908010040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:30.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.655+0000 7fd90effd700 1 -- 172.21.15.114:0/767367413 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd91405aed0 con 0x7fd9180ffad0 2024-02-19T19:01:30.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.978+0000 7fd920077700 1 -- 172.21.15.114:0/767367413 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd918069860 con 0x7fd9180ffad0 2024-02-19T19:01:30.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.980+0000 7fd90effd700 1 -- 172.21.15.114:0/767367413 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7fd914003e60 con 0x7fd9180ffad0 2024-02-19T19:01:30.999 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:01:30.999 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-02-19T19:01:30.999 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:01:30.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:01:30.999 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-02-19T19:01:30.999 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:01:31.000 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:01:31.000 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-02-19T19:01:31.000 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-02-19T19:01:31.000 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:01:31.000 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-02-19T19:01:31.000 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:01:31.000 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:01:31.000 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-02-19T19:01:31.000 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-02-19T19:01:31.000 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-02-19T19:01:31.000 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:01:31.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.983+0000 7fd920077700 1 -- 172.21.15.114:0/767367413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd90406c4e0 msgr2=0x7fd90406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:31.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.983+0000 7fd920077700 1 --2- 172.21.15.114:0/767367413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd90406c4e0 0x7fd90406e9a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd908009c60 tx=0x7fd908010040 comp rx=0 tx=0).stop 2024-02-19T19:01:31.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.983+0000 7fd920077700 1 -- 172.21.15.114:0/767367413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9180ffad0 msgr2=0x7fd918195280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:31.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.983+0000 7fd920077700 1 --2- 172.21.15.114:0/767367413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9180ffad0 0x7fd918195280 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd91400fa00 tx=0x7fd91400fa30 comp rx=0 tx=0).stop 2024-02-19T19:01:31.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.983+0000 7fd920077700 1 -- 172.21.15.114:0/767367413 shutdown_connections 2024-02-19T19:01:31.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.983+0000 7fd920077700 1 --2- 172.21.15.114:0/767367413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fd90406c4e0 0x7fd90406e9a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.983+0000 7fd920077700 1 --2- 172.21.15.114:0/767367413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9180ffad0 0x7fd918195280 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.983+0000 7fd920077700 1 --2- 172.21.15.114:0/767367413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd918100cd0 0x7fd9181957c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.983+0000 7fd920077700 1 -- 172.21.15.114:0/767367413 >> 172.21.15.114:0/767367413 conn(0x7fd9180fb050 msgr2=0x7fd918103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:31.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.983+0000 7fd920077700 1 -- 172.21.15.114:0/767367413 shutdown_connections 2024-02-19T19:01:31.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:30.983+0000 7fd920077700 1 -- 172.21.15.114:0/767367413 wait complete. 2024-02-19T19:01:31.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.098+0000 7f386582c700 1 -- 172.21.15.114:0/680578820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3860103f70 msgr2=0x7f3860106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:31.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.098+0000 7f386582c700 1 --2- 172.21.15.114:0/680578820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3860103f70 0x7f3860106360 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3854009910 tx=0x7f3854009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:31.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.098+0000 7f386582c700 1 -- 172.21.15.114:0/680578820 shutdown_connections 2024-02-19T19:01:31.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.098+0000 7f386582c700 1 --2- 172.21.15.114:0/680578820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3860103f70 0x7f3860106360 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.098+0000 7f386582c700 1 --2- 172.21.15.114:0/680578820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3860101640 0x7f3860103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.098+0000 7f386582c700 1 -- 172.21.15.114:0/680578820 >> 172.21.15.114:0/680578820 conn(0x7f38600fb050 msgr2=0x7f38600fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:31.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.098+0000 7f386582c700 1 -- 172.21.15.114:0/680578820 shutdown_connections 2024-02-19T19:01:31.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.098+0000 7f386582c700 1 -- 172.21.15.114:0/680578820 wait complete. 2024-02-19T19:01:31.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.099+0000 7f386582c700 1 Processor -- start 2024-02-19T19:01:31.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.099+0000 7f386582c700 1 -- start start 2024-02-19T19:01:31.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.099+0000 7f386582c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3860101640 0x7f3860100fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:31.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.099+0000 7f386582c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3860103f70 0x7f38600ff5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.099+0000 7f386582c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38600ffb30 con 0x7f3860101640 2024-02-19T19:01:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.099+0000 7f386582c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38600ffc70 con 0x7f3860103f70 2024-02-19T19:01:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.099+0000 7f385effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3860101640 0x7f3860100fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.099+0000 7f385effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3860101640 0x7f3860100fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54392/0 (socket says 172.21.15.114:54392) 2024-02-19T19:01:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.099+0000 7f385effd700 1 -- 172.21.15.114:0/4266493839 learned_addr learned my addr 172.21.15.114:0/4266493839 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:01:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.099+0000 7f385e7fc700 1 --2- 172.21.15.114:0/4266493839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3860103f70 0x7f38600ff5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.099+0000 7f385effd700 1 -- 172.21.15.114:0/4266493839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3860103f70 msgr2=0x7f38600ff5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.099+0000 7f385effd700 1 --2- 172.21.15.114:0/4266493839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3860103f70 0x7f38600ff5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.099+0000 7f385effd700 1 -- 172.21.15.114:0/4266493839 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38540095f0 con 0x7f3860101640 2024-02-19T19:01:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.099+0000 7f385e7fc700 1 --2- 172.21.15.114:0/4266493839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3860103f70 0x7f38600ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:01:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.100+0000 7f385effd700 1 --2- 172.21.15.114:0/4266493839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3860101640 0x7f3860100fa0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f385000c8f0 tx=0x7f385000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.100+0000 7f386482a700 1 -- 172.21.15.114:0/4266493839 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3850007a70 con 0x7f3860101640 2024-02-19T19:01:31.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.100+0000 7f386582c700 1 -- 172.21.15.114:0/4266493839 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38600fff50 con 0x7f3860101640 2024-02-19T19:01:31.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.100+0000 7f386482a700 1 -- 172.21.15.114:0/4266493839 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3850004ce0 con 0x7f3860101640 2024-02-19T19:01:31.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.100+0000 7f386482a700 1 -- 172.21.15.114:0/4266493839 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3850005610 con 0x7f3860101640 2024-02-19T19:01:31.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.100+0000 7f386582c700 1 -- 172.21.15.114:0/4266493839 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f386019e3c0 con 0x7f3860101640 2024-02-19T19:01:31.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.101+0000 7f386582c700 1 -- 172.21.15.114:0/4266493839 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3860190540 con 0x7f3860101640 2024-02-19T19:01:31.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.101+0000 7f386482a700 1 -- 172.21.15.114:0/4266493839 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3850004750 con 0x7f3860101640 2024-02-19T19:01:31.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.102+0000 7f386482a700 1 --2- 172.21.15.114:0/4266493839 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f384806c4e0 0x7f384806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:31.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.102+0000 7f386482a700 1 -- 172.21.15.114:0/4266493839 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f3850013070 con 0x7f3860101640 2024-02-19T19:01:31.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.102+0000 7f385e7fc700 1 --2- 172.21.15.114:0/4266493839 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f384806c4e0 0x7f384806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:31.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.103+0000 7f385e7fc700 1 --2- 172.21.15.114:0/4266493839 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f384806c4e0 0x7f384806e9a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f38540059d0 tx=0x7f3854005920 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:31.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.107+0000 7f386482a700 1 -- 172.21.15.114:0/4266493839 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3850059900 con 0x7f3860101640 2024-02-19T19:01:31.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:30 smithi114 ceph-mon[29441]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:31.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:30 smithi174 ceph-mon[41061]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:31.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.396+0000 7f386582c700 1 -- 172.21.15.114:0/4266493839 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f38600619a0 con 0x7f3860101640 2024-02-19T19:01:31.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.396+0000 7f386482a700 1 -- 172.21.15.114:0/4266493839 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1850 (secure 0 0 0) 0x7f3850059490 con 0x7f3860101640 2024-02-19T19:01:31.415 INFO:teuthology.orchestra.run.smithi114.stdout:e18 2024-02-19T19:01:31.415 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:01:31.415 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:01:31.416 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-02-19T19:01:31.416 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:01:31.416 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:01:31.416 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-02-19T19:01:31.416 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 18 2024-02-19T19:01:31.416 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:01:31.416 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:01:31.416 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-02-19T19:00:21.203922+0000 2024-02-19T19:01:31.416 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-02-19T19:01:31.416 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-02-19T19:01:31.416 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-02-19T19:01:31.416 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-02-19T19:01:31.416 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-02-19T19:01:31.416 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-02-19T19:01:31.417 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-02-19T19:01:31.417 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 47 2024-02-19T19:01:31.417 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:01:31.417 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 2 2024-02-19T19:01:31.417 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1 2024-02-19T19:01:31.417 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=14530,1=24269} 2024-02-19T19:01:31.417 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-02-19T19:01:31.417 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-02-19T19:01:31.417 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-02-19T19:01:31.417 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-02-19T19:01:31.417 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-02-19T19:01:31.417 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data enabled 2024-02-19T19:01:31.417 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-02-19T19:01:31.418 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-02-19T19:01:31.418 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 1 2024-02-19T19:01:31.418 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:01:31.418 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.vpoeop{0:14540} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:01:31.418 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.znhuvd{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:01:31.418 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.stnfnf{1:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:01:31.418 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:01:31.418 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:01:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.399+0000 7f386582c700 1 -- 172.21.15.114:0/4266493839 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f384806c4e0 msgr2=0x7f384806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.399+0000 7f386582c700 1 --2- 172.21.15.114:0/4266493839 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f384806c4e0 0x7f384806e9a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f38540059d0 tx=0x7f3854005920 comp rx=0 tx=0).stop 2024-02-19T19:01:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.399+0000 7f386582c700 1 -- 172.21.15.114:0/4266493839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3860101640 msgr2=0x7f3860100fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.399+0000 7f386582c700 1 --2- 172.21.15.114:0/4266493839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3860101640 0x7f3860100fa0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f385000c8f0 tx=0x7f385000ccb0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.399+0000 7f386582c700 1 -- 172.21.15.114:0/4266493839 shutdown_connections 2024-02-19T19:01:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.399+0000 7f386582c700 1 --2- 172.21.15.114:0/4266493839 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f384806c4e0 0x7f384806e9a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.399+0000 7f386582c700 1 --2- 172.21.15.114:0/4266493839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3860101640 0x7f3860100fa0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.399+0000 7f386582c700 1 --2- 172.21.15.114:0/4266493839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3860103f70 0x7f38600ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.399+0000 7f386582c700 1 -- 172.21.15.114:0/4266493839 >> 172.21.15.114:0/4266493839 conn(0x7f38600fb050 msgr2=0x7f38600fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.399+0000 7f386582c700 1 -- 172.21.15.114:0/4266493839 shutdown_connections 2024-02-19T19:01:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.399+0000 7f386582c700 1 -- 172.21.15.114:0/4266493839 wait complete. 2024-02-19T19:01:31.420 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 18 2024-02-19T19:01:31.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.507+0000 7f9666b52700 1 -- 172.21.15.114:0/4013697668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9660100cb0 msgr2=0x7f9660101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:31.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.507+0000 7f9666b52700 1 --2- 172.21.15.114:0/4013697668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9660100cb0 0x7f9660101110 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f964c009910 tx=0x7f964c009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:31.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.508+0000 7f9666b52700 1 -- 172.21.15.114:0/4013697668 shutdown_connections 2024-02-19T19:01:31.528 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.508+0000 7f9666b52700 1 --2- 172.21.15.114:0/4013697668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9660100cb0 0x7f9660101110 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.508+0000 7f9666b52700 1 --2- 172.21.15.114:0/4013697668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96600ffab0 0x7f96600ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.508+0000 7f9666b52700 1 -- 172.21.15.114:0/4013697668 >> 172.21.15.114:0/4013697668 conn(0x7f96600fb010 msgr2=0x7f96600fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:31.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.508+0000 7f9666b52700 1 -- 172.21.15.114:0/4013697668 shutdown_connections 2024-02-19T19:01:31.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.508+0000 7f9666b52700 1 -- 172.21.15.114:0/4013697668 wait complete. 2024-02-19T19:01:31.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.508+0000 7f9666b52700 1 Processor -- start 2024-02-19T19:01:31.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.508+0000 7f9666b52700 1 -- start start 2024-02-19T19:01:31.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.508+0000 7f9666b52700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96600ffab0 0x7f9660069950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:31.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.508+0000 7f9666b52700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9660100cb0 0x7f966019b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:31.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.508+0000 7f9666b52700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966019b690 con 0x7f9660100cb0 2024-02-19T19:01:31.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.508+0000 7f9666b52700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966019b7d0 con 0x7f96600ffab0 2024-02-19T19:01:31.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.508+0000 7f965ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9660100cb0 0x7f966019b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:31.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.508+0000 7f965ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9660100cb0 0x7f966019b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54418/0 (socket says 172.21.15.114:54418) 2024-02-19T19:01:31.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.508+0000 7f965ffff700 1 -- 172.21.15.114:0/2767062044 learned_addr learned my addr 172.21.15.114:0/2767062044 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:01:31.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.509+0000 7f96648ee700 1 --2- 172.21.15.114:0/2767062044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96600ffab0 0x7f9660069950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:31.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.509+0000 7f965ffff700 1 -- 172.21.15.114:0/2767062044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96600ffab0 msgr2=0x7f9660069950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:31.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.509+0000 7f965ffff700 1 --2- 172.21.15.114:0/2767062044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96600ffab0 0x7f9660069950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.509+0000 7f965ffff700 1 -- 172.21.15.114:0/2767062044 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c0095f0 con 0x7f9660100cb0 2024-02-19T19:01:31.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.509+0000 7f965ffff700 1 --2- 172.21.15.114:0/2767062044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9660100cb0 0x7f966019b0c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f964c00b5c0 tx=0x7f964c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:31.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.509+0000 7f965dffb700 1 -- 172.21.15.114:0/2767062044 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f964c01d070 con 0x7f9660100cb0 2024-02-19T19:01:31.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.509+0000 7f965dffb700 1 -- 172.21.15.114:0/2767062044 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f964c00f460 con 0x7f9660100cb0 2024-02-19T19:01:31.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.509+0000 7f9666b52700 1 -- 172.21.15.114:0/2767062044 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f966019ba50 con 0x7f9660100cb0 2024-02-19T19:01:31.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.509+0000 7f965dffb700 1 -- 172.21.15.114:0/2767062044 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f964c021620 con 0x7f9660100cb0 2024-02-19T19:01:31.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.509+0000 7f9666b52700 1 -- 172.21.15.114:0/2767062044 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f966019bf40 con 0x7f9660100cb0 2024-02-19T19:01:31.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.510+0000 7f9666b52700 1 -- 172.21.15.114:0/2767062044 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9660108610 con 0x7f9660100cb0 2024-02-19T19:01:31.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.511+0000 7f965dffb700 1 -- 172.21.15.114:0/2767062044 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f964c00f5d0 con 0x7f9660100cb0 2024-02-19T19:01:31.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.511+0000 7f965dffb700 1 --2- 172.21.15.114:0/2767062044 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f965006c4e0 0x7f965006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:31.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.511+0000 7f965dffb700 1 -- 172.21.15.114:0/2767062044 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f964c08cc20 con 0x7f9660100cb0 2024-02-19T19:01:31.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.511+0000 7f96648ee700 1 --2- 172.21.15.114:0/2767062044 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f965006c4e0 0x7f965006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:31.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.512+0000 7f96648ee700 1 --2- 172.21.15.114:0/2767062044 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f965006c4e0 0x7f965006e9a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9654005950 tx=0x7f96540058e0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:31.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.516+0000 7f965dffb700 1 -- 172.21.15.114:0/2767062044 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f964c05ae30 con 0x7f9660100cb0 2024-02-19T19:01:31.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.724+0000 7f9666b52700 1 -- 172.21.15.114:0/2767062044 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9660105600 con 0x7f965006c4e0 2024-02-19T19:01:31.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.726+0000 7f965dffb700 1 -- 172.21.15.114:0/2767062044 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f9660105600 con 0x7f965006c4e0 2024-02-19T19:01:31.745 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:01:31.745 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", 2024-02-19T19:01:31.745 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-02-19T19:01:31.745 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:01:31.745 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [], 2024-02-19T19:01:31.745 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "", 2024-02-19T19:01:31.745 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 image", 2024-02-19T19:01:31.745 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-02-19T19:01:31.745 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:01:31.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.728+0000 7f9666b52700 1 -- 172.21.15.114:0/2767062044 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f965006c4e0 msgr2=0x7f965006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:31.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.728+0000 7f9666b52700 1 --2- 172.21.15.114:0/2767062044 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f965006c4e0 0x7f965006e9a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9654005950 tx=0x7f96540058e0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.728+0000 7f9666b52700 1 -- 172.21.15.114:0/2767062044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9660100cb0 msgr2=0x7f966019b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:31.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.728+0000 7f9666b52700 1 --2- 172.21.15.114:0/2767062044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9660100cb0 0x7f966019b0c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f964c00b5c0 tx=0x7f964c00be20 comp rx=0 tx=0).stop 2024-02-19T19:01:31.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.728+0000 7f9666b52700 1 -- 172.21.15.114:0/2767062044 shutdown_connections 2024-02-19T19:01:31.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.728+0000 7f9666b52700 1 --2- 172.21.15.114:0/2767062044 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f965006c4e0 0x7f965006e9a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.728+0000 7f9666b52700 1 --2- 172.21.15.114:0/2767062044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96600ffab0 0x7f9660069950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.728+0000 7f9666b52700 1 --2- 172.21.15.114:0/2767062044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9660100cb0 0x7f966019b0c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.728+0000 7f9666b52700 1 -- 172.21.15.114:0/2767062044 >> 172.21.15.114:0/2767062044 conn(0x7f96600fb010 msgr2=0x7f9660103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:31.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.728+0000 7f9666b52700 1 -- 172.21.15.114:0/2767062044 shutdown_connections 2024-02-19T19:01:31.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.728+0000 7f9666b52700 1 -- 172.21.15.114:0/2767062044 wait complete. 2024-02-19T19:01:31.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.833+0000 7ff505ddb700 1 -- 172.21.15.114:0/47091793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff500101eb0 msgr2=0x7ff500102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:31.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.833+0000 7ff505ddb700 1 --2- 172.21.15.114:0/47091793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff500101eb0 0x7ff500102310 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0009910 tx=0x7ff4f0009c20 comp rx=0 tx=0).stop 2024-02-19T19:01:31.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.833+0000 7ff505ddb700 1 -- 172.21.15.114:0/47091793 shutdown_connections 2024-02-19T19:01:31.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.833+0000 7ff505ddb700 1 --2- 172.21.15.114:0/47091793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff500101eb0 0x7ff500102310 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.833+0000 7ff505ddb700 1 --2- 172.21.15.114:0/47091793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5000feed0 0x7ff5000ff2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.833+0000 7ff505ddb700 1 -- 172.21.15.114:0/47091793 >> 172.21.15.114:0/47091793 conn(0x7ff5000fa810 msgr2=0x7ff5000fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:31.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.833+0000 7ff505ddb700 1 -- 172.21.15.114:0/47091793 shutdown_connections 2024-02-19T19:01:31.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.833+0000 7ff505ddb700 1 -- 172.21.15.114:0/47091793 wait complete. 2024-02-19T19:01:31.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.834+0000 7ff505ddb700 1 Processor -- start 2024-02-19T19:01:31.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.834+0000 7ff505ddb700 1 -- start start 2024-02-19T19:01:31.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.834+0000 7ff505ddb700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5000feed0 0x7ff5001963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:31.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.834+0000 7ff505ddb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff500101eb0 0x7ff500196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:31.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.834+0000 7ff505ddb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff500196f40 con 0x7ff500101eb0 2024-02-19T19:01:31.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.834+0000 7ff505ddb700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff500197080 con 0x7ff5000feed0 2024-02-19T19:01:31.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.834+0000 7ff4feffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff500101eb0 0x7ff500196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:31.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.834+0000 7ff4feffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff500101eb0 0x7ff500196920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54438/0 (socket says 172.21.15.114:54438) 2024-02-19T19:01:31.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.834+0000 7ff4feffd700 1 -- 172.21.15.114:0/3435195645 learned_addr learned my addr 172.21.15.114:0/3435195645 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:01:31.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.834+0000 7ff4ff7fe700 1 --2- 172.21.15.114:0/3435195645 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5000feed0 0x7ff5001963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:31.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.834+0000 7ff4feffd700 1 -- 172.21.15.114:0/3435195645 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5000feed0 msgr2=0x7ff5001963e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:31.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.834+0000 7ff4feffd700 1 --2- 172.21.15.114:0/3435195645 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5000feed0 0x7ff5001963e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:31.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.834+0000 7ff4feffd700 1 -- 172.21.15.114:0/3435195645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f00095f0 con 0x7ff500101eb0 2024-02-19T19:01:31.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.835+0000 7ff4feffd700 1 --2- 172.21.15.114:0/3435195645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff500101eb0 0x7ff500196920 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0006010 tx=0x7ff4f000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:31.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.835+0000 7ff4fcff9700 1 -- 172.21.15.114:0/3435195645 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4f001d070 con 0x7ff500101eb0 2024-02-19T19:01:31.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.835+0000 7ff4fcff9700 1 -- 172.21.15.114:0/3435195645 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff4f000f460 con 0x7ff500101eb0 2024-02-19T19:01:31.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.835+0000 7ff4fcff9700 1 -- 172.21.15.114:0/3435195645 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4f0017610 con 0x7ff500101eb0 2024-02-19T19:01:31.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.835+0000 7ff505ddb700 1 -- 172.21.15.114:0/3435195645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50019bad0 con 0x7ff500101eb0 2024-02-19T19:01:31.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.835+0000 7ff505ddb700 1 -- 172.21.15.114:0/3435195645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff50019bfc0 con 0x7ff500101eb0 2024-02-19T19:01:31.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.836+0000 7ff505ddb700 1 -- 172.21.15.114:0/3435195645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff500109770 con 0x7ff500101eb0 2024-02-19T19:01:31.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.836+0000 7ff4fcff9700 1 -- 172.21.15.114:0/3435195645 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ff4f000f5d0 con 0x7ff500101eb0 2024-02-19T19:01:31.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.837+0000 7ff4fcff9700 1 --2- 172.21.15.114:0/3435195645 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff4e806c4e0 0x7ff4e806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:31.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.837+0000 7ff4ff7fe700 1 --2- 172.21.15.114:0/3435195645 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff4e806c4e0 0x7ff4e806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:31.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.837+0000 7ff4fcff9700 1 -- 172.21.15.114:0/3435195645 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff4f008ccd0 con 0x7ff500101eb0 2024-02-19T19:01:31.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.837+0000 7ff4ff7fe700 1 --2- 172.21.15.114:0/3435195645 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff4e806c4e0 0x7ff4e806e9a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff500068bb0 tx=0x7ff4f4007400 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:31.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:31.841+0000 7ff4fcff9700 1 -- 172.21.15.114:0/3435195645 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff4f005aee0 con 0x7ff500101eb0 2024-02-19T19:01:32.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:32.117+0000 7ff505ddb700 1 -- 172.21.15.114:0/3435195645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff500109930 con 0x7ff500101eb0 2024-02-19T19:01:32.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:32.117+0000 7ff4fcff9700 1 -- 172.21.15.114:0/3435195645 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7ff4f005aa70 con 0x7ff500101eb0 2024-02-19T19:01:32.136 INFO:teuthology.orchestra.run.smithi114.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:01:32.136 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:01:32.136 INFO:teuthology.orchestra.run.smithi114.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:01:32.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:32.119+0000 7ff505ddb700 1 -- 172.21.15.114:0/3435195645 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff4e806c4e0 msgr2=0x7ff4e806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:32.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:32.119+0000 7ff505ddb700 1 --2- 172.21.15.114:0/3435195645 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff4e806c4e0 0x7ff4e806e9a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff500068bb0 tx=0x7ff4f4007400 comp rx=0 tx=0).stop 2024-02-19T19:01:32.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:32.119+0000 7ff505ddb700 1 -- 172.21.15.114:0/3435195645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff500101eb0 msgr2=0x7ff500196920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:32.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:32.119+0000 7ff505ddb700 1 --2- 172.21.15.114:0/3435195645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff500101eb0 0x7ff500196920 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0006010 tx=0x7ff4f000bd70 comp rx=0 tx=0).stop 2024-02-19T19:01:32.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:32.119+0000 7ff505ddb700 1 -- 172.21.15.114:0/3435195645 shutdown_connections 2024-02-19T19:01:32.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:32.119+0000 7ff505ddb700 1 --2- 172.21.15.114:0/3435195645 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7ff4e806c4e0 0x7ff4e806e9a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:32.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:32.119+0000 7ff505ddb700 1 --2- 172.21.15.114:0/3435195645 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5000feed0 0x7ff5001963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:32.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:32.119+0000 7ff505ddb700 1 --2- 172.21.15.114:0/3435195645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff500101eb0 0x7ff500196920 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:32.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:32.119+0000 7ff505ddb700 1 -- 172.21.15.114:0/3435195645 >> 172.21.15.114:0/3435195645 conn(0x7ff5000fa810 msgr2=0x7ff5001050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:32.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:32.119+0000 7ff505ddb700 1 -- 172.21.15.114:0/3435195645 shutdown_connections 2024-02-19T19:01:32.139 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:01:32.119+0000 7ff505ddb700 1 -- 172.21.15.114:0/3435195645 wait complete. 2024-02-19T19:01:32.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:31 smithi114 ceph-mon[29441]: from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:32.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:31 smithi114 ceph-mon[29441]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:32.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:31 smithi114 ceph-mon[29441]: pgmap v229: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 4 op/s 2024-02-19T19:01:32.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:31 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/767367413' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:01:32.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:31 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/4266493839' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:01:32.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:31 smithi114 ceph-mon[29441]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:32.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:31 smithi174 ceph-mon[41061]: from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:32.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:31 smithi174 ceph-mon[41061]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:32.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:31 smithi174 ceph-mon[41061]: pgmap v229: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 4 op/s 2024-02-19T19:01:32.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:31 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/767367413' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:01:32.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:31 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/4266493839' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:01:32.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:31 smithi174 ceph-mon[41061]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:32 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/3435195645' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:01:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:32 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:33.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:32 smithi114 ceph-mon[29441]: pgmap v230: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 3 op/s 2024-02-19T19:01:33.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:32 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/3435195645' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:01:33.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:32 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:33.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:32 smithi174 ceph-mon[41061]: pgmap v230: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 3 op/s 2024-02-19T19:01:36.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:35 smithi114 ceph-mon[29441]: pgmap v231: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-02-19T19:01:36.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:35 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:01:36.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:35 smithi114 ceph-mon[29441]: Upgrade: Target is version 19.0.0-1450-gf78a58c0 (unknown) 2024-02-19T19:01:36.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:35 smithi114 ceph-mon[29441]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, digests ['quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526'] 2024-02-19T19:01:36.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:35 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:01:36.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:35 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:01:36.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:35 smithi114 ceph-mon[29441]: Upgrade: Need to upgrade myself (mgr.smithi114.pcvtrk) 2024-02-19T19:01:36.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:35 smithi174 ceph-mon[41061]: pgmap v231: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-02-19T19:01:36.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:35 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:01:36.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:35 smithi174 ceph-mon[41061]: Upgrade: Target is version 19.0.0-1450-gf78a58c0 (unknown) 2024-02-19T19:01:36.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:35 smithi174 ceph-mon[41061]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, digests ['quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526'] 2024-02-19T19:01:36.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:35 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:01:36.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:35 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:01:36.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:35 smithi174 ceph-mon[41061]: Upgrade: Need to upgrade myself (mgr.smithi114.pcvtrk) 2024-02-19T19:01:37.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:36 smithi114 ceph-mon[29441]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 on smithi174 2024-02-19T19:01:37.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:36 smithi174 ceph-mon[41061]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 on smithi174 2024-02-19T19:01:38.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:37 smithi114 ceph-mon[29441]: pgmap v232: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:01:38.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:37 smithi174 ceph-mon[41061]: pgmap v232: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:01:40.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:39 smithi174 ceph-mon[41061]: pgmap v233: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:01:40.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:39 smithi114 ceph-mon[29441]: pgmap v233: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:01:42.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:41 smithi114 ceph-mon[29441]: pgmap v234: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:01:42.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:41 smithi174 ceph-mon[41061]: pgmap v234: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:01:44.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:43 smithi114 ceph-mon[29441]: pgmap v235: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-19T19:01:44.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:43 smithi174 ceph-mon[41061]: pgmap v235: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-19T19:01:46.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:45 smithi114 ceph-mon[29441]: pgmap v236: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-19T19:01:46.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:45 smithi174 ceph-mon[41061]: pgmap v236: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-19T19:01:48.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:47 smithi114 ceph-mon[29441]: pgmap v237: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:01:48.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:47 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:48.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:47 smithi174 ceph-mon[41061]: pgmap v237: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:01:48.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:47 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:49.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:48 smithi114 ceph-mon[29441]: pgmap v238: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-02-19T19:01:49.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:48 smithi174 ceph-mon[41061]: pgmap v238: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-02-19T19:01:52.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:51 smithi114 ceph-mon[29441]: pgmap v239: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-02-19T19:01:52.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:51 smithi174 ceph-mon[41061]: pgmap v239: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-02-19T19:01:54.060 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:53 smithi114 ceph-mon[29441]: pgmap v240: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-02-19T19:01:54.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:53 smithi174 ceph-mon[41061]: pgmap v240: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-02-19T19:01:56.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:55 smithi114 ceph-mon[29441]: pgmap v241: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-02-19T19:01:56.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:55 smithi174 ceph-mon[41061]: pgmap v241: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-02-19T19:01:57.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:56 smithi114 ceph-mon[29441]: pgmap v242: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-02-19T19:01:57.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:56 smithi174 ceph-mon[41061]: pgmap v242: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-02-19T19:02:00.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:01:59 smithi114 ceph-mon[29441]: pgmap v243: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-02-19T19:02:00.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:01:59 smithi174 ceph-mon[41061]: pgmap v243: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-02-19T19:02:02.252 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:01 smithi114 ceph-mon[29441]: pgmap v244: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:02.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.232+0000 7f85333a1700 1 -- 172.21.15.114:0/2977680324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f852c100bc0 msgr2=0x7f852c100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:02.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.232+0000 7f85333a1700 1 --2- 172.21.15.114:0/2977680324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f852c100bc0 0x7f852c100fe0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f852400b0a0 tx=0x7f852400b3b0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.232+0000 7f85333a1700 1 -- 172.21.15.114:0/2977680324 shutdown_connections 2024-02-19T19:02:02.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.232+0000 7f85333a1700 1 --2- 172.21.15.114:0/2977680324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f852c101dc0 0x7f852c102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.232+0000 7f85333a1700 1 --2- 172.21.15.114:0/2977680324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f852c100bc0 0x7f852c100fe0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.232+0000 7f85333a1700 1 -- 172.21.15.114:0/2977680324 >> 172.21.15.114:0/2977680324 conn(0x7f852c0fc140 msgr2=0x7f852c0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:02.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.232+0000 7f85333a1700 1 -- 172.21.15.114:0/2977680324 shutdown_connections 2024-02-19T19:02:02.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.232+0000 7f85333a1700 1 -- 172.21.15.114:0/2977680324 wait complete. 2024-02-19T19:02:02.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.233+0000 7f85333a1700 1 Processor -- start 2024-02-19T19:02:02.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.233+0000 7f85333a1700 1 -- start start 2024-02-19T19:02:02.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.233+0000 7f85333a1700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f852c100bc0 0x7f852c196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:02.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.233+0000 7f85333a1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f852c101dc0 0x7f852c1968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:02.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.233+0000 7f85333a1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c196ef0 con 0x7f852c101dc0 2024-02-19T19:02:02.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.233+0000 7f85333a1700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c197030 con 0x7f852c100bc0 2024-02-19T19:02:02.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.233+0000 7f853093c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f852c101dc0 0x7f852c1968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:02.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.233+0000 7f853093c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f852c101dc0 0x7f852c1968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52394/0 (socket says 172.21.15.114:52394) 2024-02-19T19:02:02.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.233+0000 7f853093c700 1 -- 172.21.15.114:0/2262970413 learned_addr learned my addr 172.21.15.114:0/2262970413 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:02:02.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.233+0000 7f853113d700 1 --2- 172.21.15.114:0/2262970413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f852c100bc0 0x7f852c196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:02.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.234+0000 7f853093c700 1 -- 172.21.15.114:0/2262970413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f852c100bc0 msgr2=0x7f852c196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:02.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.234+0000 7f853093c700 1 --2- 172.21.15.114:0/2262970413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f852c100bc0 0x7f852c196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.234+0000 7f853093c700 1 -- 172.21.15.114:0/2262970413 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8524009d00 con 0x7f852c101dc0 2024-02-19T19:02:02.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.234+0000 7f853093c700 1 --2- 172.21.15.114:0/2262970413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f852c101dc0 0x7f852c1968d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f852000d900 tx=0x7f852000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:02.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.234+0000 7f852a7fc700 1 -- 172.21.15.114:0/2262970413 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85200041f0 con 0x7f852c101dc0 2024-02-19T19:02:02.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.234+0000 7f852a7fc700 1 -- 172.21.15.114:0/2262970413 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8520004ce0 con 0x7f852c101dc0 2024-02-19T19:02:02.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.234+0000 7f85333a1700 1 -- 172.21.15.114:0/2262970413 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f852c19bae0 con 0x7f852c101dc0 2024-02-19T19:02:02.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.234+0000 7f852a7fc700 1 -- 172.21.15.114:0/2262970413 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f852000b750 con 0x7f852c101dc0 2024-02-19T19:02:02.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.234+0000 7f85333a1700 1 -- 172.21.15.114:0/2262970413 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f852c19c030 con 0x7f852c101dc0 2024-02-19T19:02:02.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.235+0000 7f85333a1700 1 -- 172.21.15.114:0/2262970413 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f852c109720 con 0x7f852c101dc0 2024-02-19T19:02:02.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.235+0000 7f852a7fc700 1 -- 172.21.15.114:0/2262970413 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f8520004e50 con 0x7f852c101dc0 2024-02-19T19:02:02.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.236+0000 7f852a7fc700 1 --2- 172.21.15.114:0/2262970413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f851406c4e0 0x7f851406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:02.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.236+0000 7f852a7fc700 1 -- 172.21.15.114:0/2262970413 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f852008b810 con 0x7f852c101dc0 2024-02-19T19:02:02.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.236+0000 7f853113d700 1 --2- 172.21.15.114:0/2262970413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f851406c4e0 0x7f851406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:02.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.236+0000 7f853113d700 1 --2- 172.21.15.114:0/2262970413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f851406c4e0 0x7f851406e9a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f852400b830 tx=0x7f85240060c0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:02.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.241+0000 7f852a7fc700 1 -- 172.21.15.114:0/2262970413 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8520059a20 con 0x7f852c101dc0 2024-02-19T19:02:02.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:01 smithi174 ceph-mon[41061]: pgmap v244: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:02.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.447+0000 7f85333a1700 1 -- 172.21.15.114:0/2262970413 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f852c106710 con 0x7f851406c4e0 2024-02-19T19:02:02.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.448+0000 7f852a7fc700 1 -- 172.21.15.114:0/2262970413 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f852c106710 con 0x7f851406c4e0 2024-02-19T19:02:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.450+0000 7f85333a1700 1 -- 172.21.15.114:0/2262970413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f851406c4e0 msgr2=0x7f851406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.450+0000 7f85333a1700 1 --2- 172.21.15.114:0/2262970413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f851406c4e0 0x7f851406e9a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f852400b830 tx=0x7f85240060c0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.450+0000 7f85333a1700 1 -- 172.21.15.114:0/2262970413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f852c101dc0 msgr2=0x7f852c1968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.450+0000 7f85333a1700 1 --2- 172.21.15.114:0/2262970413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f852c101dc0 0x7f852c1968d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f852000d900 tx=0x7f852000dc10 comp rx=0 tx=0).stop 2024-02-19T19:02:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.450+0000 7f85333a1700 1 -- 172.21.15.114:0/2262970413 shutdown_connections 2024-02-19T19:02:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.450+0000 7f85333a1700 1 --2- 172.21.15.114:0/2262970413 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f851406c4e0 0x7f851406e9a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.450+0000 7f85333a1700 1 --2- 172.21.15.114:0/2262970413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f852c100bc0 0x7f852c196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.450+0000 7f85333a1700 1 --2- 172.21.15.114:0/2262970413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f852c101dc0 0x7f852c1968d0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.450+0000 7f85333a1700 1 -- 172.21.15.114:0/2262970413 >> 172.21.15.114:0/2262970413 conn(0x7f852c0fc140 msgr2=0x7f852c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:02.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.451+0000 7f85333a1700 1 -- 172.21.15.114:0/2262970413 shutdown_connections 2024-02-19T19:02:02.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.451+0000 7f85333a1700 1 -- 172.21.15.114:0/2262970413 wait complete. 2024-02-19T19:02:02.479 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-02-19T19:02:02.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.555+0000 7f000bbc9700 1 -- 172.21.15.114:0/2409426106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00040ffd10 msgr2=0x7f0004100190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:02.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.555+0000 7f000bbc9700 1 --2- 172.21.15.114:0/2409426106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00040ffd10 0x7f0004100190 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7efff4009910 tx=0x7efff4009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:02.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.555+0000 7f000bbc9700 1 -- 172.21.15.114:0/2409426106 shutdown_connections 2024-02-19T19:02:02.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.555+0000 7f000bbc9700 1 --2- 172.21.15.114:0/2409426106 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00040ffd10 0x7f0004100190 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.555+0000 7f000bbc9700 1 --2- 172.21.15.114:0/2409426106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0004101140 0x7f0004101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.555+0000 7f000bbc9700 1 -- 172.21.15.114:0/2409426106 >> 172.21.15.114:0/2409426106 conn(0x7f00040fa7c0 msgr2=0x7f00040fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:02.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.555+0000 7f000bbc9700 1 -- 172.21.15.114:0/2409426106 shutdown_connections 2024-02-19T19:02:02.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.555+0000 7f000bbc9700 1 -- 172.21.15.114:0/2409426106 wait complete. 2024-02-19T19:02:02.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.555+0000 7f000bbc9700 1 Processor -- start 2024-02-19T19:02:02.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.555+0000 7f000bbc9700 1 -- start start 2024-02-19T19:02:02.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.556+0000 7f000bbc9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00040ffd10 0x7f0004196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:02.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.556+0000 7f000bbc9700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0004101140 0x7f00041968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:02.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.556+0000 7f000bbc9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0004196ee0 con 0x7f00040ffd10 2024-02-19T19:02:02.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.556+0000 7f000bbc9700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0004197020 con 0x7f0004101140 2024-02-19T19:02:02.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.556+0000 7f0009965700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00040ffd10 0x7f0004196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:02.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.556+0000 7f0009965700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00040ffd10 0x7f0004196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52404/0 (socket says 172.21.15.114:52404) 2024-02-19T19:02:02.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.556+0000 7f0009965700 1 -- 172.21.15.114:0/2920243911 learned_addr learned my addr 172.21.15.114:0/2920243911 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:02:02.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.556+0000 7f0009164700 1 --2- 172.21.15.114:0/2920243911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0004101140 0x7f00041968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:02.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.556+0000 7f0009965700 1 -- 172.21.15.114:0/2920243911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0004101140 msgr2=0x7f00041968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:02.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.556+0000 7f0009965700 1 --2- 172.21.15.114:0/2920243911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0004101140 0x7f00041968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.556+0000 7f0009965700 1 -- 172.21.15.114:0/2920243911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff40095f0 con 0x7f00040ffd10 2024-02-19T19:02:02.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.556+0000 7f0009965700 1 --2- 172.21.15.114:0/2920243911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00040ffd10 0x7f0004196380 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f000000d8d0 tx=0x7f000000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:02.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.556+0000 7efffaffd700 1 -- 172.21.15.114:0/2920243911 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00000098a0 con 0x7f00040ffd10 2024-02-19T19:02:02.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.556+0000 7f000bbc9700 1 -- 172.21.15.114:0/2920243911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f000419bad0 con 0x7f00040ffd10 2024-02-19T19:02:02.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.556+0000 7efffaffd700 1 -- 172.21.15.114:0/2920243911 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0000010460 con 0x7f00040ffd10 2024-02-19T19:02:02.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.556+0000 7efffaffd700 1 -- 172.21.15.114:0/2920243911 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f000000f5f0 con 0x7f00040ffd10 2024-02-19T19:02:02.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.556+0000 7f000bbc9700 1 -- 172.21.15.114:0/2920243911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f000419c020 con 0x7f00040ffd10 2024-02-19T19:02:02.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.557+0000 7f000bbc9700 1 -- 172.21.15.114:0/2920243911 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00041086f0 con 0x7f00040ffd10 2024-02-19T19:02:02.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.558+0000 7efffaffd700 1 -- 172.21.15.114:0/2920243911 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f0000009a00 con 0x7f00040ffd10 2024-02-19T19:02:02.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.558+0000 7efffaffd700 1 --2- 172.21.15.114:0/2920243911 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7efff006c4e0 0x7efff006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:02.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.558+0000 7f0009164700 1 --2- 172.21.15.114:0/2920243911 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7efff006c4e0 0x7efff006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:02.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.558+0000 7efffaffd700 1 -- 172.21.15.114:0/2920243911 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f000008b690 con 0x7f00040ffd10 2024-02-19T19:02:02.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.559+0000 7f0009164700 1 --2- 172.21.15.114:0/2920243911 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7efff006c4e0 0x7efff006e9a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efff400b5c0 tx=0x7efff4005ee0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:02.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.563+0000 7efffaffd700 1 -- 172.21.15.114:0/2920243911 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0000059920 con 0x7f00040ffd10 2024-02-19T19:02:02.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.784+0000 7f000bbc9700 1 -- 172.21.15.114:0/2920243911 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f00040ff310 con 0x7efff006c4e0 2024-02-19T19:02:02.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.785+0000 7efffaffd700 1 -- 172.21.15.114:0/2920243911 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f00040ff310 con 0x7efff006c4e0 2024-02-19T19:02:02.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.787+0000 7f000bbc9700 1 -- 172.21.15.114:0/2920243911 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7efff006c4e0 msgr2=0x7efff006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:02.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.787+0000 7f000bbc9700 1 --2- 172.21.15.114:0/2920243911 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7efff006c4e0 0x7efff006e9a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efff400b5c0 tx=0x7efff4005ee0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.787+0000 7f000bbc9700 1 -- 172.21.15.114:0/2920243911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00040ffd10 msgr2=0x7f0004196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:02.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.787+0000 7f000bbc9700 1 --2- 172.21.15.114:0/2920243911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00040ffd10 0x7f0004196380 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f000000d8d0 tx=0x7f000000dbe0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.787+0000 7f000bbc9700 1 -- 172.21.15.114:0/2920243911 shutdown_connections 2024-02-19T19:02:02.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.787+0000 7f000bbc9700 1 --2- 172.21.15.114:0/2920243911 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7efff006c4e0 0x7efff006e9a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.787+0000 7f000bbc9700 1 --2- 172.21.15.114:0/2920243911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f00040ffd10 0x7f0004196380 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.787+0000 7f000bbc9700 1 --2- 172.21.15.114:0/2920243911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0004101140 0x7f00041968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.787+0000 7f000bbc9700 1 -- 172.21.15.114:0/2920243911 >> 172.21.15.114:0/2920243911 conn(0x7f00040fa7c0 msgr2=0x7f00040fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:02.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.787+0000 7f000bbc9700 1 -- 172.21.15.114:0/2920243911 shutdown_connections 2024-02-19T19:02:02.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.787+0000 7f000bbc9700 1 -- 172.21.15.114:0/2920243911 wait complete. 2024-02-19T19:02:02.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.891+0000 7f5e8f9ab700 1 -- 172.21.15.114:0/2802267744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e88101db0 msgr2=0x7f5e88102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:02.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.891+0000 7f5e8f9ab700 1 --2- 172.21.15.114:0/2802267744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e88101db0 0x7f5e88102210 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5e84009910 tx=0x7f5e84009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:02.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.891+0000 7f5e8f9ab700 1 -- 172.21.15.114:0/2802267744 shutdown_connections 2024-02-19T19:02:02.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.891+0000 7f5e8f9ab700 1 --2- 172.21.15.114:0/2802267744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e88101db0 0x7f5e88102210 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.891+0000 7f5e8f9ab700 1 --2- 172.21.15.114:0/2802267744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e88100bb0 0x7f5e88100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.891+0000 7f5e8f9ab700 1 -- 172.21.15.114:0/2802267744 >> 172.21.15.114:0/2802267744 conn(0x7f5e880fc110 msgr2=0x7f5e880fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:02.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.891+0000 7f5e8f9ab700 1 -- 172.21.15.114:0/2802267744 shutdown_connections 2024-02-19T19:02:02.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.891+0000 7f5e8f9ab700 1 -- 172.21.15.114:0/2802267744 wait complete. 2024-02-19T19:02:02.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.892+0000 7f5e8f9ab700 1 Processor -- start 2024-02-19T19:02:02.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.892+0000 7f5e8f9ab700 1 -- start start 2024-02-19T19:02:02.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.892+0000 7f5e8f9ab700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e88100bb0 0x7f5e88196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:02.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.892+0000 7f5e8f9ab700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e88101db0 0x7f5e881968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:02.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.892+0000 7f5e8f9ab700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e88196ec0 con 0x7f5e88101db0 2024-02-19T19:02:02.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.892+0000 7f5e8f9ab700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e88197000 con 0x7f5e88100bb0 2024-02-19T19:02:02.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.892+0000 7f5e8cf46700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e88101db0 0x7f5e881968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:02.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.892+0000 7f5e8d747700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e88100bb0 0x7f5e88196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:02.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.892+0000 7f5e8cf46700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e88101db0 0x7f5e881968a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52428/0 (socket says 172.21.15.114:52428) 2024-02-19T19:02:02.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.892+0000 7f5e8cf46700 1 -- 172.21.15.114:0/628421650 learned_addr learned my addr 172.21.15.114:0/628421650 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:02:02.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.892+0000 7f5e8cf46700 1 -- 172.21.15.114:0/628421650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e88100bb0 msgr2=0x7f5e88196360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:02.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.892+0000 7f5e8cf46700 1 --2- 172.21.15.114:0/628421650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e88100bb0 0x7f5e88196360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:02.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.892+0000 7f5e8cf46700 1 -- 172.21.15.114:0/628421650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e840095f0 con 0x7f5e88101db0 2024-02-19T19:02:02.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.893+0000 7f5e8d747700 1 --2- 172.21.15.114:0/628421650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e88100bb0 0x7f5e88196360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:02:02.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.893+0000 7f5e8cf46700 1 --2- 172.21.15.114:0/628421650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e88101db0 0x7f5e881968a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5e84000c00 tx=0x7f5e8400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:02.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.893+0000 7f5e7e7fc700 1 -- 172.21.15.114:0/628421650 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e8401d070 con 0x7f5e88101db0 2024-02-19T19:02:02.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.893+0000 7f5e8f9ab700 1 -- 172.21.15.114:0/628421650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e8819ba50 con 0x7f5e88101db0 2024-02-19T19:02:02.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.893+0000 7f5e7e7fc700 1 -- 172.21.15.114:0/628421650 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5e84005dc0 con 0x7f5e88101db0 2024-02-19T19:02:02.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.893+0000 7f5e7e7fc700 1 -- 172.21.15.114:0/628421650 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e840226a0 con 0x7f5e88101db0 2024-02-19T19:02:02.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.893+0000 7f5e8f9ab700 1 -- 172.21.15.114:0/628421650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8819bf40 con 0x7f5e88101db0 2024-02-19T19:02:02.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.894+0000 7f5e8f9ab700 1 -- 172.21.15.114:0/628421650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e88109710 con 0x7f5e88101db0 2024-02-19T19:02:02.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.894+0000 7f5e7e7fc700 1 -- 172.21.15.114:0/628421650 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f5e840038f0 con 0x7f5e88101db0 2024-02-19T19:02:02.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.895+0000 7f5e7e7fc700 1 --2- 172.21.15.114:0/628421650 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5e7406c4e0 0x7f5e7406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:02.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.895+0000 7f5e7e7fc700 1 -- 172.21.15.114:0/628421650 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f5e8408cdc0 con 0x7f5e88101db0 2024-02-19T19:02:02.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.895+0000 7f5e8d747700 1 --2- 172.21.15.114:0/628421650 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5e7406c4e0 0x7f5e7406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:02.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.895+0000 7f5e8d747700 1 --2- 172.21.15.114:0/628421650 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5e7406c4e0 0x7f5e7406e9a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5e88101c10 tx=0x7f5e78009260 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:02.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:02.899+0000 7f5e7e7fc700 1 -- 172.21.15.114:0/628421650 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5e8405afd0 con 0x7f5e88101db0 2024-02-19T19:02:03.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.101+0000 7f5e8f9ab700 1 -- 172.21.15.114:0/628421650 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e88106700 con 0x7f5e7406c4e0 2024-02-19T19:02:03.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.112+0000 7f5e7e7fc700 1 -- 172.21.15.114:0/628421650 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3528 (secure 0 0 0) 0x7f5e88106700 con 0x7f5e7406c4e0 2024-02-19T19:02:03.131 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:02:03.131 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (6m) 92s ago 8m 26.5M - 0.25.0 c8568f914cd2 30444b676ca4 2024-02-19T19:02:03.131 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (9m) 92s ago 9m 7470k - 18.2.1 94eadd990e0a 4918cd87e312 2024-02-19T19:02:03.131 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi174 smithi174 running (3m) 99s ago 6m 7394k - 18.2.1 94eadd990e0a 516c34431b93 2024-02-19T19:02:03.131 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (9m) 92s ago 9m 6660k - 18.2.1 94eadd990e0a ecfcd98907ba 2024-02-19T19:02:03.131 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi174 smithi174 running (6m) 99s ago 6m 6660k - 18.2.1 94eadd990e0a de88985f5a07 2024-02-19T19:02:03.131 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (5m) 92s ago 7m 84.7M - 9.4.7 954c08fa6188 478bd533e827 2024-02-19T19:02:03.131 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tcwpod smithi114 running (111s) 92s ago 110s 15.1M - 18.2.1 94eadd990e0a 420175777cf4 2024-02-19T19:02:03.131 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.vpoeop smithi114 running (118s) 92s ago 117s 11.8M - 18.2.1 94eadd990e0a f81a79e53ead 2024-02-19T19:02:03.131 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.stnfnf smithi174 running (108s) 99s ago 107s 14.6M - 18.2.1 94eadd990e0a 3df35954a492 2024-02-19T19:02:03.132 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.znhuvd smithi174 running (115s) 99s ago 115s 15.1M - 18.2.1 94eadd990e0a 2d2267494ae1 2024-02-19T19:02:03.132 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.pcvtrk smithi114 *:9283,8765,8443 running (10m) 92s ago 10m 489M - 18.2.1 94eadd990e0a 9d81af39d736 2024-02-19T19:02:03.132 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi174.lydvnj smithi174 *:8443,9283,8765 running (6m) 99s ago 6m 443M - 18.2.1 94eadd990e0a 9b31a7d679b9 2024-02-19T19:02:03.132 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (10m) 92s ago 10m 46.8M 2048M 18.2.1 94eadd990e0a 41468f811fc0 2024-02-19T19:02:03.132 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi174 smithi174 running (6m) 99s ago 6m 39.9M 2048M 18.2.1 94eadd990e0a 70dd7bf69fd9 2024-02-19T19:02:03.132 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (8m) 92s ago 8m 14.2M - 1.5.0 0da6a335fe13 b6b258074cce 2024-02-19T19:02:03.132 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi174 smithi174 *:9100 running (6m) 99s ago 6m 15.2M - 1.5.0 0da6a335fe13 d0b4fdf67b22 2024-02-19T19:02:03.132 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (5m) 92s ago 5m 46.8M 3946M 18.2.1 94eadd990e0a 990afc982b6c 2024-02-19T19:02:03.132 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (5m) 92s ago 5m 47.2M 3946M 18.2.1 94eadd990e0a 932e3ad37db5 2024-02-19T19:02:03.132 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (4m) 92s ago 4m 48.7M 3946M 18.2.1 94eadd990e0a 12953988af86 2024-02-19T19:02:03.132 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi174 running (4m) 99s ago 4m 46.1M 4970M 18.2.1 94eadd990e0a 979f45cabbce 2024-02-19T19:02:03.132 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi174 running (3m) 99s ago 3m 46.4M 4970M 18.2.1 94eadd990e0a 8b8c3745b8c5 2024-02-19T19:02:03.132 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi174 running (3m) 99s ago 3m 48.6M 4970M 18.2.1 94eadd990e0a 5771a995ed75 2024-02-19T19:02:03.132 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (5m) 92s ago 7m 55.2M - 2.43.0 a07b618ecd1d 52087f218c92 2024-02-19T19:02:03.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.114+0000 7f5e8f9ab700 1 -- 172.21.15.114:0/628421650 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5e7406c4e0 msgr2=0x7f5e7406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:03.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.114+0000 7f5e8f9ab700 1 --2- 172.21.15.114:0/628421650 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5e7406c4e0 0x7f5e7406e9a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5e88101c10 tx=0x7f5e78009260 comp rx=0 tx=0).stop 2024-02-19T19:02:03.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.114+0000 7f5e8f9ab700 1 -- 172.21.15.114:0/628421650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e88101db0 msgr2=0x7f5e881968a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:03.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.114+0000 7f5e8f9ab700 1 --2- 172.21.15.114:0/628421650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e88101db0 0x7f5e881968a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5e84000c00 tx=0x7f5e8400f690 comp rx=0 tx=0).stop 2024-02-19T19:02:03.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.114+0000 7f5e8f9ab700 1 -- 172.21.15.114:0/628421650 shutdown_connections 2024-02-19T19:02:03.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.114+0000 7f5e8f9ab700 1 --2- 172.21.15.114:0/628421650 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5e7406c4e0 0x7f5e7406e9a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.114+0000 7f5e8f9ab700 1 --2- 172.21.15.114:0/628421650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e88100bb0 0x7f5e88196360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.114+0000 7f5e8f9ab700 1 --2- 172.21.15.114:0/628421650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e88101db0 0x7f5e881968a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.114+0000 7f5e8f9ab700 1 -- 172.21.15.114:0/628421650 >> 172.21.15.114:0/628421650 conn(0x7f5e880fc110 msgr2=0x7f5e88104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:03.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.114+0000 7f5e8f9ab700 1 -- 172.21.15.114:0/628421650 shutdown_connections 2024-02-19T19:02:03.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.114+0000 7f5e8f9ab700 1 -- 172.21.15.114:0/628421650 wait complete. 2024-02-19T19:02:03.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.216+0000 7f5bb9c9b700 1 -- 172.21.15.114:0/2916877029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb4100be0 msgr2=0x7f5bb4101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:03.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.216+0000 7f5bb9c9b700 1 --2- 172.21.15.114:0/2916877029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb4100be0 0x7f5bb4101000 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00b0a0 tx=0x7f5bac00b3b0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.216+0000 7f5bb9c9b700 1 -- 172.21.15.114:0/2916877029 shutdown_connections 2024-02-19T19:02:03.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.216+0000 7f5bb9c9b700 1 --2- 172.21.15.114:0/2916877029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb4101de0 0x7f5bb4102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.216+0000 7f5bb9c9b700 1 --2- 172.21.15.114:0/2916877029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb4100be0 0x7f5bb4101000 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.216+0000 7f5bb9c9b700 1 -- 172.21.15.114:0/2916877029 >> 172.21.15.114:0/2916877029 conn(0x7f5bb40fc140 msgr2=0x7f5bb40fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:03.236 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:02 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:03.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.216+0000 7f5bb9c9b700 1 -- 172.21.15.114:0/2916877029 shutdown_connections 2024-02-19T19:02:03.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.217+0000 7f5bb9c9b700 1 -- 172.21.15.114:0/2916877029 wait complete. 2024-02-19T19:02:03.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.217+0000 7f5bb9c9b700 1 Processor -- start 2024-02-19T19:02:03.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.217+0000 7f5bb9c9b700 1 -- start start 2024-02-19T19:02:03.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.217+0000 7f5bb9c9b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb4100be0 0x7f5bb4198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:03.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.217+0000 7f5bb9c9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb4101de0 0x7f5bb4198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:03.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.217+0000 7f5bb9c9b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb41990e0 con 0x7f5bb4101de0 2024-02-19T19:02:03.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.217+0000 7f5bb9c9b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb4199220 con 0x7f5bb4100be0 2024-02-19T19:02:03.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.218+0000 7f5bb2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb4101de0 0x7f5bb4198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:03.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.218+0000 7f5bb37fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb4100be0 0x7f5bb4198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:03.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.218+0000 7f5bb2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb4101de0 0x7f5bb4198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52440/0 (socket says 172.21.15.114:52440) 2024-02-19T19:02:03.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.218+0000 7f5bb2ffd700 1 -- 172.21.15.114:0/2076093668 learned_addr learned my addr 172.21.15.114:0/2076093668 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:02:03.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.218+0000 7f5bb2ffd700 1 -- 172.21.15.114:0/2076093668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb4100be0 msgr2=0x7f5bb4198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:03.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.218+0000 7f5bb2ffd700 1 --2- 172.21.15.114:0/2076093668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb4100be0 0x7f5bb4198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.218+0000 7f5bb2ffd700 1 -- 172.21.15.114:0/2076093668 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bac009d00 con 0x7f5bb4101de0 2024-02-19T19:02:03.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.218+0000 7f5bb37fe700 1 --2- 172.21.15.114:0/2076093668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb4100be0 0x7f5bb4198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:02:03.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.218+0000 7f5bb2ffd700 1 --2- 172.21.15.114:0/2076093668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb4101de0 0x7f5bb4198ac0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00d8d0 tx=0x7f5b9c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:03.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.218+0000 7f5bb0ff9700 1 -- 172.21.15.114:0/2076093668 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b9c0098a0 con 0x7f5bb4101de0 2024-02-19T19:02:03.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.218+0000 7f5bb0ff9700 1 -- 172.21.15.114:0/2076093668 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5b9c010460 con 0x7f5bb4101de0 2024-02-19T19:02:03.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.218+0000 7f5bb0ff9700 1 -- 172.21.15.114:0/2076093668 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b9c00f5f0 con 0x7f5bb4101de0 2024-02-19T19:02:03.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.218+0000 7f5bb9c9b700 1 -- 172.21.15.114:0/2076093668 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb419dcd0 con 0x7f5bb4101de0 2024-02-19T19:02:03.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.218+0000 7f5bb9c9b700 1 -- 172.21.15.114:0/2076093668 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb419e2a0 con 0x7f5bb4101de0 2024-02-19T19:02:03.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.220+0000 7f5bb9c9b700 1 -- 172.21.15.114:0/2076093668 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb4105ec0 con 0x7f5bb4101de0 2024-02-19T19:02:03.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.220+0000 7f5bb0ff9700 1 -- 172.21.15.114:0/2076093668 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f5b9c009a00 con 0x7f5bb4101de0 2024-02-19T19:02:03.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.221+0000 7f5bb0ff9700 1 --2- 172.21.15.114:0/2076093668 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5ba006c4e0 0x7f5ba006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:03.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.221+0000 7f5bb37fe700 1 --2- 172.21.15.114:0/2076093668 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5ba006c4e0 0x7f5ba006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:03.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.221+0000 7f5bb0ff9700 1 -- 172.21.15.114:0/2076093668 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f5b9c08b690 con 0x7f5bb4101de0 2024-02-19T19:02:03.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.221+0000 7f5bb37fe700 1 --2- 172.21.15.114:0/2076093668 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5ba006c4e0 0x7f5ba006e9a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00b830 tx=0x7f5bac0060c0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:03.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.225+0000 7f5bb0ff9700 1 -- 172.21.15.114:0/2076093668 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5b9c059920 con 0x7f5bb4101de0 2024-02-19T19:02:03.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:02 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:03.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.504+0000 7f5bb9c9b700 1 -- 172.21.15.114:0/2076093668 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5bb402cc70 con 0x7f5bb4101de0 2024-02-19T19:02:03.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.504+0000 7f5bb0ff9700 1 -- 172.21.15.114:0/2076093668 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f5b9c0594b0 con 0x7f5bb4101de0 2024-02-19T19:02:03.524 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:02:03.524 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-02-19T19:02:03.524 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:02:03.524 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:02:03.524 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-02-19T19:02:03.524 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:02:03.524 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:02:03.524 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-02-19T19:02:03.524 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-02-19T19:02:03.524 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:02:03.524 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-02-19T19:02:03.524 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:02:03.524 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:02:03.525 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-02-19T19:02:03.525 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-02-19T19:02:03.525 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-02-19T19:02:03.525 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:02:03.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.507+0000 7f5bb9c9b700 1 -- 172.21.15.114:0/2076093668 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5ba006c4e0 msgr2=0x7f5ba006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:03.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.507+0000 7f5bb9c9b700 1 --2- 172.21.15.114:0/2076093668 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5ba006c4e0 0x7f5ba006e9a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00b830 tx=0x7f5bac0060c0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.507+0000 7f5bb9c9b700 1 -- 172.21.15.114:0/2076093668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb4101de0 msgr2=0x7f5bb4198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:03.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.507+0000 7f5bb9c9b700 1 --2- 172.21.15.114:0/2076093668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb4101de0 0x7f5bb4198ac0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00d8d0 tx=0x7f5b9c00dbe0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.507+0000 7f5bb9c9b700 1 -- 172.21.15.114:0/2076093668 shutdown_connections 2024-02-19T19:02:03.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.507+0000 7f5bb9c9b700 1 --2- 172.21.15.114:0/2076093668 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f5ba006c4e0 0x7f5ba006e9a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.507+0000 7f5bb9c9b700 1 --2- 172.21.15.114:0/2076093668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb4100be0 0x7f5bb4198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.507+0000 7f5bb9c9b700 1 --2- 172.21.15.114:0/2076093668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb4101de0 0x7f5bb4198ac0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.507+0000 7f5bb9c9b700 1 -- 172.21.15.114:0/2076093668 >> 172.21.15.114:0/2076093668 conn(0x7f5bb40fc140 msgr2=0x7f5bb4103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:03.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.507+0000 7f5bb9c9b700 1 -- 172.21.15.114:0/2076093668 shutdown_connections 2024-02-19T19:02:03.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.507+0000 7f5bb9c9b700 1 -- 172.21.15.114:0/2076093668 wait complete. 2024-02-19T19:02:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.614+0000 7fde7f895700 1 -- 172.21.15.114:0/1096565376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde78100cb0 msgr2=0x7fde78101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:03.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.614+0000 7fde7f895700 1 --2- 172.21.15.114:0/1096565376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde78100cb0 0x7fde78101110 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009910 tx=0x7fde6c009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.614+0000 7fde7f895700 1 -- 172.21.15.114:0/1096565376 shutdown_connections 2024-02-19T19:02:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.614+0000 7fde7f895700 1 --2- 172.21.15.114:0/1096565376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde78100cb0 0x7fde78101110 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.614+0000 7fde7f895700 1 --2- 172.21.15.114:0/1096565376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde780ffab0 0x7fde780ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.614+0000 7fde7f895700 1 -- 172.21.15.114:0/1096565376 >> 172.21.15.114:0/1096565376 conn(0x7fde780fb030 msgr2=0x7fde780fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.614+0000 7fde7f895700 1 -- 172.21.15.114:0/1096565376 shutdown_connections 2024-02-19T19:02:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.614+0000 7fde7f895700 1 -- 172.21.15.114:0/1096565376 wait complete. 2024-02-19T19:02:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.614+0000 7fde7f895700 1 Processor -- start 2024-02-19T19:02:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.614+0000 7fde7f895700 1 -- start start 2024-02-19T19:02:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.614+0000 7fde7f895700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde780ffab0 0x7fde78195260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.614+0000 7fde7f895700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde78100cb0 0x7fde781957a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.614+0000 7fde7f895700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde78195dc0 con 0x7fde780ffab0 2024-02-19T19:02:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.614+0000 7fde7f895700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde78195f00 con 0x7fde78100cb0 2024-02-19T19:02:03.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.615+0000 7fde7d631700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde780ffab0 0x7fde78195260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.615+0000 7fde7d631700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde780ffab0 0x7fde78195260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52458/0 (socket says 172.21.15.114:52458) 2024-02-19T19:02:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.615+0000 7fde7d631700 1 -- 172.21.15.114:0/1946302929 learned_addr learned my addr 172.21.15.114:0/1946302929 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:02:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.615+0000 7fde7ce30700 1 --2- 172.21.15.114:0/1946302929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde78100cb0 0x7fde781957a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.615+0000 7fde7d631700 1 -- 172.21.15.114:0/1946302929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde78100cb0 msgr2=0x7fde781957a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.615+0000 7fde7d631700 1 --2- 172.21.15.114:0/1946302929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde78100cb0 0x7fde781957a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.615+0000 7fde7d631700 1 -- 172.21.15.114:0/1946302929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c0095f0 con 0x7fde780ffab0 2024-02-19T19:02:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.615+0000 7fde7d631700 1 --2- 172.21.15.114:0/1946302929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde780ffab0 0x7fde78195260 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fde6800e8d0 tx=0x7fde6800ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.615+0000 7fde767fc700 1 -- 172.21.15.114:0/1946302929 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde68009770 con 0x7fde780ffab0 2024-02-19T19:02:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.615+0000 7fde767fc700 1 -- 172.21.15.114:0/1946302929 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fde6800c810 con 0x7fde780ffab0 2024-02-19T19:02:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.615+0000 7fde7f895700 1 -- 172.21.15.114:0/1946302929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7819a9b0 con 0x7fde780ffab0 2024-02-19T19:02:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.615+0000 7fde767fc700 1 -- 172.21.15.114:0/1946302929 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde68003da0 con 0x7fde780ffab0 2024-02-19T19:02:03.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.615+0000 7fde7f895700 1 -- 172.21.15.114:0/1946302929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7819af00 con 0x7fde780ffab0 2024-02-19T19:02:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.616+0000 7fde7f895700 1 -- 172.21.15.114:0/1946302929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde78108630 con 0x7fde780ffab0 2024-02-19T19:02:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.617+0000 7fde767fc700 1 -- 172.21.15.114:0/1946302929 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fde68004200 con 0x7fde780ffab0 2024-02-19T19:02:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.617+0000 7fde767fc700 1 --2- 172.21.15.114:0/1946302929 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fde6006c4e0 0x7fde6006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.617+0000 7fde767fc700 1 -- 172.21.15.114:0/1946302929 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fde68014070 con 0x7fde780ffab0 2024-02-19T19:02:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.617+0000 7fde7ce30700 1 --2- 172.21.15.114:0/1946302929 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fde6006c4e0 0x7fde6006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:03.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.618+0000 7fde7ce30700 1 --2- 172.21.15.114:0/1946302929 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fde6006c4e0 0x7fde6006e9a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00b6f0 tx=0x7fde6c005fb0 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.622+0000 7fde767fc700 1 -- 172.21.15.114:0/1946302929 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fde68059d40 con 0x7fde780ffab0 2024-02-19T19:02:03.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.856+0000 7fde7f895700 1 -- 172.21.15.114:0/1946302929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fde7819bac0 con 0x7fde780ffab0 2024-02-19T19:02:03.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.856+0000 7fde767fc700 1 -- 172.21.15.114:0/1946302929 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1850 (secure 0 0 0) 0x7fde6800ce60 con 0x7fde780ffab0 2024-02-19T19:02:03.875 INFO:teuthology.orchestra.run.smithi114.stdout:e18 2024-02-19T19:02:03.875 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:02:03.875 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:02:03.875 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-02-19T19:02:03.876 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:02:03.876 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:02:03.876 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-02-19T19:02:03.876 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 18 2024-02-19T19:02:03.876 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:02:03.876 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:02:03.876 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-02-19T19:00:21.203922+0000 2024-02-19T19:02:03.876 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-02-19T19:02:03.876 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-02-19T19:02:03.876 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-02-19T19:02:03.876 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-02-19T19:02:03.876 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-02-19T19:02:03.876 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-02-19T19:02:03.876 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-02-19T19:02:03.876 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 47 2024-02-19T19:02:03.876 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:02:03.877 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 2 2024-02-19T19:02:03.877 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1 2024-02-19T19:02:03.877 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=14530,1=24269} 2024-02-19T19:02:03.877 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-02-19T19:02:03.877 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-02-19T19:02:03.877 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-02-19T19:02:03.877 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-02-19T19:02:03.877 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-02-19T19:02:03.877 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data enabled 2024-02-19T19:02:03.877 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-02-19T19:02:03.877 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-02-19T19:02:03.877 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 1 2024-02-19T19:02:03.877 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:02:03.877 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.vpoeop{0:14540} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:02:03.878 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.znhuvd{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:02:03.878 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.stnfnf{1:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:02:03.878 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:02:03.878 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:02:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.859+0000 7fde7f895700 1 -- 172.21.15.114:0/1946302929 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fde6006c4e0 msgr2=0x7fde6006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.859+0000 7fde7f895700 1 --2- 172.21.15.114:0/1946302929 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fde6006c4e0 0x7fde6006e9a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00b6f0 tx=0x7fde6c005fb0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.859+0000 7fde7f895700 1 -- 172.21.15.114:0/1946302929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde780ffab0 msgr2=0x7fde78195260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.859+0000 7fde7f895700 1 --2- 172.21.15.114:0/1946302929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde780ffab0 0x7fde78195260 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fde6800e8d0 tx=0x7fde6800ebe0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.859+0000 7fde7f895700 1 -- 172.21.15.114:0/1946302929 shutdown_connections 2024-02-19T19:02:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.859+0000 7fde7f895700 1 --2- 172.21.15.114:0/1946302929 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7fde6006c4e0 0x7fde6006e9a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.859+0000 7fde7f895700 1 --2- 172.21.15.114:0/1946302929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde780ffab0 0x7fde78195260 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.859+0000 7fde7f895700 1 --2- 172.21.15.114:0/1946302929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde78100cb0 0x7fde781957a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.859+0000 7fde7f895700 1 -- 172.21.15.114:0/1946302929 >> 172.21.15.114:0/1946302929 conn(0x7fde780fb030 msgr2=0x7fde78103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.859+0000 7fde7f895700 1 -- 172.21.15.114:0/1946302929 shutdown_connections 2024-02-19T19:02:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.859+0000 7fde7f895700 1 -- 172.21.15.114:0/1946302929 wait complete. 2024-02-19T19:02:03.879 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 18 2024-02-19T19:02:03.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.963+0000 7effc3f5b700 1 -- 172.21.15.114:0/2724002801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effbc101640 msgr2=0x7effbc103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:03.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.963+0000 7effc3f5b700 1 --2- 172.21.15.114:0/2724002801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effbc101640 0x7effbc103a30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7effb400b0a0 tx=0x7effb400b3b0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.963+0000 7effc3f5b700 1 -- 172.21.15.114:0/2724002801 shutdown_connections 2024-02-19T19:02:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.963+0000 7effc3f5b700 1 --2- 172.21.15.114:0/2724002801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effbc103f70 0x7effbc106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.963+0000 7effc3f5b700 1 --2- 172.21.15.114:0/2724002801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effbc101640 0x7effbc103a30 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.963+0000 7effc3f5b700 1 -- 172.21.15.114:0/2724002801 >> 172.21.15.114:0/2724002801 conn(0x7effbc0fb030 msgr2=0x7effbc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.963+0000 7effc3f5b700 1 -- 172.21.15.114:0/2724002801 shutdown_connections 2024-02-19T19:02:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.963+0000 7effc3f5b700 1 -- 172.21.15.114:0/2724002801 wait complete. 2024-02-19T19:02:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.963+0000 7effc3f5b700 1 Processor -- start 2024-02-19T19:02:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.963+0000 7effc3f5b700 1 -- start start 2024-02-19T19:02:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.964+0000 7effc3f5b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effbc101640 0x7effbc196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.964+0000 7effc3f5b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effbc103f70 0x7effbc1969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.964+0000 7effc3f5b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc196fc0 con 0x7effbc101640 2024-02-19T19:02:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.964+0000 7effc3f5b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc197100 con 0x7effbc103f70 2024-02-19T19:02:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.964+0000 7effc1cf7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effbc101640 0x7effbc196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.964+0000 7effc1cf7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effbc101640 0x7effbc196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52466/0 (socket says 172.21.15.114:52466) 2024-02-19T19:02:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.964+0000 7effc1cf7700 1 -- 172.21.15.114:0/282053384 learned_addr learned my addr 172.21.15.114:0/282053384 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:02:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.964+0000 7effc14f6700 1 --2- 172.21.15.114:0/282053384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effbc103f70 0x7effbc1969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.964+0000 7effc1cf7700 1 -- 172.21.15.114:0/282053384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effbc103f70 msgr2=0x7effbc1969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.964+0000 7effc1cf7700 1 --2- 172.21.15.114:0/282053384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effbc103f70 0x7effbc1969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.964+0000 7effc1cf7700 1 -- 172.21.15.114:0/282053384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb4009d00 con 0x7effbc101640 2024-02-19T19:02:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.964+0000 7effc1cf7700 1 --2- 172.21.15.114:0/282053384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effbc101640 0x7effbc196460 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7effb400b070 tx=0x7effb4003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.964+0000 7effb2ffd700 1 -- 172.21.15.114:0/282053384 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effb400e040 con 0x7effbc101640 2024-02-19T19:02:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.964+0000 7effb2ffd700 1 -- 172.21.15.114:0/282053384 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7effb400ce60 con 0x7effbc101640 2024-02-19T19:02:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.964+0000 7effc3f5b700 1 -- 172.21.15.114:0/282053384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effbc19bb50 con 0x7effbc101640 2024-02-19T19:02:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.964+0000 7effb2ffd700 1 -- 172.21.15.114:0/282053384 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effb401da50 con 0x7effbc101640 2024-02-19T19:02:03.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.964+0000 7effc3f5b700 1 -- 172.21.15.114:0/282053384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effbc19c040 con 0x7effbc101640 2024-02-19T19:02:03.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.965+0000 7effc3f5b700 1 -- 172.21.15.114:0/282053384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effbc1905a0 con 0x7effbc101640 2024-02-19T19:02:03.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.966+0000 7effb2ffd700 1 -- 172.21.15.114:0/282053384 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7effb4004250 con 0x7effbc101640 2024-02-19T19:02:03.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.966+0000 7effb2ffd700 1 --2- 172.21.15.114:0/282053384 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7effa806c4e0 0x7effa806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:03.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.966+0000 7effc14f6700 1 --2- 172.21.15.114:0/282053384 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7effa806c4e0 0x7effa806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:03.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.966+0000 7effb2ffd700 1 -- 172.21.15.114:0/282053384 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7effb408d2b0 con 0x7effbc101640 2024-02-19T19:02:03.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.967+0000 7effc14f6700 1 --2- 172.21.15.114:0/282053384 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7effa806c4e0 0x7effa806e9a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7effac009a10 tx=0x7effac009190 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:03.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:03.971+0000 7effb2ffd700 1 -- 172.21.15.114:0/282053384 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7effb405b540 con 0x7effbc101640 2024-02-19T19:02:04.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.179+0000 7effc3f5b700 1 -- 172.21.15.114:0/282053384 --> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7effbc0611d0 con 0x7effa806c4e0 2024-02-19T19:02:04.199 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:03 smithi114 ceph-mon[29441]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:04.199 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:03 smithi114 ceph-mon[29441]: pgmap v245: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-19T19:02:04.199 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:03 smithi114 ceph-mon[29441]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:04.199 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:03 smithi114 ceph-mon[29441]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:04.199 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:03 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/2076093668' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:04.199 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:03 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1946302929' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:02:04.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.181+0000 7effb2ffd700 1 -- 172.21.15.114:0/282053384 <== mgr.14257 v2:172.21.15.114:6800/2188833308 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7effbc0611d0 con 0x7effa806c4e0 2024-02-19T19:02:04.200 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:02:04.200 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:02:04.200 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-02-19T19:02:04.200 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:02:04.200 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [], 2024-02-19T19:02:04.200 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "0/23 daemons upgraded", 2024-02-19T19:02:04.200 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 image on host smithi174", 2024-02-19T19:02:04.200 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-02-19T19:02:04.200 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:02:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.183+0000 7effc3f5b700 1 -- 172.21.15.114:0/282053384 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7effa806c4e0 msgr2=0x7effa806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.183+0000 7effc3f5b700 1 --2- 172.21.15.114:0/282053384 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7effa806c4e0 0x7effa806e9a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7effac009a10 tx=0x7effac009190 comp rx=0 tx=0).stop 2024-02-19T19:02:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.183+0000 7effc3f5b700 1 -- 172.21.15.114:0/282053384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effbc101640 msgr2=0x7effbc196460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.183+0000 7effc3f5b700 1 --2- 172.21.15.114:0/282053384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effbc101640 0x7effbc196460 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7effb400b070 tx=0x7effb4003c30 comp rx=0 tx=0).stop 2024-02-19T19:02:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.183+0000 7effc3f5b700 1 -- 172.21.15.114:0/282053384 shutdown_connections 2024-02-19T19:02:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.183+0000 7effc3f5b700 1 --2- 172.21.15.114:0/282053384 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7effa806c4e0 0x7effa806e9a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.183+0000 7effc3f5b700 1 --2- 172.21.15.114:0/282053384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effbc101640 0x7effbc196460 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.183+0000 7effc3f5b700 1 --2- 172.21.15.114:0/282053384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effbc103f70 0x7effbc1969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.183+0000 7effc3f5b700 1 -- 172.21.15.114:0/282053384 >> 172.21.15.114:0/282053384 conn(0x7effbc0fb030 msgr2=0x7effbc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.183+0000 7effc3f5b700 1 -- 172.21.15.114:0/282053384 shutdown_connections 2024-02-19T19:02:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.183+0000 7effc3f5b700 1 -- 172.21.15.114:0/282053384 wait complete. 2024-02-19T19:02:04.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.286+0000 7f7ec07f1700 1 -- 172.21.15.114:0/3328790432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eb80ff440 msgr2=0x7f7eb8101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:04.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.286+0000 7f7ec07f1700 1 --2- 172.21.15.114:0/3328790432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eb80ff440 0x7f7eb8101870 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4009910 tx=0x7f7eb4009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:04.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.287+0000 7f7ec07f1700 1 -- 172.21.15.114:0/3328790432 shutdown_connections 2024-02-19T19:02:04.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.287+0000 7f7ec07f1700 1 --2- 172.21.15.114:0/3328790432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eb8101db0 0x7f7eb8104240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:04.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.287+0000 7f7ec07f1700 1 --2- 172.21.15.114:0/3328790432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eb80ff440 0x7f7eb8101870 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:04.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.287+0000 7f7ec07f1700 1 -- 172.21.15.114:0/3328790432 >> 172.21.15.114:0/3328790432 conn(0x7f7eb80fb060 msgr2=0x7f7eb80fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:04.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.287+0000 7f7ec07f1700 1 -- 172.21.15.114:0/3328790432 shutdown_connections 2024-02-19T19:02:04.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.287+0000 7f7ec07f1700 1 -- 172.21.15.114:0/3328790432 wait complete. 2024-02-19T19:02:04.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.287+0000 7f7ec07f1700 1 Processor -- start 2024-02-19T19:02:04.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.287+0000 7f7ec07f1700 1 -- start start 2024-02-19T19:02:04.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.287+0000 7f7ec07f1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eb80ff440 0x7f7eb819ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:04.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.287+0000 7f7ec07f1700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eb8101db0 0x7f7eb819cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:04.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.287+0000 7f7ec07f1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb819d570 con 0x7f7eb80ff440 2024-02-19T19:02:04.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.287+0000 7f7ec07f1700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb819d6b0 con 0x7f7eb8101db0 2024-02-19T19:02:04.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.287+0000 7f7ebe58d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eb80ff440 0x7f7eb819ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:04.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.287+0000 7f7ebe58d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eb80ff440 0x7f7eb819ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52474/0 (socket says 172.21.15.114:52474) 2024-02-19T19:02:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.287+0000 7f7ebe58d700 1 -- 172.21.15.114:0/2818850043 learned_addr learned my addr 172.21.15.114:0/2818850043 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:02:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.288+0000 7f7ebe58d700 1 -- 172.21.15.114:0/2818850043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eb8101db0 msgr2=0x7f7eb819cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.288+0000 7f7ebdd8c700 1 --2- 172.21.15.114:0/2818850043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eb8101db0 0x7f7eb819cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.288+0000 7f7ebe58d700 1 --2- 172.21.15.114:0/2818850043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eb8101db0 0x7f7eb819cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.288+0000 7f7ebe58d700 1 -- 172.21.15.114:0/2818850043 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb40095f0 con 0x7f7eb80ff440 2024-02-19T19:02:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.288+0000 7f7ebe58d700 1 --2- 172.21.15.114:0/2818850043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eb80ff440 0x7f7eb819ca10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4006010 tx=0x7f7eb400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.288+0000 7f7eaf7fe700 1 -- 172.21.15.114:0/2818850043 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7eb401d070 con 0x7f7eb80ff440 2024-02-19T19:02:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.288+0000 7f7eaf7fe700 1 -- 172.21.15.114:0/2818850043 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7eb400f460 con 0x7f7eb80ff440 2024-02-19T19:02:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.288+0000 7f7eaf7fe700 1 -- 172.21.15.114:0/2818850043 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7eb4021640 con 0x7f7eb80ff440 2024-02-19T19:02:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.288+0000 7f7ec07f1700 1 -- 172.21.15.114:0/2818850043 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb81a2100 con 0x7f7eb80ff440 2024-02-19T19:02:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.288+0000 7f7ec07f1700 1 -- 172.21.15.114:0/2818850043 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb81a2620 con 0x7f7eb80ff440 2024-02-19T19:02:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.289+0000 7f7ec07f1700 1 -- 172.21.15.114:0/2818850043 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb8196ba0 con 0x7f7eb80ff440 2024-02-19T19:02:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.290+0000 7f7eaf7fe700 1 -- 172.21.15.114:0/2818850043 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f7eb400fab0 con 0x7f7eb80ff440 2024-02-19T19:02:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.290+0000 7f7eaf7fe700 1 --2- 172.21.15.114:0/2818850043 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7ea406c4e0 0x7f7ea406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.290+0000 7f7eaf7fe700 1 -- 172.21.15.114:0/2818850043 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7eb408cd10 con 0x7f7eb80ff440 2024-02-19T19:02:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.290+0000 7f7ebdd8c700 1 --2- 172.21.15.114:0/2818850043 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7ea406c4e0 0x7f7ea406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.290+0000 7f7ebdd8c700 1 --2- 172.21.15.114:0/2818850043 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7ea406c4e0 0x7f7ea406e9a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8006fd0 tx=0x7f7ea8008040 comp rx=0 tx=0).ready entity=mgr.14257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:04.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.294+0000 7f7eaf7fe700 1 -- 172.21.15.114:0/2818850043 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7eb405b040 con 0x7f7eb80ff440 2024-02-19T19:02:04.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:03 smithi174 ceph-mon[41061]: from='client.14662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:04.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:03 smithi174 ceph-mon[41061]: pgmap v245: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-19T19:02:04.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:03 smithi174 ceph-mon[41061]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:04.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:03 smithi174 ceph-mon[41061]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:04.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:03 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/2076093668' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:04.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:03 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1946302929' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:02:04.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.576+0000 7f7ec07f1700 1 -- 172.21.15.114:0/2818850043 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7eb802cc70 con 0x7f7eb80ff440 2024-02-19T19:02:04.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.576+0000 7f7eaf7fe700 1 -- 172.21.15.114:0/2818850043 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f7eb40260e0 con 0x7f7eb80ff440 2024-02-19T19:02:04.595 INFO:teuthology.orchestra.run.smithi114.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:02:04.595 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:02:04.596 INFO:teuthology.orchestra.run.smithi114.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:02:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.578+0000 7f7ec07f1700 1 -- 172.21.15.114:0/2818850043 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7ea406c4e0 msgr2=0x7f7ea406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.578+0000 7f7ec07f1700 1 --2- 172.21.15.114:0/2818850043 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7ea406c4e0 0x7f7ea406e9a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8006fd0 tx=0x7f7ea8008040 comp rx=0 tx=0).stop 2024-02-19T19:02:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.578+0000 7f7ec07f1700 1 -- 172.21.15.114:0/2818850043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eb80ff440 msgr2=0x7f7eb819ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.578+0000 7f7ec07f1700 1 --2- 172.21.15.114:0/2818850043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eb80ff440 0x7f7eb819ca10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4006010 tx=0x7f7eb400be20 comp rx=0 tx=0).stop 2024-02-19T19:02:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.578+0000 7f7ec07f1700 1 -- 172.21.15.114:0/2818850043 shutdown_connections 2024-02-19T19:02:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.578+0000 7f7ec07f1700 1 --2- 172.21.15.114:0/2818850043 >> [v2:172.21.15.114:6800/2188833308,v1:172.21.15.114:6801/2188833308] conn(0x7f7ea406c4e0 0x7f7ea406e9a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.578+0000 7f7ec07f1700 1 --2- 172.21.15.114:0/2818850043 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7eb80ff440 0x7f7eb819ca10 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.578+0000 7f7ec07f1700 1 --2- 172.21.15.114:0/2818850043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eb8101db0 0x7f7eb819cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.578+0000 7f7ec07f1700 1 -- 172.21.15.114:0/2818850043 >> 172.21.15.114:0/2818850043 conn(0x7f7eb80fb060 msgr2=0x7f7eb80fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.578+0000 7f7ec07f1700 1 -- 172.21.15.114:0/2818850043 shutdown_connections 2024-02-19T19:02:04.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:04.578+0000 7f7ec07f1700 1 -- 172.21.15.114:0/2818850043 wait complete. 2024-02-19T19:02:05.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:04 smithi114 ceph-mon[29441]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:05.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:04 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/2818850043' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:02:05.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:04 smithi114 ceph-mon[29441]: pgmap v246: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-19T19:02:05.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:04 smithi174 ceph-mon[41061]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:05.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:04 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/2818850043' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:02:05.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:04 smithi174 ceph-mon[41061]: pgmap v246: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-19T19:02:07.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:07 smithi174 ceph-mon[41061]: pgmap v247: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:08.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:07 smithi114 ceph-mon[29441]: pgmap v247: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:09.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:09 smithi174 ceph-mon[41061]: Upgrade: Updating mgr.smithi174.lydvnj 2024-02-19T19:02:09.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:09 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:09.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:09 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi174.lydvnj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:02:09.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:09 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:02:09.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:09 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:09.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:09 smithi174 ceph-mon[41061]: Deploying daemon mgr.smithi174.lydvnj on smithi174 2024-02-19T19:02:09.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:09 smithi174 ceph-mon[41061]: pgmap v248: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:02:10.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:09 smithi114 ceph-mon[29441]: Upgrade: Updating mgr.smithi174.lydvnj 2024-02-19T19:02:10.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:09 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:10.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:09 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi174.lydvnj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:02:10.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:09 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:02:10.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:09 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:10.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:09 smithi114 ceph-mon[29441]: Deploying daemon mgr.smithi174.lydvnj on smithi174 2024-02-19T19:02:10.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:09 smithi114 ceph-mon[29441]: pgmap v248: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:02:12.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:11 smithi114 ceph-mon[29441]: pgmap v249: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:12.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:11 smithi174 ceph-mon[41061]: pgmap v249: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:13.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:13 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:13.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:13 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:13.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:13 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:13.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:13 smithi174 ceph-mon[41061]: pgmap v250: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-19T19:02:13.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:13 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:13.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:13 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:13.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:13 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:13.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:13 smithi114 ceph-mon[29441]: pgmap v250: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-19T19:02:15.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:15 smithi174 ceph-mon[41061]: pgmap v251: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-19T19:02:15.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:15 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:15.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:15 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:16.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:15 smithi114 ceph-mon[29441]: pgmap v251: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-19T19:02:16.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:15 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:16.037 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:15 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:17.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:16 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:17.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:16 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:17.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:17.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:16 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:18.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:17 smithi174 ceph-mon[41061]: pgmap v252: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:18.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:17 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:18.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:17 smithi114 ceph-mon[29441]: pgmap v252: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:18.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:17 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:20.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: pgmap v253: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:02:20.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: Standby manager daemon smithi174.lydvnj restarted 2024-02-19T19:02:20.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: Standby manager daemon smithi174.lydvnj started 2024-02-19T19:02:20.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/crt"}]: dispatch 2024-02-19T19:02:20.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:02:20.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/key"}]: dispatch 2024-02-19T19:02:20.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:02:20.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:20.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:20.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:20.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:20.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:20.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:20.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr fail", "who": "smithi114.pcvtrk"}]: dispatch 2024-02-19T19:02:20.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: osdmap e48: 6 total, 6 up, 6 in 2024-02-19T19:02:20.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T19:02:20.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "mgr fail", "who": "smithi114.pcvtrk"}]': finished 2024-02-19T19:02:20.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T19:02:20.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: mgrmap e20: smithi174.lydvnj(active, starting, since 0.0091537s) 2024-02-19T19:02:20.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.tcwpod"}]: dispatch 2024-02-19T19:02:20.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.vpoeop"}]: dispatch 2024-02-19T19:02:20.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.znhuvd"}]: dispatch 2024-02-19T19:02:20.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.stnfnf"}]: dispatch 2024-02-19T19:02:20.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mgr metadata", "who": "smithi174.lydvnj", "id": "smithi174.lydvnj"}]: dispatch 2024-02-19T19:02:20.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:02:20.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:02:20.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:02:20.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:02:20.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:02:20.291 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:02:20.291 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T19:02:20.291 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T19:02:20.291 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T19:02:20.291 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: Manager daemon smithi174.lydvnj is now available 2024-02-19T19:02:20.291 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:20.291 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:20.291 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:20.291 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi174.lydvnj/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:02:20.291 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi174.lydvnj/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:02:20.291 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi174.lydvnj/trash_purge_schedule"}]: dispatch 2024-02-19T19:02:20.292 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:19 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi174.lydvnj/trash_purge_schedule"}]: dispatch 2024-02-19T19:02:20.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: pgmap v253: 65 pgs: 65 active+clean; 589 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:02:20.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: Standby manager daemon smithi174.lydvnj restarted 2024-02-19T19:02:20.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: Standby manager daemon smithi174.lydvnj started 2024-02-19T19:02:20.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/crt"}]: dispatch 2024-02-19T19:02:20.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:02:20.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/key"}]: dispatch 2024-02-19T19:02:20.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:02:20.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:20.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:20.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:20.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:20.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:20.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:20.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr fail", "who": "smithi114.pcvtrk"}]: dispatch 2024-02-19T19:02:20.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: osdmap e48: 6 total, 6 up, 6 in 2024-02-19T19:02:20.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T19:02:20.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.14257 172.21.15.114:0/1873249492' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "mgr fail", "who": "smithi114.pcvtrk"}]': finished 2024-02-19T19:02:20.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T19:02:20.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: mgrmap e20: smithi174.lydvnj(active, starting, since 0.0091537s) 2024-02-19T19:02:20.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.tcwpod"}]: dispatch 2024-02-19T19:02:20.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.vpoeop"}]: dispatch 2024-02-19T19:02:20.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.znhuvd"}]: dispatch 2024-02-19T19:02:20.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.stnfnf"}]: dispatch 2024-02-19T19:02:20.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mgr metadata", "who": "smithi174.lydvnj", "id": "smithi174.lydvnj"}]: dispatch 2024-02-19T19:02:20.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:02:20.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:02:20.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:02:20.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:02:20.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:02:20.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:02:20.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T19:02:20.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T19:02:20.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T19:02:20.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: Manager daemon smithi174.lydvnj is now available 2024-02-19T19:02:20.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:20.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:20.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:20.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi174.lydvnj/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:02:20.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi174.lydvnj/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:02:20.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi174.lydvnj/trash_purge_schedule"}]: dispatch 2024-02-19T19:02:20.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:19 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi174.lydvnj/trash_purge_schedule"}]: dispatch 2024-02-19T19:02:21.784 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:21 smithi114 ceph-mon[29441]: Deploying cephadm binary to smithi114 2024-02-19T19:02:21.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:21 smithi114 ceph-mon[29441]: Deploying cephadm binary to smithi174 2024-02-19T19:02:21.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:21 smithi114 ceph-mon[29441]: [19/Feb/2024:19:02:20] ENGINE Bus STARTING 2024-02-19T19:02:21.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:21 smithi114 ceph-mon[29441]: [19/Feb/2024:19:02:20] ENGINE Serving on https://172.21.15.174:7150 2024-02-19T19:02:21.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:21 smithi114 ceph-mon[29441]: mgrmap e21: smithi174.lydvnj(active, since 1.00965s) 2024-02-19T19:02:21.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:21 smithi114 ceph-mon[29441]: pgmap v3: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:21.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:21 smithi114 ceph-mon[29441]: [19/Feb/2024:19:02:20] ENGINE Serving on http://172.21.15.174:8765 2024-02-19T19:02:21.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:21 smithi114 ceph-mon[29441]: [19/Feb/2024:19:02:20] ENGINE Bus STARTED 2024-02-19T19:02:21.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:21 smithi174 ceph-mon[41061]: Deploying cephadm binary to smithi114 2024-02-19T19:02:21.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:21 smithi174 ceph-mon[41061]: Deploying cephadm binary to smithi174 2024-02-19T19:02:21.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:21 smithi174 ceph-mon[41061]: [19/Feb/2024:19:02:20] ENGINE Bus STARTING 2024-02-19T19:02:21.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:21 smithi174 ceph-mon[41061]: [19/Feb/2024:19:02:20] ENGINE Serving on https://172.21.15.174:7150 2024-02-19T19:02:21.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:21 smithi174 ceph-mon[41061]: mgrmap e21: smithi174.lydvnj(active, since 1.00965s) 2024-02-19T19:02:21.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:21 smithi174 ceph-mon[41061]: pgmap v3: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:21.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:21 smithi174 ceph-mon[41061]: [19/Feb/2024:19:02:20] ENGINE Serving on http://172.21.15.174:8765 2024-02-19T19:02:21.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:21 smithi174 ceph-mon[41061]: [19/Feb/2024:19:02:20] ENGINE Bus STARTED 2024-02-19T19:02:22.787 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:22 smithi114 ceph-mon[29441]: pgmap v4: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:22.787 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:22 smithi114 ceph-mon[29441]: mgrmap e22: smithi174.lydvnj(active, since 2s) 2024-02-19T19:02:22.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:22 smithi174 ceph-mon[41061]: pgmap v4: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:22.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:22 smithi174 ceph-mon[41061]: mgrmap e22: smithi174.lydvnj(active, since 2s) 2024-02-19T19:02:24.201 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:23 smithi114 ceph-mon[29441]: pgmap v5: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:24.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:23 smithi174 ceph-mon[41061]: pgmap v5: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:25.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:24 smithi114 ceph-mon[29441]: mgrmap e23: smithi174.lydvnj(active, since 4s) 2024-02-19T19:02:25.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:24 smithi174 ceph-mon[41061]: mgrmap e23: smithi174.lydvnj(active, since 4s) 2024-02-19T19:02:26.251 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:26 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:26.251 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:26 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:26.251 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:26 smithi114 ceph-mon[29441]: pgmap v6: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:26.415 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:26 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:26.415 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:26 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:26.415 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:26 smithi174 ceph-mon[41061]: pgmap v6: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:27.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:27 smithi174 ceph-mon[41061]: Standby manager daemon smithi114.pcvtrk started 2024-02-19T19:02:27.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:27 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.114:0/893421569' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi114.pcvtrk/crt"}]: dispatch 2024-02-19T19:02:27.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:27 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.114:0/893421569' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:02:27.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:27 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.114:0/893421569' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi114.pcvtrk/key"}]: dispatch 2024-02-19T19:02:27.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:27 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.114:0/893421569' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:02:27.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:27 smithi114 ceph-mon[29441]: Standby manager daemon smithi114.pcvtrk started 2024-02-19T19:02:27.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:27 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.114:0/893421569' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi114.pcvtrk/crt"}]: dispatch 2024-02-19T19:02:27.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:27 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.114:0/893421569' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:02:27.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:27 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.114:0/893421569' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi114.pcvtrk/key"}]: dispatch 2024-02-19T19:02:27.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:27 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.114:0/893421569' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:02:28.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:28 smithi114 ceph-mon[29441]: mgrmap e24: smithi174.lydvnj(active, since 7s), standbys: smithi114.pcvtrk 2024-02-19T19:02:28.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:28 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mgr metadata", "who": "smithi114.pcvtrk", "id": "smithi114.pcvtrk"}]: dispatch 2024-02-19T19:02:28.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:28 smithi114 ceph-mon[29441]: pgmap v7: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-02-19T19:02:28.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:28 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:28.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:28 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:28.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:28 smithi174 ceph-mon[41061]: mgrmap e24: smithi174.lydvnj(active, since 7s), standbys: smithi114.pcvtrk 2024-02-19T19:02:28.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:28 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mgr metadata", "who": "smithi114.pcvtrk", "id": "smithi114.pcvtrk"}]: dispatch 2024-02-19T19:02:28.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:28 smithi174 ceph-mon[41061]: pgmap v7: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-02-19T19:02:28.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:28 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:28.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:28 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:30.111 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:29 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:30.111 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:29 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:30.287 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:29 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:30.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:29 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:31.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:30 smithi174 ceph-mon[41061]: pgmap v8: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-02-19T19:02:31.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:30 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:31.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:30 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:31.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:30 smithi114 ceph-mon[29441]: pgmap v8: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-02-19T19:02:31.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:30 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:31.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:30 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:32.120 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:31 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:32.120 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:31 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:32.120 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:31 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:32.120 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:31 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:32.120 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:31 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:32.120 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:31 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:32.121 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:31 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:32.121 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:31 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:32.121 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:31 smithi174 ceph-mon[41061]: Adjusting osd_memory_target on smithi114 to 1216M 2024-02-19T19:02:32.121 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:31 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:32.121 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:31 smithi174 ceph-mon[41061]: pgmap v9: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:02:32.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:31 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:32.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:31 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:32.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:31 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:32.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:31 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:32.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:31 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:32.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:31 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:32.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:31 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:32.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:31 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:32.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:31 smithi114 ceph-mon[29441]: Adjusting osd_memory_target on smithi114 to 1216M 2024-02-19T19:02:32.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:31 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:32.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:31 smithi114 ceph-mon[29441]: pgmap v9: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:02:33.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:33.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:33.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:33.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:33.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:33.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:33.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:33.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:33.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: Adjusting osd_memory_target on smithi174 to 2240M 2024-02-19T19:02:33.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:33.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:33.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:33.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: Updating smithi114:/etc/ceph/ceph.conf 2024-02-19T19:02:33.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: Updating smithi174:/etc/ceph/ceph.conf 2024-02-19T19:02:33.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:02:33.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:02:33.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:02:33.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:02:33.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:33.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:33 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:33.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:33.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:33.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:02:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: Adjusting osd_memory_target on smithi174 to 2240M 2024-02-19T19:02:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: Updating smithi114:/etc/ceph/ceph.conf 2024-02-19T19:02:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: Updating smithi174:/etc/ceph/ceph.conf 2024-02-19T19:02:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:02:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:02:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:02:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:02:33.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:33.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:33 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:34.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:34 smithi114 ceph-mon[29441]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:02:34.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:34 smithi114 ceph-mon[29441]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:02:34.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:34 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:34.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:34 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:34.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:34 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:34.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:34 smithi114 ceph-mon[29441]: Reconfiguring prometheus.smithi114 (dependencies changed)... 2024-02-19T19:02:34.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:34 smithi114 ceph-mon[29441]: pgmap v10: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:02:34.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:34 smithi114 ceph-mon[29441]: Reconfiguring daemon prometheus.smithi114 on smithi114 2024-02-19T19:02:34.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:34 smithi174 ceph-mon[41061]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:02:34.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:34 smithi174 ceph-mon[41061]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:02:34.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:34 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:34.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:34 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:34.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:34 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:34.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:34 smithi174 ceph-mon[41061]: Reconfiguring prometheus.smithi114 (dependencies changed)... 2024-02-19T19:02:34.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:34 smithi174 ceph-mon[41061]: pgmap v10: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:02:34.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:34 smithi174 ceph-mon[41061]: Reconfiguring daemon prometheus.smithi114 on smithi114 2024-02-19T19:02:34.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.697+0000 7fcf9d6b3700 1 -- 172.21.15.114:0/216358038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf98100cb0 msgr2=0x7fcf98101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:34.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.697+0000 7fcf9d6b3700 1 --2- 172.21.15.114:0/216358038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf98100cb0 0x7fcf98101110 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009910 tx=0x7fcf8c009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:34.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.697+0000 7fcf9d6b3700 1 -- 172.21.15.114:0/216358038 shutdown_connections 2024-02-19T19:02:34.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.697+0000 7fcf9d6b3700 1 --2- 172.21.15.114:0/216358038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf98100cb0 0x7fcf98101110 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:34.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.697+0000 7fcf9d6b3700 1 --2- 172.21.15.114:0/216358038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf980ffab0 0x7fcf980ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:34.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.697+0000 7fcf9d6b3700 1 -- 172.21.15.114:0/216358038 >> 172.21.15.114:0/216358038 conn(0x7fcf980fb030 msgr2=0x7fcf980fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:34.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.697+0000 7fcf9d6b3700 1 -- 172.21.15.114:0/216358038 shutdown_connections 2024-02-19T19:02:34.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.697+0000 7fcf9d6b3700 1 -- 172.21.15.114:0/216358038 wait complete. 2024-02-19T19:02:34.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.698+0000 7fcf9d6b3700 1 Processor -- start 2024-02-19T19:02:34.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.698+0000 7fcf9d6b3700 1 -- start start 2024-02-19T19:02:34.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.698+0000 7fcf9d6b3700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf980ffab0 0x7fcf981963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:34.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.698+0000 7fcf9d6b3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf98100cb0 0x7fcf98196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:34.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.698+0000 7fcf9d6b3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf98196f40 con 0x7fcf98100cb0 2024-02-19T19:02:34.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.698+0000 7fcf9d6b3700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf98197080 con 0x7fcf980ffab0 2024-02-19T19:02:34.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.698+0000 7fcf967fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf98100cb0 0x7fcf98196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:34.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.698+0000 7fcf967fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf98100cb0 0x7fcf98196920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53154/0 (socket says 172.21.15.114:53154) 2024-02-19T19:02:34.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.698+0000 7fcf967fc700 1 -- 172.21.15.114:0/980158840 learned_addr learned my addr 172.21.15.114:0/980158840 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:02:34.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.698+0000 7fcf96ffd700 1 --2- 172.21.15.114:0/980158840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf980ffab0 0x7fcf981963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:34.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.699+0000 7fcf967fc700 1 -- 172.21.15.114:0/980158840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf980ffab0 msgr2=0x7fcf981963e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:34.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.699+0000 7fcf967fc700 1 --2- 172.21.15.114:0/980158840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf980ffab0 0x7fcf981963e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:34.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.699+0000 7fcf967fc700 1 -- 172.21.15.114:0/980158840 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8c0095f0 con 0x7fcf98100cb0 2024-02-19T19:02:34.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.699+0000 7fcf967fc700 1 --2- 172.21.15.114:0/980158840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf98100cb0 0x7fcf98196920 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c006010 tx=0x7fcf8c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:34.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.699+0000 7fcf7ffff700 1 -- 172.21.15.114:0/980158840 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf8c01d070 con 0x7fcf98100cb0 2024-02-19T19:02:34.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.699+0000 7fcf9d6b3700 1 -- 172.21.15.114:0/980158840 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9819bad0 con 0x7fcf98100cb0 2024-02-19T19:02:34.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.699+0000 7fcf7ffff700 1 -- 172.21.15.114:0/980158840 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcf8c00f460 con 0x7fcf98100cb0 2024-02-19T19:02:34.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.699+0000 7fcf7ffff700 1 -- 172.21.15.114:0/980158840 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf8c021640 con 0x7fcf98100cb0 2024-02-19T19:02:34.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.699+0000 7fcf9d6b3700 1 -- 172.21.15.114:0/980158840 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9819bfc0 con 0x7fcf98100cb0 2024-02-19T19:02:34.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.700+0000 7fcf9d6b3700 1 -- 172.21.15.114:0/980158840 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf981097b0 con 0x7fcf98100cb0 2024-02-19T19:02:34.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.701+0000 7fcf7ffff700 1 -- 172.21.15.114:0/980158840 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) v1 ==== 90933+0+0 (secure 0 0 0) 0x7fcf8c00fab0 con 0x7fcf98100cb0 2024-02-19T19:02:34.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.701+0000 7fcf7ffff700 1 --2- 172.21.15.114:0/980158840 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7fcf8006ce40 0x7fcf8006f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:34.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.701+0000 7fcf7ffff700 1 -- 172.21.15.114:0/980158840 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fcf8c08da20 con 0x7fcf98100cb0 2024-02-19T19:02:34.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.701+0000 7fcf96ffd700 1 --2- 172.21.15.114:0/980158840 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7fcf8006ce40 0x7fcf8006f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:34.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.702+0000 7fcf96ffd700 1 --2- 172.21.15.114:0/980158840 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7fcf8006ce40 0x7fcf8006f300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcf98100b10 tx=0x7fcf88009190 comp rx=0 tx=0).ready entity=mgr.24391 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:34.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.706+0000 7fcf7ffff700 1 -- 172.21.15.114:0/980158840 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7fcf8c05b160 con 0x7fcf98100cb0 2024-02-19T19:02:34.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.916+0000 7fcf9d6b3700 1 -- 172.21.15.114:0/980158840 --> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf98105600 con 0x7fcf8006ce40 2024-02-19T19:02:34.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.918+0000 7fcf7ffff700 1 -- 172.21.15.114:0/980158840 <== mgr.24391 v2:172.21.15.174:6828/2185377847 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7fcf98105600 con 0x7fcf8006ce40 2024-02-19T19:02:34.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.920+0000 7fcf9d6b3700 1 -- 172.21.15.114:0/980158840 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7fcf8006ce40 msgr2=0x7fcf8006f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:34.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.920+0000 7fcf9d6b3700 1 --2- 172.21.15.114:0/980158840 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7fcf8006ce40 0x7fcf8006f300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcf98100b10 tx=0x7fcf88009190 comp rx=0 tx=0).stop 2024-02-19T19:02:34.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.920+0000 7fcf9d6b3700 1 -- 172.21.15.114:0/980158840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf98100cb0 msgr2=0x7fcf98196920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:34.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.920+0000 7fcf9d6b3700 1 --2- 172.21.15.114:0/980158840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf98100cb0 0x7fcf98196920 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c006010 tx=0x7fcf8c00bd70 comp rx=0 tx=0).stop 2024-02-19T19:02:34.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.920+0000 7fcf9d6b3700 1 -- 172.21.15.114:0/980158840 shutdown_connections 2024-02-19T19:02:34.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.920+0000 7fcf9d6b3700 1 --2- 172.21.15.114:0/980158840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf980ffab0 0x7fcf981963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:34.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.920+0000 7fcf9d6b3700 1 --2- 172.21.15.114:0/980158840 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7fcf8006ce40 0x7fcf8006f300 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:34.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.920+0000 7fcf9d6b3700 1 --2- 172.21.15.114:0/980158840 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf98100cb0 0x7fcf98196920 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:34.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.920+0000 7fcf9d6b3700 1 -- 172.21.15.114:0/980158840 >> 172.21.15.114:0/980158840 conn(0x7fcf980fb030 msgr2=0x7fcf98103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:34.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.920+0000 7fcf9d6b3700 1 -- 172.21.15.114:0/980158840 shutdown_connections 2024-02-19T19:02:34.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:34.920+0000 7fcf9d6b3700 1 -- 172.21.15.114:0/980158840 wait complete. 2024-02-19T19:02:34.950 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-02-19T19:02:35.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.027+0000 7efcf8c9c700 1 -- 172.21.15.114:0/1443209263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcf40ff440 msgr2=0x7efcf4101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:35.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.027+0000 7efcf8c9c700 1 --2- 172.21.15.114:0/1443209263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcf40ff440 0x7efcf4101870 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7efce8009910 tx=0x7efce8009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:35.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.027+0000 7efcf8c9c700 1 -- 172.21.15.114:0/1443209263 shutdown_connections 2024-02-19T19:02:35.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.027+0000 7efcf8c9c700 1 --2- 172.21.15.114:0/1443209263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcf4101db0 0x7efcf4104240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:35.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.027+0000 7efcf8c9c700 1 --2- 172.21.15.114:0/1443209263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcf40ff440 0x7efcf4101870 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:35.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.027+0000 7efcf8c9c700 1 -- 172.21.15.114:0/1443209263 >> 172.21.15.114:0/1443209263 conn(0x7efcf40fb060 msgr2=0x7efcf40fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:35.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.027+0000 7efcf8c9c700 1 -- 172.21.15.114:0/1443209263 shutdown_connections 2024-02-19T19:02:35.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.027+0000 7efcf8c9c700 1 -- 172.21.15.114:0/1443209263 wait complete. 2024-02-19T19:02:35.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.027+0000 7efcf8c9c700 1 Processor -- start 2024-02-19T19:02:35.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efcf8c9c700 1 -- start start 2024-02-19T19:02:35.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efcf8c9c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcf40ff440 0x7efcf419c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:35.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efcf8c9c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcf4101db0 0x7efcf419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:35.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efcf8c9c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf419d4f0 con 0x7efcf40ff440 2024-02-19T19:02:35.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efcf8c9c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf419d630 con 0x7efcf4101db0 2024-02-19T19:02:35.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efcf259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcf40ff440 0x7efcf419c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:35.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efcf259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcf40ff440 0x7efcf419c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53174/0 (socket says 172.21.15.114:53174) 2024-02-19T19:02:35.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efcf259c700 1 -- 172.21.15.114:0/3549276797 learned_addr learned my addr 172.21.15.114:0/3549276797 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:02:35.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efcf1d9b700 1 --2- 172.21.15.114:0/3549276797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcf4101db0 0x7efcf419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:35.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efcf259c700 1 -- 172.21.15.114:0/3549276797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcf4101db0 msgr2=0x7efcf419ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:35.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efcf259c700 1 --2- 172.21.15.114:0/3549276797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcf4101db0 0x7efcf419ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:35.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efcf259c700 1 -- 172.21.15.114:0/3549276797 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce80095f0 con 0x7efcf40ff440 2024-02-19T19:02:35.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efcf259c700 1 --2- 172.21.15.114:0/3549276797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcf40ff440 0x7efcf419c990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7efce8006010 tx=0x7efce800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:35.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efce37fe700 1 -- 172.21.15.114:0/3549276797 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efce801d070 con 0x7efcf40ff440 2024-02-19T19:02:35.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efcf8c9c700 1 -- 172.21.15.114:0/3549276797 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf41a2080 con 0x7efcf40ff440 2024-02-19T19:02:35.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efce37fe700 1 -- 172.21.15.114:0/3549276797 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efce800f460 con 0x7efcf40ff440 2024-02-19T19:02:35.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efce37fe700 1 -- 172.21.15.114:0/3549276797 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efce8017630 con 0x7efcf40ff440 2024-02-19T19:02:35.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.028+0000 7efcf8c9c700 1 -- 172.21.15.114:0/3549276797 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf41a2570 con 0x7efcf40ff440 2024-02-19T19:02:35.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.029+0000 7efcf8c9c700 1 -- 172.21.15.114:0/3549276797 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf4196b20 con 0x7efcf40ff440 2024-02-19T19:02:35.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.030+0000 7efce37fe700 1 -- 172.21.15.114:0/3549276797 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) v1 ==== 90933+0+0 (secure 0 0 0) 0x7efce8003a60 con 0x7efcf40ff440 2024-02-19T19:02:35.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.030+0000 7efce37fe700 1 --2- 172.21.15.114:0/3549276797 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7efcdc06ce40 0x7efcdc06f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:35.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.030+0000 7efce37fe700 1 -- 172.21.15.114:0/3549276797 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7efce808dad0 con 0x7efcf40ff440 2024-02-19T19:02:35.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.031+0000 7efcf1d9b700 1 --2- 172.21.15.114:0/3549276797 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7efcdc06ce40 0x7efcdc06f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:35.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.031+0000 7efcf1d9b700 1 --2- 172.21.15.114:0/3549276797 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7efcdc06ce40 0x7efcdc06f300 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efce4005950 tx=0x7efce40058e0 comp rx=0 tx=0).ready entity=mgr.24391 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:35.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.035+0000 7efce37fe700 1 -- 172.21.15.114:0/3549276797 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7efce805b290 con 0x7efcf40ff440 2024-02-19T19:02:35.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.250+0000 7efcf8c9c700 1 -- 172.21.15.114:0/3549276797 --> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efcf40611d0 con 0x7efcdc06ce40 2024-02-19T19:02:35.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.252+0000 7efce37fe700 1 -- 172.21.15.114:0/3549276797 <== mgr.24391 v2:172.21.15.174:6828/2185377847 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7efcf40611d0 con 0x7efcdc06ce40 2024-02-19T19:02:35.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.254+0000 7efcf8c9c700 1 -- 172.21.15.114:0/3549276797 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7efcdc06ce40 msgr2=0x7efcdc06f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:35.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.254+0000 7efcf8c9c700 1 --2- 172.21.15.114:0/3549276797 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7efcdc06ce40 0x7efcdc06f300 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efce4005950 tx=0x7efce40058e0 comp rx=0 tx=0).stop 2024-02-19T19:02:35.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.254+0000 7efcf8c9c700 1 -- 172.21.15.114:0/3549276797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcf40ff440 msgr2=0x7efcf419c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:35.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.254+0000 7efcf8c9c700 1 --2- 172.21.15.114:0/3549276797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcf40ff440 0x7efcf419c990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7efce8006010 tx=0x7efce800be20 comp rx=0 tx=0).stop 2024-02-19T19:02:35.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.254+0000 7efcf8c9c700 1 -- 172.21.15.114:0/3549276797 shutdown_connections 2024-02-19T19:02:35.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.254+0000 7efcf8c9c700 1 --2- 172.21.15.114:0/3549276797 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7efcdc06ce40 0x7efcdc06f300 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:35.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.254+0000 7efcf8c9c700 1 --2- 172.21.15.114:0/3549276797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcf40ff440 0x7efcf419c990 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:35.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.254+0000 7efcf8c9c700 1 --2- 172.21.15.114:0/3549276797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcf4101db0 0x7efcf419ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:35.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.254+0000 7efcf8c9c700 1 -- 172.21.15.114:0/3549276797 >> 172.21.15.114:0/3549276797 conn(0x7efcf40fb060 msgr2=0x7efcf40fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:35.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.255+0000 7efcf8c9c700 1 -- 172.21.15.114:0/3549276797 shutdown_connections 2024-02-19T19:02:35.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.255+0000 7efcf8c9c700 1 -- 172.21.15.114:0/3549276797 wait complete. 2024-02-19T19:02:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.360+0000 7f8f4c705700 1 -- 172.21.15.114:0/544509314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f441016c0 msgr2=0x7f8f44103ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.360+0000 7f8f4c705700 1 --2- 172.21.15.114:0/544509314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f441016c0 0x7f8f44103ab0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009910 tx=0x7f8f40009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.360+0000 7f8f4c705700 1 -- 172.21.15.114:0/544509314 shutdown_connections 2024-02-19T19:02:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.360+0000 7f8f4c705700 1 --2- 172.21.15.114:0/544509314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f44103ff0 0x7f8f441063e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.360+0000 7f8f4c705700 1 --2- 172.21.15.114:0/544509314 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f441016c0 0x7f8f44103ab0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.360+0000 7f8f4c705700 1 -- 172.21.15.114:0/544509314 >> 172.21.15.114:0/544509314 conn(0x7f8f440fb030 msgr2=0x7f8f440fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.360+0000 7f8f4c705700 1 -- 172.21.15.114:0/544509314 shutdown_connections 2024-02-19T19:02:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.360+0000 7f8f4c705700 1 -- 172.21.15.114:0/544509314 wait complete. 2024-02-19T19:02:35.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.360+0000 7f8f4c705700 1 Processor -- start 2024-02-19T19:02:35.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.360+0000 7f8f4c705700 1 -- start start 2024-02-19T19:02:35.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.361+0000 7f8f4c705700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f441016c0 0x7f8f44196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:35.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.361+0000 7f8f4c705700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f44103ff0 0x7f8f441967d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:35.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.361+0000 7f8f4c705700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f44196df0 con 0x7f8f44103ff0 2024-02-19T19:02:35.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.361+0000 7f8f4c705700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f44196f30 con 0x7f8f441016c0 2024-02-19T19:02:35.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.361+0000 7f8f49ca0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f44103ff0 0x7f8f441967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:35.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.361+0000 7f8f49ca0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f44103ff0 0x7f8f441967d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53196/0 (socket says 172.21.15.114:53196) 2024-02-19T19:02:35.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.361+0000 7f8f49ca0700 1 -- 172.21.15.114:0/1809724460 learned_addr learned my addr 172.21.15.114:0/1809724460 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:02:35.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.361+0000 7f8f4a4a1700 1 --2- 172.21.15.114:0/1809724460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f441016c0 0x7f8f44196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:35.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.361+0000 7f8f49ca0700 1 -- 172.21.15.114:0/1809724460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f441016c0 msgr2=0x7f8f44196290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:35.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.361+0000 7f8f49ca0700 1 --2- 172.21.15.114:0/1809724460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f441016c0 0x7f8f44196290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:35.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.361+0000 7f8f49ca0700 1 -- 172.21.15.114:0/1809724460 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f400095f0 con 0x7f8f44103ff0 2024-02-19T19:02:35.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.361+0000 7f8f49ca0700 1 --2- 172.21.15.114:0/1809724460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f44103ff0 0x7f8f441967d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8f3400e920 tx=0x7f8f3400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:35.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.361+0000 7f8f3b7fe700 1 -- 172.21.15.114:0/1809724460 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f3400c9d0 con 0x7f8f44103ff0 2024-02-19T19:02:35.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.361+0000 7f8f3b7fe700 1 -- 172.21.15.114:0/1809724460 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8f34010430 con 0x7f8f44103ff0 2024-02-19T19:02:35.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.361+0000 7f8f4c705700 1 -- 172.21.15.114:0/1809724460 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f4419b9e0 con 0x7f8f44103ff0 2024-02-19T19:02:35.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.362+0000 7f8f3b7fe700 1 -- 172.21.15.114:0/1809724460 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f34018650 con 0x7f8f44103ff0 2024-02-19T19:02:35.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.362+0000 7f8f4c705700 1 -- 172.21.15.114:0/1809724460 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4419bf30 con 0x7f8f44103ff0 2024-02-19T19:02:35.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.363+0000 7f8f4c705700 1 -- 172.21.15.114:0/1809724460 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f44190520 con 0x7f8f44103ff0 2024-02-19T19:02:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.363+0000 7f8f3b7fe700 1 -- 172.21.15.114:0/1809724460 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) v1 ==== 90933+0+0 (secure 0 0 0) 0x7f8f340105a0 con 0x7f8f44103ff0 2024-02-19T19:02:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.363+0000 7f8f3b7fe700 1 --2- 172.21.15.114:0/1809724460 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f8f3006ce40 0x7f8f3006f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.364+0000 7f8f3b7fe700 1 -- 172.21.15.114:0/1809724460 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f8f34014070 con 0x7f8f44103ff0 2024-02-19T19:02:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.364+0000 7f8f4a4a1700 1 --2- 172.21.15.114:0/1809724460 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f8f3006ce40 0x7f8f3006f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:35.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.364+0000 7f8f4a4a1700 1 --2- 172.21.15.114:0/1809724460 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f8f3006ce40 0x7f8f3006f300 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009fd0 tx=0x7f8f40005940 comp rx=0 tx=0).ready entity=mgr.24391 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:35.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.368+0000 7f8f3b7fe700 1 -- 172.21.15.114:0/1809724460 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f8f34052a70 con 0x7f8f44103ff0 2024-02-19T19:02:35.524 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:35 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:35.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.578+0000 7f8f4c705700 1 -- 172.21.15.114:0/1809724460 --> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f440611d0 con 0x7f8f3006ce40 2024-02-19T19:02:35.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.591+0000 7f8f3b7fe700 1 -- 172.21.15.114:0/1809724460 <== mgr.24391 v2:172.21.15.174:6828/2185377847 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f8f440611d0 con 0x7f8f3006ce40 2024-02-19T19:02:35.609 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:02:35.609 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (6m) 5s ago 9m 26.4M - 0.25.0 c8568f914cd2 30444b676ca4 2024-02-19T19:02:35.609 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (9m) 5s ago 9m 7750k - 18.2.1 94eadd990e0a 4918cd87e312 2024-02-19T19:02:35.610 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi174 smithi174 running (4m) 7s ago 7m 7608k - 18.2.1 94eadd990e0a 516c34431b93 2024-02-19T19:02:35.610 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (9m) 5s ago 9m 6660k - 18.2.1 94eadd990e0a ecfcd98907ba 2024-02-19T19:02:35.610 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi174 smithi174 running (7m) 7s ago 7m 6660k - 18.2.1 94eadd990e0a de88985f5a07 2024-02-19T19:02:35.610 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (6m) 5s ago 8m 86.1M - 9.4.7 954c08fa6188 478bd533e827 2024-02-19T19:02:35.610 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tcwpod smithi114 running (2m) 5s ago 2m 26.3M - 18.2.1 94eadd990e0a 420175777cf4 2024-02-19T19:02:35.610 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.vpoeop smithi114 running (2m) 5s ago 2m 18.6M - 18.2.1 94eadd990e0a f81a79e53ead 2024-02-19T19:02:35.610 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.stnfnf smithi174 running (2m) 7s ago 2m 19.6M - 18.2.1 94eadd990e0a 3df35954a492 2024-02-19T19:02:35.610 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.znhuvd smithi174 running (2m) 7s ago 2m 19.4M - 18.2.1 94eadd990e0a 2d2267494ae1 2024-02-19T19:02:35.610 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.pcvtrk smithi114 *:9283,8765,8443 running (10m) 5s ago 10m 399M - 18.2.1 94eadd990e0a 9d81af39d736 2024-02-19T19:02:35.610 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi174.lydvnj smithi174 *:8443,9283,8765 running (24s) 7s ago 7m 487M - 19.0.0-1450-gf78a58c0 f081f847070a 3c0ee4f00329 2024-02-19T19:02:35.610 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (10m) 5s ago 11m 49.0M 2048M 18.2.1 94eadd990e0a 41468f811fc0 2024-02-19T19:02:35.610 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi174 smithi174 running (6m) 7s ago 6m 38.6M 2048M 18.2.1 94eadd990e0a 70dd7bf69fd9 2024-02-19T19:02:35.610 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (9m) 5s ago 9m 14.6M - 1.5.0 0da6a335fe13 b6b258074cce 2024-02-19T19:02:35.610 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi174 smithi174 *:9100 running (7m) 7s ago 7m 15.5M - 1.5.0 0da6a335fe13 d0b4fdf67b22 2024-02-19T19:02:35.610 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (6m) 5s ago 6m 47.8M 1216M 18.2.1 94eadd990e0a 990afc982b6c 2024-02-19T19:02:35.611 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (5m) 5s ago 5m 48.1M 1216M 18.2.1 94eadd990e0a 932e3ad37db5 2024-02-19T19:02:35.611 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (5m) 5s ago 5m 50.2M 1216M 18.2.1 94eadd990e0a 12953988af86 2024-02-19T19:02:35.611 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi174 running (4m) 7s ago 4m 46.7M 2240M 18.2.1 94eadd990e0a 979f45cabbce 2024-02-19T19:02:35.611 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi174 running (4m) 7s ago 4m 48.2M 2240M 18.2.1 94eadd990e0a 8b8c3745b8c5 2024-02-19T19:02:35.611 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi174 running (3m) 7s ago 3m 50.5M 2240M 18.2.1 94eadd990e0a 5771a995ed75 2024-02-19T19:02:35.611 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (6m) 5s ago 7m 59.5M - 2.43.0 a07b618ecd1d 52087f218c92 2024-02-19T19:02:35.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.593+0000 7f8f4c705700 1 -- 172.21.15.114:0/1809724460 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f8f3006ce40 msgr2=0x7f8f3006f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:35.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.593+0000 7f8f4c705700 1 --2- 172.21.15.114:0/1809724460 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f8f3006ce40 0x7f8f3006f300 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009fd0 tx=0x7f8f40005940 comp rx=0 tx=0).stop 2024-02-19T19:02:35.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.593+0000 7f8f4c705700 1 -- 172.21.15.114:0/1809724460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f44103ff0 msgr2=0x7f8f441967d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:35.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.593+0000 7f8f4c705700 1 --2- 172.21.15.114:0/1809724460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f44103ff0 0x7f8f441967d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8f3400e920 tx=0x7f8f3400ec30 comp rx=0 tx=0).stop 2024-02-19T19:02:35.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.593+0000 7f8f4c705700 1 -- 172.21.15.114:0/1809724460 shutdown_connections 2024-02-19T19:02:35.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.593+0000 7f8f4c705700 1 --2- 172.21.15.114:0/1809724460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f441016c0 0x7f8f44196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:35.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.593+0000 7f8f4c705700 1 --2- 172.21.15.114:0/1809724460 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f8f3006ce40 0x7f8f3006f300 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:35.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.593+0000 7f8f4c705700 1 --2- 172.21.15.114:0/1809724460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f44103ff0 0x7f8f441967d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:35.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.593+0000 7f8f4c705700 1 -- 172.21.15.114:0/1809724460 >> 172.21.15.114:0/1809724460 conn(0x7f8f440fb030 msgr2=0x7f8f440fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:35.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.593+0000 7f8f4c705700 1 -- 172.21.15.114:0/1809724460 shutdown_connections 2024-02-19T19:02:35.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.593+0000 7f8f4c705700 1 -- 172.21.15.114:0/1809724460 wait complete. 2024-02-19T19:02:35.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:35 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:35.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.700+0000 7f50cb381700 1 -- 172.21.15.114:0/1114655325 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c4073790 msgr2=0x7f50c4073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:35.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.700+0000 7f50cb381700 1 --2- 172.21.15.114:0/1114655325 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c4073790 0x7f50c4073c10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f50b4009910 tx=0x7f50b4009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:35.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.700+0000 7f50cb381700 1 -- 172.21.15.114:0/1114655325 shutdown_connections 2024-02-19T19:02:35.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.700+0000 7f50cb381700 1 --2- 172.21.15.114:0/1114655325 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c4073790 0x7f50c4073c10 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:35.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.700+0000 7f50cb381700 1 --2- 172.21.15.114:0/1114655325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c4074d60 0x7f50c40731c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:35.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.700+0000 7f50cb381700 1 -- 172.21.15.114:0/1114655325 >> 172.21.15.114:0/1114655325 conn(0x7f50c40fc140 msgr2=0x7f50c40fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:35.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.700+0000 7f50cb381700 1 -- 172.21.15.114:0/1114655325 shutdown_connections 2024-02-19T19:02:35.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.700+0000 7f50cb381700 1 -- 172.21.15.114:0/1114655325 wait complete. 2024-02-19T19:02:35.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.700+0000 7f50cb381700 1 Processor -- start 2024-02-19T19:02:35.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.700+0000 7f50cb381700 1 -- start start 2024-02-19T19:02:35.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50cb381700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c4073790 0x7f50c41985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:35.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50cb381700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c4074d60 0x7f50c4198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:35.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50cb381700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c4199140 con 0x7f50c4074d60 2024-02-19T19:02:35.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50cb381700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c4199280 con 0x7f50c4073790 2024-02-19T19:02:35.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50c891c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c4074d60 0x7f50c4198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:35.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50c891c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c4074d60 0x7f50c4198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50740/0 (socket says 172.21.15.114:50740) 2024-02-19T19:02:35.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50c891c700 1 -- 172.21.15.114:0/1521003230 learned_addr learned my addr 172.21.15.114:0/1521003230 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:02:35.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50c911d700 1 --2- 172.21.15.114:0/1521003230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c4073790 0x7f50c41985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:35.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50c891c700 1 -- 172.21.15.114:0/1521003230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c4073790 msgr2=0x7f50c41985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:35.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50c891c700 1 --2- 172.21.15.114:0/1521003230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c4073790 0x7f50c41985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:35.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50c891c700 1 -- 172.21.15.114:0/1521003230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b40095f0 con 0x7f50c4074d60 2024-02-19T19:02:35.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50c911d700 1 --2- 172.21.15.114:0/1521003230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c4073790 0x7f50c41985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:02:35.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50c891c700 1 --2- 172.21.15.114:0/1521003230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c4074d60 0x7f50c4198b20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f50b4006010 tx=0x7f50b400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:35.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50ba7fc700 1 -- 172.21.15.114:0/1521003230 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50b401d070 con 0x7f50c4074d60 2024-02-19T19:02:35.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50ba7fc700 1 -- 172.21.15.114:0/1521003230 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f50b400f460 con 0x7f50c4074d60 2024-02-19T19:02:35.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50cb381700 1 -- 172.21.15.114:0/1521003230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c419dcd0 con 0x7f50c4074d60 2024-02-19T19:02:35.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50ba7fc700 1 -- 172.21.15.114:0/1521003230 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50b4021640 con 0x7f50c4074d60 2024-02-19T19:02:35.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.701+0000 7f50cb381700 1 -- 172.21.15.114:0/1521003230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c419e240 con 0x7f50c4074d60 2024-02-19T19:02:35.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.702+0000 7f50cb381700 1 -- 172.21.15.114:0/1521003230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c410b990 con 0x7f50c4074d60 2024-02-19T19:02:35.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.703+0000 7f50ba7fc700 1 -- 172.21.15.114:0/1521003230 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) v1 ==== 90933+0+0 (secure 0 0 0) 0x7f50b400fab0 con 0x7f50c4074d60 2024-02-19T19:02:35.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.703+0000 7f50ba7fc700 1 --2- 172.21.15.114:0/1521003230 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f50b006ce40 0x7f50b006f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:35.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.703+0000 7f50ba7fc700 1 -- 172.21.15.114:0/1521003230 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f50b408db00 con 0x7f50c4074d60 2024-02-19T19:02:35.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.703+0000 7f50c911d700 1 --2- 172.21.15.114:0/1521003230 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f50b006ce40 0x7f50b006f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:35.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.704+0000 7f50c911d700 1 --2- 172.21.15.114:0/1521003230 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f50b006ce40 0x7f50b006f300 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f50c00099d0 tx=0x7f50c0008040 comp rx=0 tx=0).ready entity=mgr.24391 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:35.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.708+0000 7f50ba7fc700 1 -- 172.21.15.114:0/1521003230 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f50b405b240 con 0x7f50c4074d60 2024-02-19T19:02:36.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.986+0000 7f50cb381700 1 -- 172.21.15.114:0/1521003230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f50c419e910 con 0x7f50c4074d60 2024-02-19T19:02:36.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.987+0000 7f50ba7fc700 1 -- 172.21.15.114:0/1521003230 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+766 (secure 0 0 0) 0x7f50b405add0 con 0x7f50c4074d60 2024-02-19T19:02:36.006 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:02:36.006 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-02-19T19:02:36.006 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:02:36.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:02:36.007 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-02-19T19:02:36.007 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-02-19T19:02:36.007 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 1 2024-02-19T19:02:36.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:02:36.007 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-02-19T19:02:36.007 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-02-19T19:02:36.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:02:36.007 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-02-19T19:02:36.007 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:02:36.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:02:36.007 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-02-19T19:02:36.007 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 13, 2024-02-19T19:02:36.007 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 1 2024-02-19T19:02:36.008 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-02-19T19:02:36.008 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:02:36.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.989+0000 7f50cb381700 1 -- 172.21.15.114:0/1521003230 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f50b006ce40 msgr2=0x7f50b006f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:36.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.989+0000 7f50cb381700 1 --2- 172.21.15.114:0/1521003230 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f50b006ce40 0x7f50b006f300 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f50c00099d0 tx=0x7f50c0008040 comp rx=0 tx=0).stop 2024-02-19T19:02:36.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.989+0000 7f50cb381700 1 -- 172.21.15.114:0/1521003230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c4074d60 msgr2=0x7f50c4198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:36.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.989+0000 7f50cb381700 1 --2- 172.21.15.114:0/1521003230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c4074d60 0x7f50c4198b20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f50b4006010 tx=0x7f50b400be20 comp rx=0 tx=0).stop 2024-02-19T19:02:36.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.989+0000 7f50cb381700 1 -- 172.21.15.114:0/1521003230 shutdown_connections 2024-02-19T19:02:36.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.989+0000 7f50cb381700 1 --2- 172.21.15.114:0/1521003230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c4073790 0x7f50c41985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.989+0000 7f50cb381700 1 --2- 172.21.15.114:0/1521003230 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f50b006ce40 0x7f50b006f300 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.989+0000 7f50cb381700 1 --2- 172.21.15.114:0/1521003230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50c4074d60 0x7f50c4198b20 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.989+0000 7f50cb381700 1 -- 172.21.15.114:0/1521003230 >> 172.21.15.114:0/1521003230 conn(0x7f50c40fc140 msgr2=0x7f50c4106a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:36.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.989+0000 7f50cb381700 1 -- 172.21.15.114:0/1521003230 shutdown_connections 2024-02-19T19:02:36.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:35.989+0000 7f50cb381700 1 -- 172.21.15.114:0/1521003230 wait complete. 2024-02-19T19:02:36.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.094+0000 7f47511a6700 1 -- 172.21.15.114:0/2744955432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c100cb0 msgr2=0x7f474c101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:36.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.094+0000 7f47511a6700 1 --2- 172.21.15.114:0/2744955432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c100cb0 0x7f474c101110 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f473c009910 tx=0x7f473c009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:36.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.094+0000 7f47511a6700 1 -- 172.21.15.114:0/2744955432 shutdown_connections 2024-02-19T19:02:36.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.094+0000 7f47511a6700 1 --2- 172.21.15.114:0/2744955432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c100cb0 0x7f474c101110 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.094+0000 7f47511a6700 1 --2- 172.21.15.114:0/2744955432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f474c0ffab0 0x7f474c0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.094+0000 7f47511a6700 1 -- 172.21.15.114:0/2744955432 >> 172.21.15.114:0/2744955432 conn(0x7f474c0fb050 msgr2=0x7f474c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:36.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.094+0000 7f47511a6700 1 -- 172.21.15.114:0/2744955432 shutdown_connections 2024-02-19T19:02:36.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.094+0000 7f47511a6700 1 -- 172.21.15.114:0/2744955432 wait complete. 2024-02-19T19:02:36.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.095+0000 7f47511a6700 1 Processor -- start 2024-02-19T19:02:36.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.095+0000 7f47511a6700 1 -- start start 2024-02-19T19:02:36.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.095+0000 7f47511a6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c0ffab0 0x7f474c069820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:36.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.095+0000 7f47511a6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f474c100cb0 0x7f474c19b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:36.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.095+0000 7f47511a6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c19b670 con 0x7f474c0ffab0 2024-02-19T19:02:36.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.095+0000 7f47511a6700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c19b7e0 con 0x7f474c100cb0 2024-02-19T19:02:36.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.095+0000 7f474ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c0ffab0 0x7f474c069820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:36.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.095+0000 7f474ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c0ffab0 0x7f474c069820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50754/0 (socket says 172.21.15.114:50754) 2024-02-19T19:02:36.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.095+0000 7f474ad9d700 1 -- 172.21.15.114:0/2830661271 learned_addr learned my addr 172.21.15.114:0/2830661271 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:02:36.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.095+0000 7f474a59c700 1 --2- 172.21.15.114:0/2830661271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f474c100cb0 0x7f474c19b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:36.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.096+0000 7f474ad9d700 1 -- 172.21.15.114:0/2830661271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f474c100cb0 msgr2=0x7f474c19b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:36.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.096+0000 7f474ad9d700 1 --2- 172.21.15.114:0/2830661271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f474c100cb0 0x7f474c19b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.096+0000 7f474ad9d700 1 -- 172.21.15.114:0/2830661271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473c0095f0 con 0x7f474c0ffab0 2024-02-19T19:02:36.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.096+0000 7f474ad9d700 1 --2- 172.21.15.114:0/2830661271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c0ffab0 0x7f474c069820 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f474000d900 tx=0x7f474000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:36.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.096+0000 7f4733fff700 1 -- 172.21.15.114:0/2830661271 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f474000b920 con 0x7f474c0ffab0 2024-02-19T19:02:36.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.096+0000 7f47511a6700 1 -- 172.21.15.114:0/2830661271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f474c19bac0 con 0x7f474c0ffab0 2024-02-19T19:02:36.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.096+0000 7f4733fff700 1 -- 172.21.15.114:0/2830661271 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4740010460 con 0x7f474c0ffab0 2024-02-19T19:02:36.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.096+0000 7f4733fff700 1 -- 172.21.15.114:0/2830661271 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f474000f590 con 0x7f474c0ffab0 2024-02-19T19:02:36.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.096+0000 7f47511a6700 1 -- 172.21.15.114:0/2830661271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f474c19c010 con 0x7f474c0ffab0 2024-02-19T19:02:36.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.097+0000 7f47511a6700 1 -- 172.21.15.114:0/2830661271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f474c108610 con 0x7f474c0ffab0 2024-02-19T19:02:36.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.097+0000 7f4733fff700 1 -- 172.21.15.114:0/2830661271 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) v1 ==== 90933+0+0 (secure 0 0 0) 0x7f474000f850 con 0x7f474c0ffab0 2024-02-19T19:02:36.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.098+0000 7f4733fff700 1 --2- 172.21.15.114:0/2830661271 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f473406ce40 0x7f473406f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:36.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.098+0000 7f4733fff700 1 -- 172.21.15.114:0/2830661271 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f474008c320 con 0x7f474c0ffab0 2024-02-19T19:02:36.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.098+0000 7f474a59c700 1 --2- 172.21.15.114:0/2830661271 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f473406ce40 0x7f473406f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:36.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.098+0000 7f474a59c700 1 --2- 172.21.15.114:0/2830661271 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f473406ce40 0x7f473406f300 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f474c069d80 tx=0x7f473c00b540 comp rx=0 tx=0).ready entity=mgr.24391 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:36.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.102+0000 7f4733fff700 1 -- 172.21.15.114:0/2830661271 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f4740059ad0 con 0x7f474c0ffab0 2024-02-19T19:02:36.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.349+0000 7f47511a6700 1 -- 172.21.15.114:0/2830661271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f474c19c620 con 0x7f474c0ffab0 2024-02-19T19:02:36.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.349+0000 7f4733fff700 1 -- 172.21.15.114:0/2830661271 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1850 (secure 0 0 0) 0x7f4740016750 con 0x7f474c0ffab0 2024-02-19T19:02:36.369 INFO:teuthology.orchestra.run.smithi114.stdout:e18 2024-02-19T19:02:36.369 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:02:36.369 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:02:36.369 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-02-19T19:02:36.369 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:02:36.369 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:02:36.369 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-02-19T19:02:36.369 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 18 2024-02-19T19:02:36.369 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:02:36.369 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:02:36.370 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-02-19T19:00:21.203922+0000 2024-02-19T19:02:36.370 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-02-19T19:02:36.370 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-02-19T19:02:36.370 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-02-19T19:02:36.370 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-02-19T19:02:36.370 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-02-19T19:02:36.370 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-02-19T19:02:36.370 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-02-19T19:02:36.370 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 47 2024-02-19T19:02:36.370 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:02:36.370 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 2 2024-02-19T19:02:36.370 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1 2024-02-19T19:02:36.370 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=14530,1=24269} 2024-02-19T19:02:36.370 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-02-19T19:02:36.370 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-02-19T19:02:36.371 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-02-19T19:02:36.371 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-02-19T19:02:36.371 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-02-19T19:02:36.371 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data enabled 2024-02-19T19:02:36.371 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-02-19T19:02:36.371 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-02-19T19:02:36.371 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 1 2024-02-19T19:02:36.371 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:02:36.371 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.vpoeop{0:14540} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:02:36.371 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.znhuvd{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:02:36.371 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.stnfnf{1:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:02:36.371 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:02:36.371 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:02:36.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.352+0000 7f47511a6700 1 -- 172.21.15.114:0/2830661271 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f473406ce40 msgr2=0x7f473406f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:36.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.352+0000 7f47511a6700 1 --2- 172.21.15.114:0/2830661271 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f473406ce40 0x7f473406f300 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f474c069d80 tx=0x7f473c00b540 comp rx=0 tx=0).stop 2024-02-19T19:02:36.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.352+0000 7f47511a6700 1 -- 172.21.15.114:0/2830661271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c0ffab0 msgr2=0x7f474c069820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:36.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.352+0000 7f47511a6700 1 --2- 172.21.15.114:0/2830661271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c0ffab0 0x7f474c069820 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f474000d900 tx=0x7f474000dc10 comp rx=0 tx=0).stop 2024-02-19T19:02:36.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.352+0000 7f47511a6700 1 -- 172.21.15.114:0/2830661271 shutdown_connections 2024-02-19T19:02:36.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.352+0000 7f47511a6700 1 --2- 172.21.15.114:0/2830661271 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f473406ce40 0x7f473406f300 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.352+0000 7f47511a6700 1 --2- 172.21.15.114:0/2830661271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c0ffab0 0x7f474c069820 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.352+0000 7f47511a6700 1 --2- 172.21.15.114:0/2830661271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f474c100cb0 0x7f474c19b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.352+0000 7f47511a6700 1 -- 172.21.15.114:0/2830661271 >> 172.21.15.114:0/2830661271 conn(0x7f474c0fb050 msgr2=0x7f474c103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:36.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.352+0000 7f47511a6700 1 -- 172.21.15.114:0/2830661271 shutdown_connections 2024-02-19T19:02:36.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.352+0000 7f47511a6700 1 -- 172.21.15.114:0/2830661271 wait complete. 2024-02-19T19:02:36.373 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 18 2024-02-19T19:02:36.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.461+0000 7fa82e94e700 1 -- 172.21.15.114:0/1211071950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa828101dc0 msgr2=0x7fa828102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:36.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.461+0000 7fa82e94e700 1 --2- 172.21.15.114:0/1211071950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa828101dc0 0x7fa828102220 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa81c009910 tx=0x7fa81c009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:36.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.461+0000 7fa82e94e700 1 -- 172.21.15.114:0/1211071950 shutdown_connections 2024-02-19T19:02:36.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.461+0000 7fa82e94e700 1 --2- 172.21.15.114:0/1211071950 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa828101dc0 0x7fa828102220 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.461+0000 7fa82e94e700 1 --2- 172.21.15.114:0/1211071950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa828100bc0 0x7fa828100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.461+0000 7fa82e94e700 1 -- 172.21.15.114:0/1211071950 >> 172.21.15.114:0/1211071950 conn(0x7fa8280fc140 msgr2=0x7fa8280fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:36.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.461+0000 7fa82e94e700 1 -- 172.21.15.114:0/1211071950 shutdown_connections 2024-02-19T19:02:36.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.461+0000 7fa82e94e700 1 -- 172.21.15.114:0/1211071950 wait complete. 2024-02-19T19:02:36.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.462+0000 7fa82e94e700 1 Processor -- start 2024-02-19T19:02:36.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.462+0000 7fa82e94e700 1 -- start start 2024-02-19T19:02:36.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.462+0000 7fa82e94e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa828100bc0 0x7fa828196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:36.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.462+0000 7fa82e94e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa828101dc0 0x7fa8281968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:36.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.462+0000 7fa82e94e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa828196ed0 con 0x7fa828101dc0 2024-02-19T19:02:36.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.462+0000 7fa82e94e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa828197010 con 0x7fa828100bc0 2024-02-19T19:02:36.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.462+0000 7fa8277fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa828101dc0 0x7fa8281968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:36.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.462+0000 7fa8277fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa828101dc0 0x7fa8281968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50760/0 (socket says 172.21.15.114:50760) 2024-02-19T19:02:36.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.462+0000 7fa8277fe700 1 -- 172.21.15.114:0/955700122 learned_addr learned my addr 172.21.15.114:0/955700122 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:02:36.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.462+0000 7fa827fff700 1 --2- 172.21.15.114:0/955700122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa828100bc0 0x7fa828196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:36.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.463+0000 7fa8277fe700 1 -- 172.21.15.114:0/955700122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa828100bc0 msgr2=0x7fa828196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:36.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.463+0000 7fa8277fe700 1 --2- 172.21.15.114:0/955700122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa828100bc0 0x7fa828196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.463+0000 7fa8277fe700 1 -- 172.21.15.114:0/955700122 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c0095f0 con 0x7fa828101dc0 2024-02-19T19:02:36.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.463+0000 7fa8277fe700 1 --2- 172.21.15.114:0/955700122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa828101dc0 0x7fa8281968b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00bd70 tx=0x7fa81c00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:36.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.463+0000 7fa8257fa700 1 -- 172.21.15.114:0/955700122 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa81c01d070 con 0x7fa828101dc0 2024-02-19T19:02:36.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.463+0000 7fa8257fa700 1 -- 172.21.15.114:0/955700122 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa81c005dc0 con 0x7fa828101dc0 2024-02-19T19:02:36.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.463+0000 7fa82e94e700 1 -- 172.21.15.114:0/955700122 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82819ba60 con 0x7fa828101dc0 2024-02-19T19:02:36.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.463+0000 7fa8257fa700 1 -- 172.21.15.114:0/955700122 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa81c021410 con 0x7fa828101dc0 2024-02-19T19:02:36.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.463+0000 7fa82e94e700 1 -- 172.21.15.114:0/955700122 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82819bf50 con 0x7fa828101dc0 2024-02-19T19:02:36.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.464+0000 7fa82e94e700 1 -- 172.21.15.114:0/955700122 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa828109720 con 0x7fa828101dc0 2024-02-19T19:02:36.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.464+0000 7fa8257fa700 1 -- 172.21.15.114:0/955700122 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) v1 ==== 90933+0+0 (secure 0 0 0) 0x7fa81c00f7d0 con 0x7fa828101dc0 2024-02-19T19:02:36.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.465+0000 7fa8257fa700 1 --2- 172.21.15.114:0/955700122 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7fa81006ce40 0x7fa81006f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:36.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.465+0000 7fa8257fa700 1 -- 172.21.15.114:0/955700122 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fa81c08d9d0 con 0x7fa828101dc0 2024-02-19T19:02:36.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.465+0000 7fa827fff700 1 --2- 172.21.15.114:0/955700122 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7fa81006ce40 0x7fa81006f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:36.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.466+0000 7fa827fff700 1 --2- 172.21.15.114:0/955700122 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7fa81006ce40 0x7fa81006f300 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa828101c20 tx=0x7fa81800b410 comp rx=0 tx=0).ready entity=mgr.24391 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:36.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.469+0000 7fa8257fa700 1 -- 172.21.15.114:0/955700122 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7fa81c05b110 con 0x7fa828101dc0 2024-02-19T19:02:36.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:36 smithi114 ceph-mon[29441]: from='client.14716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:36.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:36 smithi114 ceph-mon[29441]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:36.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:36 smithi114 ceph-mon[29441]: pgmap v11: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:02:36.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:36 smithi114 ceph-mon[29441]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:36.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:36 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1521003230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:36.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:36 smithi174 ceph-mon[41061]: from='client.14716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:36.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:36 smithi174 ceph-mon[41061]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:36.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:36 smithi174 ceph-mon[41061]: pgmap v11: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:02:36.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:36 smithi174 ceph-mon[41061]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:36.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:36 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1521003230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:36.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.690+0000 7fa82e94e700 1 -- 172.21.15.114:0/955700122 --> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa828106710 con 0x7fa81006ce40 2024-02-19T19:02:36.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.692+0000 7fa8257fa700 1 -- 172.21.15.114:0/955700122 <== mgr.24391 v2:172.21.15.174:6828/2185377847 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7fa828106710 con 0x7fa81006ce40 2024-02-19T19:02:36.711 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:02:36.712 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:02:36.712 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-02-19T19:02:36.712 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:02:36.712 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [], 2024-02-19T19:02:36.712 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "1/23 daemons upgraded", 2024-02-19T19:02:36.712 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "", 2024-02-19T19:02:36.712 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-02-19T19:02:36.712 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:02:36.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.694+0000 7fa82e94e700 1 -- 172.21.15.114:0/955700122 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7fa81006ce40 msgr2=0x7fa81006f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:36.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.694+0000 7fa82e94e700 1 --2- 172.21.15.114:0/955700122 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7fa81006ce40 0x7fa81006f300 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa828101c20 tx=0x7fa81800b410 comp rx=0 tx=0).stop 2024-02-19T19:02:36.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.694+0000 7fa82e94e700 1 -- 172.21.15.114:0/955700122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa828101dc0 msgr2=0x7fa8281968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:36.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.694+0000 7fa82e94e700 1 --2- 172.21.15.114:0/955700122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa828101dc0 0x7fa8281968b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00bd70 tx=0x7fa81c00bda0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.694+0000 7fa82e94e700 1 -- 172.21.15.114:0/955700122 shutdown_connections 2024-02-19T19:02:36.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.694+0000 7fa82e94e700 1 --2- 172.21.15.114:0/955700122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa828100bc0 0x7fa828196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.694+0000 7fa82e94e700 1 --2- 172.21.15.114:0/955700122 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7fa81006ce40 0x7fa81006f300 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.694+0000 7fa82e94e700 1 --2- 172.21.15.114:0/955700122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa828101dc0 0x7fa8281968b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.694+0000 7fa82e94e700 1 -- 172.21.15.114:0/955700122 >> 172.21.15.114:0/955700122 conn(0x7fa8280fc140 msgr2=0x7fa828104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:36.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.694+0000 7fa82e94e700 1 -- 172.21.15.114:0/955700122 shutdown_connections 2024-02-19T19:02:36.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.694+0000 7fa82e94e700 1 -- 172.21.15.114:0/955700122 wait complete. 2024-02-19T19:02:36.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.799+0000 7f8de28c5700 1 -- 172.21.15.114:0/2471414808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ddc0ffac0 msgr2=0x7f8ddc0ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:36.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.799+0000 7f8de28c5700 1 --2- 172.21.15.114:0/2471414808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ddc0ffac0 0x7f8ddc0ffee0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc009910 tx=0x7f8dcc009c20 comp rx=0 tx=0).stop 2024-02-19T19:02:36.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.800+0000 7f8de28c5700 1 -- 172.21.15.114:0/2471414808 shutdown_connections 2024-02-19T19:02:36.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.800+0000 7f8de28c5700 1 --2- 172.21.15.114:0/2471414808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ddc100cc0 0x7f8ddc101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.800+0000 7f8de28c5700 1 --2- 172.21.15.114:0/2471414808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ddc0ffac0 0x7f8ddc0ffee0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.800+0000 7f8de28c5700 1 -- 172.21.15.114:0/2471414808 >> 172.21.15.114:0/2471414808 conn(0x7f8ddc0fb060 msgr2=0x7f8ddc0fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:36.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.800+0000 7f8de28c5700 1 -- 172.21.15.114:0/2471414808 shutdown_connections 2024-02-19T19:02:36.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.800+0000 7f8de28c5700 1 -- 172.21.15.114:0/2471414808 wait complete. 2024-02-19T19:02:36.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.800+0000 7f8de28c5700 1 Processor -- start 2024-02-19T19:02:36.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.800+0000 7f8de28c5700 1 -- start start 2024-02-19T19:02:36.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.801+0000 7f8de28c5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ddc0ffac0 0x7f8ddc069930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:36.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.801+0000 7f8de28c5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ddc100cc0 0x7f8ddc19b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:36.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.801+0000 7f8de28c5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc19b6b0 con 0x7f8ddc100cc0 2024-02-19T19:02:36.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.801+0000 7f8de28c5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc19b820 con 0x7f8ddc0ffac0 2024-02-19T19:02:36.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.801+0000 7f8ddb7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ddc100cc0 0x7f8ddc19b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:36.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.801+0000 7f8ddb7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ddc100cc0 0x7f8ddc19b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50778/0 (socket says 172.21.15.114:50778) 2024-02-19T19:02:36.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.801+0000 7f8ddb7fe700 1 -- 172.21.15.114:0/20616315 learned_addr learned my addr 172.21.15.114:0/20616315 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:02:36.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.801+0000 7f8ddbfff700 1 --2- 172.21.15.114:0/20616315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ddc0ffac0 0x7f8ddc069930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:36.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.801+0000 7f8ddb7fe700 1 -- 172.21.15.114:0/20616315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ddc0ffac0 msgr2=0x7f8ddc069930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:36.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.801+0000 7f8ddb7fe700 1 --2- 172.21.15.114:0/20616315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ddc0ffac0 0x7f8ddc069930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:36.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.801+0000 7f8ddb7fe700 1 -- 172.21.15.114:0/20616315 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dcc0095f0 con 0x7f8ddc100cc0 2024-02-19T19:02:36.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.801+0000 7f8ddb7fe700 1 --2- 172.21.15.114:0/20616315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ddc100cc0 0x7f8ddc19b0e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8dd000d8d0 tx=0x7f8dd000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:36.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.801+0000 7f8dd97fa700 1 -- 172.21.15.114:0/20616315 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dd00098a0 con 0x7f8ddc100cc0 2024-02-19T19:02:36.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.801+0000 7f8de28c5700 1 -- 172.21.15.114:0/20616315 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ddc19bb00 con 0x7f8ddc100cc0 2024-02-19T19:02:36.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.801+0000 7f8dd97fa700 1 -- 172.21.15.114:0/20616315 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8dd0010460 con 0x7f8ddc100cc0 2024-02-19T19:02:36.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.801+0000 7f8dd97fa700 1 -- 172.21.15.114:0/20616315 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dd000f5f0 con 0x7f8ddc100cc0 2024-02-19T19:02:36.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.801+0000 7f8de28c5700 1 -- 172.21.15.114:0/20616315 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ddc19c050 con 0x7f8ddc100cc0 2024-02-19T19:02:36.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.802+0000 7f8de28c5700 1 -- 172.21.15.114:0/20616315 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ddc108620 con 0x7f8ddc100cc0 2024-02-19T19:02:36.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.803+0000 7f8dd97fa700 1 -- 172.21.15.114:0/20616315 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) v1 ==== 90933+0+0 (secure 0 0 0) 0x7f8dd0009a00 con 0x7f8ddc100cc0 2024-02-19T19:02:36.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.803+0000 7f8dd97fa700 1 --2- 172.21.15.114:0/20616315 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f8dc406ce40 0x7f8dc406f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:36.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.803+0000 7f8dd97fa700 1 -- 172.21.15.114:0/20616315 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f8dd008c510 con 0x7f8ddc100cc0 2024-02-19T19:02:36.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.804+0000 7f8ddbfff700 1 --2- 172.21.15.114:0/20616315 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f8dc406ce40 0x7f8dc406f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:36.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.804+0000 7f8ddbfff700 1 --2- 172.21.15.114:0/20616315 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f8dc406ce40 0x7f8dc406f300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc006010 tx=0x7f8dcc00b540 comp rx=0 tx=0).ready entity=mgr.24391 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:36.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:36.808+0000 7f8dd97fa700 1 -- 172.21.15.114:0/20616315 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f8dd0059c50 con 0x7f8ddc100cc0 2024-02-19T19:02:37.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:37.089+0000 7f8de28c5700 1 -- 172.21.15.114:0/20616315 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8ddc19c720 con 0x7f8ddc100cc0 2024-02-19T19:02:37.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:37.090+0000 7f8dd97fa700 1 -- 172.21.15.114:0/20616315 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f8dd00597e0 con 0x7f8ddc100cc0 2024-02-19T19:02:37.109 INFO:teuthology.orchestra.run.smithi114.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:02:37.109 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:02:37.109 INFO:teuthology.orchestra.run.smithi114.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:02:37.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:37.092+0000 7f8de28c5700 1 -- 172.21.15.114:0/20616315 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f8dc406ce40 msgr2=0x7f8dc406f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:37.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:37.092+0000 7f8de28c5700 1 --2- 172.21.15.114:0/20616315 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f8dc406ce40 0x7f8dc406f300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc006010 tx=0x7f8dcc00b540 comp rx=0 tx=0).stop 2024-02-19T19:02:37.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:37.092+0000 7f8de28c5700 1 -- 172.21.15.114:0/20616315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ddc100cc0 msgr2=0x7f8ddc19b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:37.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:37.092+0000 7f8de28c5700 1 --2- 172.21.15.114:0/20616315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ddc100cc0 0x7f8ddc19b0e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8dd000d8d0 tx=0x7f8dd000dbe0 comp rx=0 tx=0).stop 2024-02-19T19:02:37.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:37.092+0000 7f8de28c5700 1 -- 172.21.15.114:0/20616315 shutdown_connections 2024-02-19T19:02:37.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:37.092+0000 7f8de28c5700 1 --2- 172.21.15.114:0/20616315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ddc0ffac0 0x7f8ddc069930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:37.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:37.092+0000 7f8de28c5700 1 --2- 172.21.15.114:0/20616315 >> [v2:172.21.15.174:6828/2185377847,v1:172.21.15.174:6829/2185377847] conn(0x7f8dc406ce40 0x7f8dc406f300 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:37.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:37.092+0000 7f8de28c5700 1 --2- 172.21.15.114:0/20616315 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ddc100cc0 0x7f8ddc19b0e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:37.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:37.092+0000 7f8de28c5700 1 -- 172.21.15.114:0/20616315 >> 172.21.15.114:0/20616315 conn(0x7f8ddc0fb060 msgr2=0x7f8ddc103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:37.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:37.092+0000 7f8de28c5700 1 -- 172.21.15.114:0/20616315 shutdown_connections 2024-02-19T19:02:37.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:02:37.092+0000 7f8de28c5700 1 -- 172.21.15.114:0/20616315 wait complete. 2024-02-19T19:02:37.386 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:37 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/2830661271' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:02:37.386 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:37 smithi114 ceph-mon[29441]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:37.386 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:37 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/20616315' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:02:37.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/2830661271' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:02:37.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:37 smithi174 ceph-mon[41061]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:37.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:37 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/20616315' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:02:38.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:38 smithi114 ceph-mon[29441]: pgmap v12: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:02:38.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:38 smithi174 ceph-mon[41061]: pgmap v12: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:02:40.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:40 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:40.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:40 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:40.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:40 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:02:40.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:40 smithi114 ceph-mon[29441]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:02:40.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:40 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:40.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:40 smithi114 ceph-mon[29441]: Upgrade: Need to upgrade myself (mgr.smithi174.lydvnj) 2024-02-19T19:02:40.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:40 smithi114 ceph-mon[29441]: Upgrade: Need to upgrade myself (mgr.smithi174.lydvnj) 2024-02-19T19:02:40.285 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:40 smithi114 ceph-mon[29441]: pgmap v13: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:40.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:40 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:40.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:40 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:40.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:40 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:02:40.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:40 smithi174 ceph-mon[41061]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:02:40.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:40 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:40.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:40 smithi174 ceph-mon[41061]: Upgrade: Need to upgrade myself (mgr.smithi174.lydvnj) 2024-02-19T19:02:40.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:40 smithi174 ceph-mon[41061]: Upgrade: Need to upgrade myself (mgr.smithi174.lydvnj) 2024-02-19T19:02:40.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:40 smithi174 ceph-mon[41061]: pgmap v13: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:42.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:41 smithi114 ceph-mon[29441]: Upgrade: Updating mgr.smithi114.pcvtrk 2024-02-19T19:02:42.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:41 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:42.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:41 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi114.pcvtrk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:02:42.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:41 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi114.pcvtrk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:02:42.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:41 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:02:42.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:41 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:42.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:41 smithi114 ceph-mon[29441]: Deploying daemon mgr.smithi114.pcvtrk on smithi114 2024-02-19T19:02:42.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:41 smithi174 ceph-mon[41061]: Upgrade: Updating mgr.smithi114.pcvtrk 2024-02-19T19:02:42.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:41 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:42.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:41 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi114.pcvtrk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:02:42.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:41 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi114.pcvtrk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:02:42.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:41 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:02:42.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:41 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:42.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:41 smithi174 ceph-mon[41061]: Deploying daemon mgr.smithi114.pcvtrk on smithi114 2024-02-19T19:02:42.914 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:42 smithi114 ceph-mon[29441]: pgmap v14: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:02:43.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:42 smithi174 ceph-mon[41061]: pgmap v14: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:02:44.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:43 smithi114 ceph-mon[29441]: pgmap v15: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:44.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:43 smithi174 ceph-mon[41061]: pgmap v15: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:45.783 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:45 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:45.783 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:45 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:45.783 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:45 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:45.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:45 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:45.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:45 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:45.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:45 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:46.643 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:46 smithi114 ceph-mon[29441]: pgmap v16: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:46.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:46 smithi174 ceph-mon[41061]: pgmap v16: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:48.787 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:48 smithi114 ceph-mon[29441]: pgmap v17: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:02:48.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:48 smithi174 ceph-mon[41061]: pgmap v17: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:02:49.614 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:49 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:49.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:49 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:50.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:50 smithi114 ceph-mon[29441]: pgmap v18: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:50.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:50 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:50.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:50 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:50.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:50 smithi174 ceph-mon[41061]: pgmap v18: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:50.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:50 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:50.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:50 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:52.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:52 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:52.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:52 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:52.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:52 smithi114 ceph-mon[29441]: pgmap v19: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:02:52.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:52 smithi114 ceph-mon[29441]: Standby manager daemon smithi114.pcvtrk restarted 2024-02-19T19:02:52.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:52 smithi114 ceph-mon[29441]: Standby manager daemon smithi114.pcvtrk started 2024-02-19T19:02:52.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:52 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi114.pcvtrk/crt"}]: dispatch 2024-02-19T19:02:52.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:52 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:02:52.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:52 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi114.pcvtrk/key"}]: dispatch 2024-02-19T19:02:52.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:52 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:02:52.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:52 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:52.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:52 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:52.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:52 smithi174 ceph-mon[41061]: pgmap v19: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:02:52.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:52 smithi174 ceph-mon[41061]: Standby manager daemon smithi114.pcvtrk restarted 2024-02-19T19:02:52.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:52 smithi174 ceph-mon[41061]: Standby manager daemon smithi114.pcvtrk started 2024-02-19T19:02:52.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:52 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi114.pcvtrk/crt"}]: dispatch 2024-02-19T19:02:52.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:52 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:02:52.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:52 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi114.pcvtrk/key"}]: dispatch 2024-02-19T19:02:52.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:52 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:02:53.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:53 smithi114 ceph-mon[29441]: mgrmap e25: smithi174.lydvnj(active, since 32s), standbys: smithi114.pcvtrk 2024-02-19T19:02:53.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:53 smithi174 ceph-mon[41061]: mgrmap e25: smithi174.lydvnj(active, since 32s), standbys: smithi114.pcvtrk 2024-02-19T19:02:54.690 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:54 smithi114 ceph-mon[29441]: pgmap v20: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:54.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:54 smithi174 ceph-mon[41061]: pgmap v20: 65 pgs: 65 active+clean; 591 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:02:56.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:56.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:56.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:56.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:56.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:56.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:56.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mgr fail", "who": "smithi174.lydvnj"}]: dispatch 2024-02-19T19:02:56.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mgr fail", "who": "smithi174.lydvnj"}]: dispatch 2024-02-19T19:02:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: osdmap e49: 6 total, 6 up, 6 in 2024-02-19T19:02:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd='[{"prefix": "mgr fail", "who": "smithi174.lydvnj"}]': finished 2024-02-19T19:02:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: mgrmap e26: smithi114.pcvtrk(active, starting, since 0.00616541s) 2024-02-19T19:02:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T19:02:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T19:02:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.tcwpod"}]: dispatch 2024-02-19T19:02:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.vpoeop"}]: dispatch 2024-02-19T19:02:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.znhuvd"}]: dispatch 2024-02-19T19:02:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.stnfnf"}]: dispatch 2024-02-19T19:02:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr metadata", "who": "smithi114.pcvtrk", "id": "smithi114.pcvtrk"}]: dispatch 2024-02-19T19:02:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:02:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:02:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:02:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:02:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:02:56.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:02:56.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T19:02:56.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T19:02:56.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T19:02:56.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: Manager daemon smithi114.pcvtrk is now available 2024-02-19T19:02:56.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:56.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:56.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:56.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:02:56.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:56 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/trash_purge_schedule"}]: dispatch 2024-02-19T19:02:56.456 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:56.456 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:56.457 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:56.457 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:56.457 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' 2024-02-19T19:02:56.457 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:56.457 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.24391 172.21.15.174:0/1505455593' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mgr fail", "who": "smithi174.lydvnj"}]: dispatch 2024-02-19T19:02:56.457 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "mgr fail", "who": "smithi174.lydvnj"}]: dispatch 2024-02-19T19:02:56.457 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: osdmap e49: 6 total, 6 up, 6 in 2024-02-19T19:02:56.457 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.24391 ' entity='mgr.smithi174.lydvnj' cmd='[{"prefix": "mgr fail", "who": "smithi174.lydvnj"}]': finished 2024-02-19T19:02:56.457 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: mgrmap e26: smithi114.pcvtrk(active, starting, since 0.00616541s) 2024-02-19T19:02:56.457 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T19:02:56.457 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T19:02:56.457 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.tcwpod"}]: dispatch 2024-02-19T19:02:56.457 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.vpoeop"}]: dispatch 2024-02-19T19:02:56.457 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.znhuvd"}]: dispatch 2024-02-19T19:02:56.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.stnfnf"}]: dispatch 2024-02-19T19:02:56.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr metadata", "who": "smithi114.pcvtrk", "id": "smithi114.pcvtrk"}]: dispatch 2024-02-19T19:02:56.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:02:56.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:02:56.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:02:56.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:02:56.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:02:56.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:02:56.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T19:02:56.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T19:02:56.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T19:02:56.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: Manager daemon smithi114.pcvtrk is now available 2024-02-19T19:02:56.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:56.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:56.459 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:56.459 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:02:56.459 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:56 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/trash_purge_schedule"}]: dispatch 2024-02-19T19:02:57.343 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:57 smithi174 ceph-mon[41061]: mgrmap e27: smithi114.pcvtrk(active, since 1.00871s) 2024-02-19T19:02:57.344 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:57 smithi174 ceph-mon[41061]: pgmap v3: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:57.537 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:57 smithi114 ceph-mon[29441]: mgrmap e27: smithi114.pcvtrk(active, since 1.00871s) 2024-02-19T19:02:57.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:57 smithi114 ceph-mon[29441]: pgmap v3: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:58.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:58 smithi114 ceph-mon[29441]: [19/Feb/2024:19:02:56] ENGINE Bus STARTING 2024-02-19T19:02:58.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:58 smithi114 ceph-mon[29441]: [19/Feb/2024:19:02:56] ENGINE Serving on http://172.21.15.114:8765 2024-02-19T19:02:58.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:58 smithi114 ceph-mon[29441]: [19/Feb/2024:19:02:56] ENGINE Serving on https://172.21.15.114:7150 2024-02-19T19:02:58.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:58 smithi114 ceph-mon[29441]: [19/Feb/2024:19:02:56] ENGINE Bus STARTED 2024-02-19T19:02:58.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:58 smithi114 ceph-mon[29441]: pgmap v4: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:58.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:58 smithi174 ceph-mon[41061]: [19/Feb/2024:19:02:56] ENGINE Bus STARTING 2024-02-19T19:02:58.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:58 smithi174 ceph-mon[41061]: [19/Feb/2024:19:02:56] ENGINE Serving on http://172.21.15.114:8765 2024-02-19T19:02:58.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:58 smithi174 ceph-mon[41061]: [19/Feb/2024:19:02:56] ENGINE Serving on https://172.21.15.114:7150 2024-02-19T19:02:58.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:58 smithi174 ceph-mon[41061]: [19/Feb/2024:19:02:56] ENGINE Bus STARTED 2024-02-19T19:02:58.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:58 smithi174 ceph-mon[41061]: pgmap v4: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:02:59.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:59 smithi114 ceph-mon[29441]: mgrmap e28: smithi114.pcvtrk(active, since 3s) 2024-02-19T19:02:59.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:59 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:59.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:02:59 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:59.550 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:59 smithi174 ceph-mon[41061]: mgrmap e28: smithi114.pcvtrk(active, since 3s) 2024-02-19T19:02:59.550 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:59 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:02:59.550 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:02:59 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:00.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:00 smithi114 ceph-mon[29441]: pgmap v5: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:03:00.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:00 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:00.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:00 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:00.539 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:00 smithi174 ceph-mon[41061]: pgmap v5: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:03:00.539 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:00 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:00.539 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:00 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:02.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:01 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:02.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:01 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:02.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:01 smithi114 ceph-mon[29441]: pgmap v6: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 205 B/s rd, 1 op/s 2024-02-19T19:03:02.303 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:01 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:02.303 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:01 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:02.304 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:01 smithi174 ceph-mon[41061]: pgmap v6: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 205 B/s rd, 1 op/s 2024-02-19T19:03:03.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:03 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:03.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:03 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:03.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:03 smithi114 ceph-mon[29441]: Standby manager daemon smithi174.lydvnj started 2024-02-19T19:03:03.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:03 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.174:0/1944559334' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/crt"}]: dispatch 2024-02-19T19:03:03.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:03 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.174:0/1944559334' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:03:03.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:03 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.174:0/1944559334' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/key"}]: dispatch 2024-02-19T19:03:03.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:03 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.174:0/1944559334' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:03:03.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:03 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:03.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:03 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:03.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:03 smithi174 ceph-mon[41061]: Standby manager daemon smithi174.lydvnj started 2024-02-19T19:03:03.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:03 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/1944559334' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/crt"}]: dispatch 2024-02-19T19:03:03.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:03 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/1944559334' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:03:03.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:03 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/1944559334' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/key"}]: dispatch 2024-02-19T19:03:03.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:03 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/1944559334' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:03:04.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:04 smithi114 ceph-mon[29441]: mgrmap e29: smithi114.pcvtrk(active, since 7s), standbys: smithi174.lydvnj 2024-02-19T19:03:04.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:04 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr metadata", "who": "smithi174.lydvnj", "id": "smithi174.lydvnj"}]: dispatch 2024-02-19T19:03:04.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:04 smithi114 ceph-mon[29441]: pgmap v7: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-02-19T19:03:04.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:04 smithi174 ceph-mon[41061]: mgrmap e29: smithi114.pcvtrk(active, since 7s), standbys: smithi174.lydvnj 2024-02-19T19:03:04.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:04 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr metadata", "who": "smithi174.lydvnj", "id": "smithi174.lydvnj"}]: dispatch 2024-02-19T19:03:04.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:04 smithi174 ceph-mon[41061]: pgmap v7: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-02-19T19:03:06.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:06 smithi174 ceph-mon[41061]: pgmap v8: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-02-19T19:03:06.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:06 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:06.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:06 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:06.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:06 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:03:06.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:06 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:03:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:06 smithi114 ceph-mon[29441]: pgmap v8: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-02-19T19:03:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:06 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:06 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:06 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:03:06.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:06 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:03:07.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.203+0000 7f5be687d700 1 -- 172.21.15.114:0/2056556913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5be0101670 msgr2=0x7f5be0103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:07.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.203+0000 7f5be687d700 1 --2- 172.21.15.114:0/2056556913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5be0101670 0x7f5be0103a60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009910 tx=0x7f5bd0009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:07.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.203+0000 7f5be687d700 1 -- 172.21.15.114:0/2056556913 shutdown_connections 2024-02-19T19:03:07.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.203+0000 7f5be687d700 1 --2- 172.21.15.114:0/2056556913 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5be0103fa0 0x7f5be0106390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:07.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.203+0000 7f5be687d700 1 --2- 172.21.15.114:0/2056556913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5be0101670 0x7f5be0103a60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:07.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.203+0000 7f5be687d700 1 -- 172.21.15.114:0/2056556913 >> 172.21.15.114:0/2056556913 conn(0x7f5be00fb080 msgr2=0x7f5be00fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:07.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.203+0000 7f5be687d700 1 -- 172.21.15.114:0/2056556913 shutdown_connections 2024-02-19T19:03:07.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.203+0000 7f5be687d700 1 -- 172.21.15.114:0/2056556913 wait complete. 2024-02-19T19:03:07.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.204+0000 7f5be687d700 1 Processor -- start 2024-02-19T19:03:07.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.204+0000 7f5be687d700 1 -- start start 2024-02-19T19:03:07.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.204+0000 7f5be687d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5be0101670 0x7f5be0196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:07.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.204+0000 7f5be687d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5be0103fa0 0x7f5be0196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:07.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.204+0000 7f5be687d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be0196e70 con 0x7f5be0103fa0 2024-02-19T19:03:07.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.204+0000 7f5be687d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be0196fb0 con 0x7f5be0101670 2024-02-19T19:03:07.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.205+0000 7f5bdf7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5be0103fa0 0x7f5be0196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:07.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.205+0000 7f5bdffff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5be0101670 0x7f5be0196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:07.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.205+0000 7f5bdffff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5be0101670 0x7f5be0196310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.114:47992/0 (socket says 172.21.15.114:47992) 2024-02-19T19:03:07.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.205+0000 7f5bdf7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5be0103fa0 0x7f5be0196850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52580/0 (socket says 172.21.15.114:52580) 2024-02-19T19:03:07.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.205+0000 7f5bdffff700 1 -- 172.21.15.114:0/3324582521 learned_addr learned my addr 172.21.15.114:0/3324582521 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:03:07.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.205+0000 7f5bdffff700 1 -- 172.21.15.114:0/3324582521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5be0103fa0 msgr2=0x7f5be0196850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:07.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.205+0000 7f5bdffff700 1 --2- 172.21.15.114:0/3324582521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5be0103fa0 0x7f5be0196850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:07.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.205+0000 7f5bdffff700 1 -- 172.21.15.114:0/3324582521 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd00095f0 con 0x7f5be0101670 2024-02-19T19:03:07.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.205+0000 7f5bdf7fe700 1 --2- 172.21.15.114:0/3324582521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5be0103fa0 0x7f5be0196850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:03:07.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.205+0000 7f5bdffff700 1 --2- 172.21.15.114:0/3324582521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5be0101670 0x7f5be0196310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0000c00 tx=0x7f5bd000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:07.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.205+0000 7f5bdd7fa700 1 -- 172.21.15.114:0/3324582521 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bd001d070 con 0x7f5be0101670 2024-02-19T19:03:07.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.205+0000 7f5be687d700 1 -- 172.21.15.114:0/3324582521 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be019ba00 con 0x7f5be0101670 2024-02-19T19:03:07.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.205+0000 7f5bdd7fa700 1 -- 172.21.15.114:0/3324582521 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5bd0005dc0 con 0x7f5be0101670 2024-02-19T19:03:07.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.205+0000 7f5bdd7fa700 1 -- 172.21.15.114:0/3324582521 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5bd00226a0 con 0x7f5be0101670 2024-02-19T19:03:07.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.205+0000 7f5be687d700 1 -- 172.21.15.114:0/3324582521 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be019bef0 con 0x7f5be0101670 2024-02-19T19:03:07.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.206+0000 7f5be687d700 1 -- 172.21.15.114:0/3324582521 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be0190550 con 0x7f5be0101670 2024-02-19T19:03:07.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.207+0000 7f5bdd7fa700 1 -- 172.21.15.114:0/3324582521 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 29) v1 ==== 91450+0+0 (secure 0 0 0) 0x7f5bd00038f0 con 0x7f5be0101670 2024-02-19T19:03:07.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.207+0000 7f5bdd7fa700 1 --2- 172.21.15.114:0/3324582521 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f5bc806d660 0x7f5bc806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:07.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.207+0000 7f5bdd7fa700 1 -- 172.21.15.114:0/3324582521 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f5bd008e740 con 0x7f5be0101670 2024-02-19T19:03:07.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.207+0000 7f5bdf7fe700 1 --2- 172.21.15.114:0/3324582521 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f5bc806d660 0x7f5bc806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:07.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.208+0000 7f5bdf7fe700 1 --2- 172.21.15.114:0/3324582521 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f5bc806d660 0x7f5bc806fb20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4011490 tx=0x7f5bd4009260 comp rx=0 tx=0).ready entity=mgr.14744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:07.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.212+0000 7f5bdd7fa700 1 -- 172.21.15.114:0/3324582521 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f5bd005bd30 con 0x7f5be0101670 2024-02-19T19:03:07.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.446+0000 7f5be687d700 1 -- 172.21.15.114:0/3324582521 --> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5be0109820 con 0x7f5bc806d660 2024-02-19T19:03:07.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.447+0000 7f5bdd7fa700 1 -- 172.21.15.114:0/3324582521 <== mgr.14744 v2:172.21.15.114:6800/1408766633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f5be0109820 con 0x7f5bc806d660 2024-02-19T19:03:07.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.449+0000 7f5be687d700 1 -- 172.21.15.114:0/3324582521 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f5bc806d660 msgr2=0x7f5bc806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:07.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.449+0000 7f5be687d700 1 --2- 172.21.15.114:0/3324582521 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f5bc806d660 0x7f5bc806fb20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4011490 tx=0x7f5bd4009260 comp rx=0 tx=0).stop 2024-02-19T19:03:07.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.449+0000 7f5be687d700 1 -- 172.21.15.114:0/3324582521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5be0101670 msgr2=0x7f5be0196310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:07.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.449+0000 7f5be687d700 1 --2- 172.21.15.114:0/3324582521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5be0101670 0x7f5be0196310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0000c00 tx=0x7f5bd000f690 comp rx=0 tx=0).stop 2024-02-19T19:03:07.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.449+0000 7f5be687d700 1 -- 172.21.15.114:0/3324582521 shutdown_connections 2024-02-19T19:03:07.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.449+0000 7f5be687d700 1 --2- 172.21.15.114:0/3324582521 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f5bc806d660 0x7f5bc806fb20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:07.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.449+0000 7f5be687d700 1 --2- 172.21.15.114:0/3324582521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5be0101670 0x7f5be0196310 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:07.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.449+0000 7f5be687d700 1 --2- 172.21.15.114:0/3324582521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5be0103fa0 0x7f5be0196850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:07.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.449+0000 7f5be687d700 1 -- 172.21.15.114:0/3324582521 >> 172.21.15.114:0/3324582521 conn(0x7f5be00fb080 msgr2=0x7f5be00fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:07.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.449+0000 7f5be687d700 1 -- 172.21.15.114:0/3324582521 shutdown_connections 2024-02-19T19:03:07.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.449+0000 7f5be687d700 1 -- 172.21.15.114:0/3324582521 wait complete. 2024-02-19T19:03:07.479 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-02-19T19:03:07.479 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:07 smithi114 ceph-mon[29441]: Updating smithi114:/etc/ceph/ceph.conf 2024-02-19T19:03:07.479 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:07 smithi114 ceph-mon[29441]: Updating smithi174:/etc/ceph/ceph.conf 2024-02-19T19:03:07.479 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:07 smithi114 ceph-mon[29441]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:03:07.479 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:07 smithi114 ceph-mon[29441]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:03:07.479 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:07 smithi114 ceph-mon[29441]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:03:07.480 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:07 smithi114 ceph-mon[29441]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:03:07.480 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:07 smithi114 ceph-mon[29441]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:03:07.480 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:07 smithi114 ceph-mon[29441]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:03:07.480 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:07 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:07.480 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:07 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:07.480 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:07 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:07.480 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:07 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:07.480 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:07 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:07.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.555+0000 7ffaf44ac700 1 -- 172.21.15.114:0/906330846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaec100cb0 msgr2=0x7ffaec101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:07.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.555+0000 7ffaf44ac700 1 --2- 172.21.15.114:0/906330846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaec100cb0 0x7ffaec101110 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009910 tx=0x7ffae8009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:07.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.555+0000 7ffaf44ac700 1 -- 172.21.15.114:0/906330846 shutdown_connections 2024-02-19T19:03:07.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.555+0000 7ffaf44ac700 1 --2- 172.21.15.114:0/906330846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaec100cb0 0x7ffaec101110 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:07.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.555+0000 7ffaf44ac700 1 --2- 172.21.15.114:0/906330846 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaec0ffab0 0x7ffaec0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:07.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.555+0000 7ffaf44ac700 1 -- 172.21.15.114:0/906330846 >> 172.21.15.114:0/906330846 conn(0x7ffaec0fb030 msgr2=0x7ffaec0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:07.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.555+0000 7ffaf44ac700 1 -- 172.21.15.114:0/906330846 shutdown_connections 2024-02-19T19:03:07.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.555+0000 7ffaf44ac700 1 -- 172.21.15.114:0/906330846 wait complete. 2024-02-19T19:03:07.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.556+0000 7ffaf44ac700 1 Processor -- start 2024-02-19T19:03:07.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.556+0000 7ffaf44ac700 1 -- start start 2024-02-19T19:03:07.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.556+0000 7ffaf44ac700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaec0ffab0 0x7ffaec103a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:07.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.556+0000 7ffaf44ac700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaec100cb0 0x7ffaec102090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:07.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.556+0000 7ffaf44ac700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec1025d0 con 0x7ffaec0ffab0 2024-02-19T19:03:07.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.556+0000 7ffaf44ac700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec102710 con 0x7ffaec100cb0 2024-02-19T19:03:07.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.556+0000 7ffaf2248700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaec0ffab0 0x7ffaec103a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:07.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.556+0000 7ffaf2248700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaec0ffab0 0x7ffaec103a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52596/0 (socket says 172.21.15.114:52596) 2024-02-19T19:03:07.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.556+0000 7ffaf2248700 1 -- 172.21.15.114:0/3591208309 learned_addr learned my addr 172.21.15.114:0/3591208309 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:03:07.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.556+0000 7ffaf2248700 1 -- 172.21.15.114:0/3591208309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaec100cb0 msgr2=0x7ffaec102090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:07.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.556+0000 7ffaf1a47700 1 --2- 172.21.15.114:0/3591208309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaec100cb0 0x7ffaec102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:07.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.556+0000 7ffaf2248700 1 --2- 172.21.15.114:0/3591208309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaec100cb0 0x7ffaec102090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.556+0000 7ffaf2248700 1 -- 172.21.15.114:0/3591208309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae80095f0 con 0x7ffaec0ffab0 2024-02-19T19:03:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.556+0000 7ffaf1a47700 1 --2- 172.21.15.114:0/3591208309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaec100cb0 0x7ffaec102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:03:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.556+0000 7ffaf2248700 1 --2- 172.21.15.114:0/3591208309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaec0ffab0 0x7ffaec103a40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ffadc00e8f0 tx=0x7ffadc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.557+0000 7ffae37fe700 1 -- 172.21.15.114:0/3591208309 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffadc009790 con 0x7ffaec0ffab0 2024-02-19T19:03:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.557+0000 7ffaf44ac700 1 -- 172.21.15.114:0/3591208309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaec1029f0 con 0x7ffaec0ffab0 2024-02-19T19:03:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.557+0000 7ffae37fe700 1 -- 172.21.15.114:0/3591208309 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ffadc0044d0 con 0x7ffaec0ffab0 2024-02-19T19:03:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.557+0000 7ffae37fe700 1 -- 172.21.15.114:0/3591208309 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffadc003da0 con 0x7ffaec0ffab0 2024-02-19T19:03:07.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.557+0000 7ffaf44ac700 1 -- 172.21.15.114:0/3591208309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaec19e3a0 con 0x7ffaec0ffab0 2024-02-19T19:03:07.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.558+0000 7ffaf44ac700 1 -- 172.21.15.114:0/3591208309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaec108610 con 0x7ffaec0ffab0 2024-02-19T19:03:07.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.558+0000 7ffae37fe700 1 -- 172.21.15.114:0/3591208309 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 29) v1 ==== 91450+0+0 (secure 0 0 0) 0x7ffadc00cba0 con 0x7ffaec0ffab0 2024-02-19T19:03:07.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.558+0000 7ffae37fe700 1 --2- 172.21.15.114:0/3591208309 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7ffad806d660 0x7ffad806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:07.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.559+0000 7ffae37fe700 1 -- 172.21.15.114:0/3591208309 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7ffadc014070 con 0x7ffaec0ffab0 2024-02-19T19:03:07.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.559+0000 7ffaf1a47700 1 --2- 172.21.15.114:0/3591208309 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7ffad806d660 0x7ffad806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:07.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.559+0000 7ffaf1a47700 1 --2- 172.21.15.114:0/3591208309 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7ffad806d660 0x7ffad806fb20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffae8000c00 tx=0x7ffae8009f70 comp rx=0 tx=0).ready entity=mgr.14744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:07.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.563+0000 7ffae37fe700 1 -- 172.21.15.114:0/3591208309 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7ffadc05ac60 con 0x7ffaec0ffab0 2024-02-19T19:03:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.784+0000 7ffaf44ac700 1 -- 172.21.15.114:0/3591208309 --> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffaec068b20 con 0x7ffad806d660 2024-02-19T19:03:07.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.785+0000 7ffae37fe700 1 -- 172.21.15.114:0/3591208309 <== mgr.14744 v2:172.21.15.114:6800/1408766633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7ffaec068b20 con 0x7ffad806d660 2024-02-19T19:03:07.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.787+0000 7ffaf44ac700 1 -- 172.21.15.114:0/3591208309 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7ffad806d660 msgr2=0x7ffad806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:07.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.787+0000 7ffaf44ac700 1 --2- 172.21.15.114:0/3591208309 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7ffad806d660 0x7ffad806fb20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffae8000c00 tx=0x7ffae8009f70 comp rx=0 tx=0).stop 2024-02-19T19:03:07.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.787+0000 7ffaf44ac700 1 -- 172.21.15.114:0/3591208309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaec0ffab0 msgr2=0x7ffaec103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:07.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.787+0000 7ffaf44ac700 1 --2- 172.21.15.114:0/3591208309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaec0ffab0 0x7ffaec103a40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ffadc00e8f0 tx=0x7ffadc00ecb0 comp rx=0 tx=0).stop 2024-02-19T19:03:07.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.787+0000 7ffaf44ac700 1 -- 172.21.15.114:0/3591208309 shutdown_connections 2024-02-19T19:03:07.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.787+0000 7ffaf44ac700 1 --2- 172.21.15.114:0/3591208309 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7ffad806d660 0x7ffad806fb20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:07.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.787+0000 7ffaf44ac700 1 --2- 172.21.15.114:0/3591208309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffaec0ffab0 0x7ffaec103a40 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:07.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.787+0000 7ffaf44ac700 1 --2- 172.21.15.114:0/3591208309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaec100cb0 0x7ffaec102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:07.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.787+0000 7ffaf44ac700 1 -- 172.21.15.114:0/3591208309 >> 172.21.15.114:0/3591208309 conn(0x7ffaec0fb030 msgr2=0x7ffaec068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:07.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.787+0000 7ffaf44ac700 1 -- 172.21.15.114:0/3591208309 shutdown_connections 2024-02-19T19:03:07.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.787+0000 7ffaf44ac700 1 -- 172.21.15.114:0/3591208309 wait complete. 2024-02-19T19:03:07.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:07 smithi174 ceph-mon[41061]: Updating smithi114:/etc/ceph/ceph.conf 2024-02-19T19:03:07.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:07 smithi174 ceph-mon[41061]: Updating smithi174:/etc/ceph/ceph.conf 2024-02-19T19:03:07.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:07 smithi174 ceph-mon[41061]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:03:07.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:07 smithi174 ceph-mon[41061]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:03:07.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:07 smithi174 ceph-mon[41061]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:03:07.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:07 smithi174 ceph-mon[41061]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:03:07.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:07 smithi174 ceph-mon[41061]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:03:07.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:07 smithi174 ceph-mon[41061]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:03:07.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:07 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:07.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:07 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:07.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:07 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:07.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:07 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:07.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:07 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:07.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.892+0000 7fa0c7d46700 1 -- 172.21.15.114:0/475030256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0c0100bb0 msgr2=0x7fa0c0100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:07.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.892+0000 7fa0c7d46700 1 --2- 172.21.15.114:0/475030256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0c0100bb0 0x7fa0c0100fd0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009910 tx=0x7fa0bc009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:07.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.892+0000 7fa0c7d46700 1 -- 172.21.15.114:0/475030256 shutdown_connections 2024-02-19T19:03:07.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.892+0000 7fa0c7d46700 1 --2- 172.21.15.114:0/475030256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0c0101db0 0x7fa0c0102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:07.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.892+0000 7fa0c7d46700 1 --2- 172.21.15.114:0/475030256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0c0100bb0 0x7fa0c0100fd0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:07.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.892+0000 7fa0c7d46700 1 -- 172.21.15.114:0/475030256 >> 172.21.15.114:0/475030256 conn(0x7fa0c00fc110 msgr2=0x7fa0c00fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:07.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.892+0000 7fa0c7d46700 1 -- 172.21.15.114:0/475030256 shutdown_connections 2024-02-19T19:03:07.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.892+0000 7fa0c7d46700 1 -- 172.21.15.114:0/475030256 wait complete. 2024-02-19T19:03:07.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.892+0000 7fa0c7d46700 1 Processor -- start 2024-02-19T19:03:07.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.892+0000 7fa0c7d46700 1 -- start start 2024-02-19T19:03:07.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.893+0000 7fa0c7d46700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0c0100bb0 0x7fa0c0196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:07.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.893+0000 7fa0c7d46700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0c0101db0 0x7fa0c01968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:07.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.893+0000 7fa0c7d46700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c0196ef0 con 0x7fa0c0100bb0 2024-02-19T19:03:07.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.893+0000 7fa0c7d46700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c0197030 con 0x7fa0c0101db0 2024-02-19T19:03:07.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.893+0000 7fa0c5ae2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0c0100bb0 0x7fa0c0196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:07.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.893+0000 7fa0c5ae2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0c0100bb0 0x7fa0c0196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52622/0 (socket says 172.21.15.114:52622) 2024-02-19T19:03:07.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.893+0000 7fa0c5ae2700 1 -- 172.21.15.114:0/2418192163 learned_addr learned my addr 172.21.15.114:0/2418192163 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:03:07.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.893+0000 7fa0c52e1700 1 --2- 172.21.15.114:0/2418192163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0c0101db0 0x7fa0c01968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:07.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.893+0000 7fa0c5ae2700 1 -- 172.21.15.114:0/2418192163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0c0101db0 msgr2=0x7fa0c01968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:07.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.893+0000 7fa0c5ae2700 1 --2- 172.21.15.114:0/2418192163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0c0101db0 0x7fa0c01968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:07.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.893+0000 7fa0c5ae2700 1 -- 172.21.15.114:0/2418192163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0bc0095f0 con 0x7fa0c0100bb0 2024-02-19T19:03:07.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.893+0000 7fa0c5ae2700 1 --2- 172.21.15.114:0/2418192163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0c0100bb0 0x7fa0c0196390 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00bda0 tx=0x7fa0bc00bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:07.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.893+0000 7fa0b6ffd700 1 -- 172.21.15.114:0/2418192163 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0bc01d070 con 0x7fa0c0100bb0 2024-02-19T19:03:07.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.893+0000 7fa0b6ffd700 1 -- 172.21.15.114:0/2418192163 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa0bc00f460 con 0x7fa0c0100bb0 2024-02-19T19:03:07.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.893+0000 7fa0c7d46700 1 -- 172.21.15.114:0/2418192163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c019ba80 con 0x7fa0c0100bb0 2024-02-19T19:03:07.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.893+0000 7fa0b6ffd700 1 -- 172.21.15.114:0/2418192163 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0bc021620 con 0x7fa0c0100bb0 2024-02-19T19:03:07.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.893+0000 7fa0c7d46700 1 -- 172.21.15.114:0/2418192163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c019bf70 con 0x7fa0c0100bb0 2024-02-19T19:03:07.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.894+0000 7fa0c7d46700 1 -- 172.21.15.114:0/2418192163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c0109710 con 0x7fa0c0100bb0 2024-02-19T19:03:07.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.895+0000 7fa0b6ffd700 1 -- 172.21.15.114:0/2418192163 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 29) v1 ==== 91450+0+0 (secure 0 0 0) 0x7fa0bc00f5d0 con 0x7fa0c0100bb0 2024-02-19T19:03:07.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.895+0000 7fa0b6ffd700 1 --2- 172.21.15.114:0/2418192163 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7fa0ac06d660 0x7fa0ac06fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:07.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.895+0000 7fa0b6ffd700 1 -- 172.21.15.114:0/2418192163 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fa0bc08e460 con 0x7fa0c0100bb0 2024-02-19T19:03:07.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.895+0000 7fa0c52e1700 1 --2- 172.21.15.114:0/2418192163 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7fa0ac06d660 0x7fa0ac06fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:07.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.896+0000 7fa0c52e1700 1 --2- 172.21.15.114:0/2418192163 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7fa0ac06d660 0x7fa0ac06fb20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0007c60 tx=0x7fa0b00073d0 comp rx=0 tx=0).ready entity=mgr.14744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:07.919 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:07.900+0000 7fa0b6ffd700 1 -- 172.21.15.114:0/2418192163 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7fa0bc05ba50 con 0x7fa0c0100bb0 2024-02-19T19:03:08.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.116+0000 7fa0c7d46700 1 -- 172.21.15.114:0/2418192163 --> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0c0106700 con 0x7fa0ac06d660 2024-02-19T19:03:08.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.127+0000 7fa0b6ffd700 1 -- 172.21.15.114:0/2418192163 <== mgr.14744 v2:172.21.15.114:6800/1408766633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fa0c0106700 con 0x7fa0ac06d660 2024-02-19T19:03:08.146 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:03:08.146 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (7m) 7s ago 9m 26.4M - 0.25.0 c8568f914cd2 30444b676ca4 2024-02-19T19:03:08.146 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (10m) 7s ago 10m 7792k - 18.2.1 94eadd990e0a 4918cd87e312 2024-02-19T19:03:08.146 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi174 smithi174 running (4m) 9s ago 7m 7692k - 18.2.1 94eadd990e0a 516c34431b93 2024-02-19T19:03:08.146 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (10m) 7s ago 10m 6660k - 18.2.1 94eadd990e0a ecfcd98907ba 2024-02-19T19:03:08.146 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi174 smithi174 running (7m) 9s ago 7m 6660k - 18.2.1 94eadd990e0a de88985f5a07 2024-02-19T19:03:08.146 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (6m) 7s ago 8m 85.5M - 9.4.7 954c08fa6188 478bd533e827 2024-02-19T19:03:08.146 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tcwpod smithi114 running (2m) 7s ago 2m 26.3M - 18.2.1 94eadd990e0a 420175777cf4 2024-02-19T19:03:08.147 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.vpoeop smithi114 running (3m) 7s ago 3m 19.7M - 18.2.1 94eadd990e0a f81a79e53ead 2024-02-19T19:03:08.147 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.stnfnf smithi174 running (2m) 9s ago 2m 21.5M - 18.2.1 94eadd990e0a 3df35954a492 2024-02-19T19:03:08.147 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.znhuvd smithi174 running (3m) 9s ago 3m 21.2M - 18.2.1 94eadd990e0a 2d2267494ae1 2024-02-19T19:03:08.147 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.pcvtrk smithi114 *:8443,9283,8765 running (24s) 7s ago 11m 470M - 19.0.0-1450-gf78a58c0 f081f847070a 05b5c06122b1 2024-02-19T19:03:08.147 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi174.lydvnj smithi174 *:8443,9283,8765 running (56s) 9s ago 7m 71.2M - 19.0.0-1450-gf78a58c0 f081f847070a 3c0ee4f00329 2024-02-19T19:03:08.147 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (11m) 7s ago 11m 50.8M 2048M 18.2.1 94eadd990e0a 41468f811fc0 2024-02-19T19:03:08.147 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi174 smithi174 running (7m) 9s ago 7m 40.2M 2048M 18.2.1 94eadd990e0a 70dd7bf69fd9 2024-02-19T19:03:08.147 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (10m) 7s ago 10m 14.4M - 1.5.0 0da6a335fe13 b6b258074cce 2024-02-19T19:03:08.147 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi174 smithi174 *:9100 running (7m) 9s ago 7m 15.1M - 1.5.0 0da6a335fe13 d0b4fdf67b22 2024-02-19T19:03:08.147 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (6m) 7s ago 6m 48.4M 1216M 18.2.1 94eadd990e0a 990afc982b6c 2024-02-19T19:03:08.147 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (6m) 7s ago 6m 48.6M 1216M 18.2.1 94eadd990e0a 932e3ad37db5 2024-02-19T19:03:08.147 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (5m) 7s ago 5m 50.5M 1216M 18.2.1 94eadd990e0a 12953988af86 2024-02-19T19:03:08.147 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi174 running (5m) 9s ago 5m 47.6M 2240M 18.2.1 94eadd990e0a 979f45cabbce 2024-02-19T19:03:08.147 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi174 running (4m) 9s ago 4m 49.3M 2240M 18.2.1 94eadd990e0a 8b8c3745b8c5 2024-02-19T19:03:08.148 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi174 running (4m) 9s ago 4m 51.2M 2240M 18.2.1 94eadd990e0a 5771a995ed75 2024-02-19T19:03:08.148 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (29s) 7s ago 8m 42.1M - 2.43.0 a07b618ecd1d 14a72b4c5e6e 2024-02-19T19:03:08.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.128+0000 7fa0c7d46700 1 -- 172.21.15.114:0/2418192163 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7fa0ac06d660 msgr2=0x7fa0ac06fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:08.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.128+0000 7fa0c7d46700 1 --2- 172.21.15.114:0/2418192163 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7fa0ac06d660 0x7fa0ac06fb20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0007c60 tx=0x7fa0b00073d0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.128+0000 7fa0c7d46700 1 -- 172.21.15.114:0/2418192163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0c0100bb0 msgr2=0x7fa0c0196390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:08.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.128+0000 7fa0c7d46700 1 --2- 172.21.15.114:0/2418192163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0c0100bb0 0x7fa0c0196390 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00bda0 tx=0x7fa0bc00bdd0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.129+0000 7fa0c7d46700 1 -- 172.21.15.114:0/2418192163 shutdown_connections 2024-02-19T19:03:08.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.129+0000 7fa0c7d46700 1 --2- 172.21.15.114:0/2418192163 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7fa0ac06d660 0x7fa0ac06fb20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.129+0000 7fa0c7d46700 1 --2- 172.21.15.114:0/2418192163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0c0100bb0 0x7fa0c0196390 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.129+0000 7fa0c7d46700 1 --2- 172.21.15.114:0/2418192163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0c0101db0 0x7fa0c01968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.129+0000 7fa0c7d46700 1 -- 172.21.15.114:0/2418192163 >> 172.21.15.114:0/2418192163 conn(0x7fa0c00fc110 msgr2=0x7fa0c0104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:08.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.129+0000 7fa0c7d46700 1 -- 172.21.15.114:0/2418192163 shutdown_connections 2024-02-19T19:03:08.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.129+0000 7fa0c7d46700 1 -- 172.21.15.114:0/2418192163 wait complete. 2024-02-19T19:03:08.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.232+0000 7fab10b24700 1 -- 172.21.15.114:0/2282380019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab0c1016a0 msgr2=0x7fab0c103a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:08.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.232+0000 7fab10b24700 1 --2- 172.21.15.114:0/2282380019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab0c1016a0 0x7fab0c103a90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fab0400b0a0 tx=0x7fab0400b3b0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.232+0000 7fab10b24700 1 -- 172.21.15.114:0/2282380019 shutdown_connections 2024-02-19T19:03:08.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.232+0000 7fab10b24700 1 --2- 172.21.15.114:0/2282380019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab0c103fd0 0x7fab0c1063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.232+0000 7fab10b24700 1 --2- 172.21.15.114:0/2282380019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab0c1016a0 0x7fab0c103a90 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.232+0000 7fab10b24700 1 -- 172.21.15.114:0/2282380019 >> 172.21.15.114:0/2282380019 conn(0x7fab0c0fb030 msgr2=0x7fab0c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:08.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.232+0000 7fab10b24700 1 -- 172.21.15.114:0/2282380019 shutdown_connections 2024-02-19T19:03:08.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.232+0000 7fab10b24700 1 -- 172.21.15.114:0/2282380019 wait complete. 2024-02-19T19:03:08.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.233+0000 7fab10b24700 1 Processor -- start 2024-02-19T19:03:08.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.233+0000 7fab10b24700 1 -- start start 2024-02-19T19:03:08.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.233+0000 7fab10b24700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab0c1016a0 0x7fab0c1963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:08.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.233+0000 7fab10b24700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab0c103fd0 0x7fab0c196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:08.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.233+0000 7fab10b24700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c196f40 con 0x7fab0c1016a0 2024-02-19T19:03:08.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.233+0000 7fab10b24700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c197080 con 0x7fab0c103fd0 2024-02-19T19:03:08.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.233+0000 7fab0a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab0c1016a0 0x7fab0c1963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:08.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.233+0000 7fab0a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab0c1016a0 0x7fab0c1963e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52638/0 (socket says 172.21.15.114:52638) 2024-02-19T19:03:08.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.233+0000 7fab0a59c700 1 -- 172.21.15.114:0/631191033 learned_addr learned my addr 172.21.15.114:0/631191033 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:03:08.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.234+0000 7fab0a59c700 1 -- 172.21.15.114:0/631191033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab0c103fd0 msgr2=0x7fab0c196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:08.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.234+0000 7fab0a59c700 1 --2- 172.21.15.114:0/631191033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab0c103fd0 0x7fab0c196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.234+0000 7fab0a59c700 1 -- 172.21.15.114:0/631191033 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab04009d00 con 0x7fab0c1016a0 2024-02-19T19:03:08.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.234+0000 7fab0a59c700 1 --2- 172.21.15.114:0/631191033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab0c1016a0 0x7fab0c1963e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fab040077b0 tx=0x7fab04003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:08.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.234+0000 7faafb7fe700 1 -- 172.21.15.114:0/631191033 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab0400e040 con 0x7fab0c1016a0 2024-02-19T19:03:08.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.234+0000 7fab10b24700 1 -- 172.21.15.114:0/631191033 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab0c19bad0 con 0x7fab0c1016a0 2024-02-19T19:03:08.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.234+0000 7fab10b24700 1 -- 172.21.15.114:0/631191033 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab0c19c040 con 0x7fab0c1016a0 2024-02-19T19:03:08.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.234+0000 7faafb7fe700 1 -- 172.21.15.114:0/631191033 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fab0400ce60 con 0x7fab0c1016a0 2024-02-19T19:03:08.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.234+0000 7faafb7fe700 1 -- 172.21.15.114:0/631191033 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab0401daf0 con 0x7fab0c1016a0 2024-02-19T19:03:08.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.235+0000 7fab10b24700 1 -- 172.21.15.114:0/631191033 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaec005320 con 0x7fab0c1016a0 2024-02-19T19:03:08.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.236+0000 7faafb7fe700 1 -- 172.21.15.114:0/631191033 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 29) v1 ==== 91450+0+0 (secure 0 0 0) 0x7fab0401dd30 con 0x7fab0c1016a0 2024-02-19T19:03:08.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.236+0000 7faafb7fe700 1 --2- 172.21.15.114:0/631191033 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7faaf406d660 0x7faaf406fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:08.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.236+0000 7faafb7fe700 1 -- 172.21.15.114:0/631191033 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fab0408ebf0 con 0x7fab0c1016a0 2024-02-19T19:03:08.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.236+0000 7fab09d9b700 1 --2- 172.21.15.114:0/631191033 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7faaf406d660 0x7faaf406fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:08.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.237+0000 7fab09d9b700 1 --2- 172.21.15.114:0/631191033 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7faaf406d660 0x7faaf406fb20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fab000097b0 tx=0x7fab00006d20 comp rx=0 tx=0).ready entity=mgr.14744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:08.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.241+0000 7faafb7fe700 1 -- 172.21.15.114:0/631191033 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7fab0405c260 con 0x7fab0c1016a0 2024-02-19T19:03:08.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.533+0000 7fab10b24700 1 -- 172.21.15.114:0/631191033 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7faaec005cc0 con 0x7fab0c1016a0 2024-02-19T19:03:08.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.533+0000 7faafb7fe700 1 -- 172.21.15.114:0/631191033 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7fab0405bdf0 con 0x7fab0c1016a0 2024-02-19T19:03:08.553 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:03:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-02-19T19:03:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-02-19T19:03:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:03:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-02-19T19:03:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:03:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:03:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-02-19T19:03:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-02-19T19:03:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:03:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-02-19T19:03:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:03:08.554 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:03:08.554 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-02-19T19:03:08.554 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-02-19T19:03:08.554 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:03:08.554 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-02-19T19:03:08.554 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:03:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.535+0000 7fab10b24700 1 -- 172.21.15.114:0/631191033 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7faaf406d660 msgr2=0x7faaf406fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.535+0000 7fab10b24700 1 --2- 172.21.15.114:0/631191033 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7faaf406d660 0x7faaf406fb20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fab000097b0 tx=0x7fab00006d20 comp rx=0 tx=0).stop 2024-02-19T19:03:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.535+0000 7fab10b24700 1 -- 172.21.15.114:0/631191033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab0c1016a0 msgr2=0x7fab0c1963e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.535+0000 7fab10b24700 1 --2- 172.21.15.114:0/631191033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab0c1016a0 0x7fab0c1963e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fab040077b0 tx=0x7fab04003ce0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.535+0000 7fab10b24700 1 -- 172.21.15.114:0/631191033 shutdown_connections 2024-02-19T19:03:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.535+0000 7fab10b24700 1 --2- 172.21.15.114:0/631191033 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7faaf406d660 0x7faaf406fb20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.535+0000 7fab10b24700 1 --2- 172.21.15.114:0/631191033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab0c1016a0 0x7fab0c1963e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.535+0000 7fab10b24700 1 --2- 172.21.15.114:0/631191033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab0c103fd0 0x7fab0c196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.535+0000 7fab10b24700 1 -- 172.21.15.114:0/631191033 >> 172.21.15.114:0/631191033 conn(0x7fab0c0fb030 msgr2=0x7fab0c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.535+0000 7fab10b24700 1 -- 172.21.15.114:0/631191033 shutdown_connections 2024-02-19T19:03:08.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.535+0000 7fab10b24700 1 -- 172.21.15.114:0/631191033 wait complete. 2024-02-19T19:03:08.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.642+0000 7f75a0a6f700 1 -- 172.21.15.114:0/1471887080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f759c076990 msgr2=0x7f759c076e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:08.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.642+0000 7f75a0a6f700 1 --2- 172.21.15.114:0/1471887080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f759c076990 0x7f759c076e10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f758c009910 tx=0x7f758c009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:08.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.642+0000 7f75a0a6f700 1 -- 172.21.15.114:0/1471887080 shutdown_connections 2024-02-19T19:03:08.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.642+0000 7f75a0a6f700 1 --2- 172.21.15.114:0/1471887080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f759c076990 0x7f759c076e10 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.642+0000 7f75a0a6f700 1 --2- 172.21.15.114:0/1471887080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f759c075740 0x7f759c075b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.642+0000 7f75a0a6f700 1 -- 172.21.15.114:0/1471887080 >> 172.21.15.114:0/1471887080 conn(0x7f759c0fe390 msgr2=0x7f759c1007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:08.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.642+0000 7f75a0a6f700 1 -- 172.21.15.114:0/1471887080 shutdown_connections 2024-02-19T19:03:08.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.642+0000 7f75a0a6f700 1 -- 172.21.15.114:0/1471887080 wait complete. 2024-02-19T19:03:08.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.642+0000 7f75a0a6f700 1 Processor -- start 2024-02-19T19:03:08.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.642+0000 7f75a0a6f700 1 -- start start 2024-02-19T19:03:08.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.643+0000 7f75a0a6f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f759c075740 0x7f759c19c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:08.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.643+0000 7f75a0a6f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f759c076990 0x7f759c19cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:08.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.643+0000 7f75a0a6f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c19d500 con 0x7f759c076990 2024-02-19T19:03:08.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.643+0000 7f75a0a6f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c19d640 con 0x7f759c075740 2024-02-19T19:03:08.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.643+0000 7f7599d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f759c076990 0x7f759c19cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:08.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.643+0000 7f7599d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f759c076990 0x7f759c19cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52660/0 (socket says 172.21.15.114:52660) 2024-02-19T19:03:08.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.643+0000 7f7599d9b700 1 -- 172.21.15.114:0/2113181495 learned_addr learned my addr 172.21.15.114:0/2113181495 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:03:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.643+0000 7f759a59c700 1 --2- 172.21.15.114:0/2113181495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f759c075740 0x7f759c19c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.643+0000 7f7599d9b700 1 -- 172.21.15.114:0/2113181495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f759c075740 msgr2=0x7f759c19c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.643+0000 7f7599d9b700 1 --2- 172.21.15.114:0/2113181495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f759c075740 0x7f759c19c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.643+0000 7f7599d9b700 1 -- 172.21.15.114:0/2113181495 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758c0095f0 con 0x7f759c076990 2024-02-19T19:03:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.643+0000 7f7599d9b700 1 --2- 172.21.15.114:0/2113181495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f759c076990 0x7f759c19cee0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f758c006010 tx=0x7f758c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.643+0000 7f758b7fe700 1 -- 172.21.15.114:0/2113181495 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f758c01d070 con 0x7f759c076990 2024-02-19T19:03:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.643+0000 7f758b7fe700 1 -- 172.21.15.114:0/2113181495 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f758c00f460 con 0x7f759c076990 2024-02-19T19:03:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.643+0000 7f75a0a6f700 1 -- 172.21.15.114:0/2113181495 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759c1a2090 con 0x7f759c076990 2024-02-19T19:03:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.643+0000 7f758b7fe700 1 -- 172.21.15.114:0/2113181495 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f758c021640 con 0x7f759c076990 2024-02-19T19:03:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.643+0000 7f75a0a6f700 1 -- 172.21.15.114:0/2113181495 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759c1a2600 con 0x7f759c076990 2024-02-19T19:03:08.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.644+0000 7f75a0a6f700 1 -- 172.21.15.114:0/2113181495 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759c10fda0 con 0x7f759c076990 2024-02-19T19:03:08.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.645+0000 7f758b7fe700 1 -- 172.21.15.114:0/2113181495 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 29) v1 ==== 91450+0+0 (secure 0 0 0) 0x7f758c00fab0 con 0x7f759c076990 2024-02-19T19:03:08.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.645+0000 7f758b7fe700 1 --2- 172.21.15.114:0/2113181495 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f758406d660 0x7f758406fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:08.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.646+0000 7f758b7fe700 1 -- 172.21.15.114:0/2113181495 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f758c08e640 con 0x7f759c076990 2024-02-19T19:03:08.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.646+0000 7f759a59c700 1 --2- 172.21.15.114:0/2113181495 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f758406d660 0x7f758406fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:08.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.646+0000 7f759a59c700 1 --2- 172.21.15.114:0/2113181495 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f758406d660 0x7f758406fb20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7590009a10 tx=0x7f7590009190 comp rx=0 tx=0).ready entity=mgr.14744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:08.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.650+0000 7f758b7fe700 1 -- 172.21.15.114:0/2113181495 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f758c05bc30 con 0x7f759c076990 2024-02-19T19:03:08.717 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:08 smithi114 ceph-mon[29441]: Reconfiguring prometheus.smithi114 (dependencies changed)... 2024-02-19T19:03:08.717 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:08 smithi114 ceph-mon[29441]: Reconfiguring daemon prometheus.smithi114 on smithi114 2024-02-19T19:03:08.717 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:08 smithi114 ceph-mon[29441]: pgmap v9: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:03:08.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:08 smithi174 ceph-mon[41061]: Reconfiguring prometheus.smithi114 (dependencies changed)... 2024-02-19T19:03:08.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:08 smithi174 ceph-mon[41061]: Reconfiguring daemon prometheus.smithi114 on smithi114 2024-02-19T19:03:08.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:08 smithi174 ceph-mon[41061]: pgmap v9: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:03:08.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.904+0000 7f75a0a6f700 1 -- 172.21.15.114:0/2113181495 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f759c1a2c40 con 0x7f759c076990 2024-02-19T19:03:08.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.904+0000 7f758b7fe700 1 -- 172.21.15.114:0/2113181495 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1850 (secure 0 0 0) 0x7f758c05b7c0 con 0x7f759c076990 2024-02-19T19:03:08.923 INFO:teuthology.orchestra.run.smithi114.stdout:e18 2024-02-19T19:03:08.924 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:03:08.924 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:03:08.924 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-02-19T19:03:08.924 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:03:08.924 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:03:08.924 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-02-19T19:03:08.924 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 18 2024-02-19T19:03:08.924 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:03:08.924 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:03:08.924 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-02-19T19:00:21.203922+0000 2024-02-19T19:03:08.924 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-02-19T19:03:08.924 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-02-19T19:03:08.924 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-02-19T19:03:08.924 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-02-19T19:03:08.925 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-02-19T19:03:08.925 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-02-19T19:03:08.925 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-02-19T19:03:08.925 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 47 2024-02-19T19:03:08.925 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:03:08.925 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 2 2024-02-19T19:03:08.925 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1 2024-02-19T19:03:08.925 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=14530,1=24269} 2024-02-19T19:03:08.925 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-02-19T19:03:08.925 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-02-19T19:03:08.925 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-02-19T19:03:08.925 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-02-19T19:03:08.925 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-02-19T19:03:08.925 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data enabled 2024-02-19T19:03:08.926 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-02-19T19:03:08.926 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-02-19T19:03:08.926 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 1 2024-02-19T19:03:08.926 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:08.926 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.vpoeop{0:14540} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:08.926 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.znhuvd{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:08.926 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.stnfnf{1:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:08.926 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:03:08.926 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:03:08.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.906+0000 7f75a0a6f700 1 -- 172.21.15.114:0/2113181495 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f758406d660 msgr2=0x7f758406fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:08.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.906+0000 7f75a0a6f700 1 --2- 172.21.15.114:0/2113181495 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f758406d660 0x7f758406fb20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7590009a10 tx=0x7f7590009190 comp rx=0 tx=0).stop 2024-02-19T19:03:08.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.906+0000 7f75a0a6f700 1 -- 172.21.15.114:0/2113181495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f759c076990 msgr2=0x7f759c19cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:08.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.906+0000 7f75a0a6f700 1 --2- 172.21.15.114:0/2113181495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f759c076990 0x7f759c19cee0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f758c006010 tx=0x7f758c00bd70 comp rx=0 tx=0).stop 2024-02-19T19:03:08.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.906+0000 7f75a0a6f700 1 -- 172.21.15.114:0/2113181495 shutdown_connections 2024-02-19T19:03:08.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.906+0000 7f75a0a6f700 1 --2- 172.21.15.114:0/2113181495 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f758406d660 0x7f758406fb20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.906+0000 7f75a0a6f700 1 --2- 172.21.15.114:0/2113181495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f759c075740 0x7f759c19c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.906+0000 7f75a0a6f700 1 --2- 172.21.15.114:0/2113181495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f759c076990 0x7f759c19cee0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:08.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.906+0000 7f75a0a6f700 1 -- 172.21.15.114:0/2113181495 >> 172.21.15.114:0/2113181495 conn(0x7f759c0fe390 msgr2=0x7f759c10d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:08.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.906+0000 7f75a0a6f700 1 -- 172.21.15.114:0/2113181495 shutdown_connections 2024-02-19T19:03:08.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:08.906+0000 7f75a0a6f700 1 -- 172.21.15.114:0/2113181495 wait complete. 2024-02-19T19:03:08.927 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 18 2024-02-19T19:03:09.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.013+0000 7f0836bda700 1 -- 172.21.15.114:0/4163275647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08300ffab0 msgr2=0x7f08300ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:09.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.013+0000 7f0836bda700 1 --2- 172.21.15.114:0/4163275647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08300ffab0 0x7f08300ffed0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f0820009910 tx=0x7f0820009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:09.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.013+0000 7f0836bda700 1 -- 172.21.15.114:0/4163275647 shutdown_connections 2024-02-19T19:03:09.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.013+0000 7f0836bda700 1 --2- 172.21.15.114:0/4163275647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0830100cb0 0x7f0830101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:09.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.013+0000 7f0836bda700 1 --2- 172.21.15.114:0/4163275647 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08300ffab0 0x7f08300ffed0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:09.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.013+0000 7f0836bda700 1 -- 172.21.15.114:0/4163275647 >> 172.21.15.114:0/4163275647 conn(0x7f08300fb030 msgr2=0x7f08300fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:09.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.013+0000 7f0836bda700 1 -- 172.21.15.114:0/4163275647 shutdown_connections 2024-02-19T19:03:09.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.013+0000 7f0836bda700 1 -- 172.21.15.114:0/4163275647 wait complete. 2024-02-19T19:03:09.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.014+0000 7f0836bda700 1 Processor -- start 2024-02-19T19:03:09.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.014+0000 7f0836bda700 1 -- start start 2024-02-19T19:03:09.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.014+0000 7f0836bda700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08300ffab0 0x7f08301952e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:09.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.014+0000 7f0836bda700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0830100cb0 0x7f0830195820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.014+0000 7f0836bda700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0830195e40 con 0x7f0830100cb0 2024-02-19T19:03:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.014+0000 7f0836bda700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0830195f80 con 0x7f08300ffab0 2024-02-19T19:03:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.014+0000 7f082ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0830100cb0 0x7f0830195820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.014+0000 7f082ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0830100cb0 0x7f0830195820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52666/0 (socket says 172.21.15.114:52666) 2024-02-19T19:03:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.014+0000 7f082ffff700 1 -- 172.21.15.114:0/1862077851 learned_addr learned my addr 172.21.15.114:0/1862077851 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:03:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.014+0000 7f0834976700 1 --2- 172.21.15.114:0/1862077851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08300ffab0 0x7f08301952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.014+0000 7f082ffff700 1 -- 172.21.15.114:0/1862077851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08300ffab0 msgr2=0x7f08301952e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.014+0000 7f082ffff700 1 --2- 172.21.15.114:0/1862077851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08300ffab0 0x7f08301952e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.014+0000 7f082ffff700 1 -- 172.21.15.114:0/1862077851 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08200095f0 con 0x7f0830100cb0 2024-02-19T19:03:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.014+0000 7f082ffff700 1 --2- 172.21.15.114:0/1862077851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0830100cb0 0x7f0830195820 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f082400b4b0 tx=0x7f082400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.015+0000 7f082dffb700 1 -- 172.21.15.114:0/1862077851 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0824011800 con 0x7f0830100cb0 2024-02-19T19:03:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.015+0000 7f082dffb700 1 -- 172.21.15.114:0/1862077851 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0824011e10 con 0x7f0830100cb0 2024-02-19T19:03:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.015+0000 7f0836bda700 1 -- 172.21.15.114:0/1862077851 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083019aa30 con 0x7f0830100cb0 2024-02-19T19:03:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.015+0000 7f082dffb700 1 -- 172.21.15.114:0/1862077851 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f082400f2e0 con 0x7f0830100cb0 2024-02-19T19:03:09.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.015+0000 7f0836bda700 1 -- 172.21.15.114:0/1862077851 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083019b000 con 0x7f0830100cb0 2024-02-19T19:03:09.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.016+0000 7f0836bda700 1 -- 172.21.15.114:0/1862077851 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0830108690 con 0x7f0830100cb0 2024-02-19T19:03:09.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.016+0000 7f082dffb700 1 -- 172.21.15.114:0/1862077851 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 29) v1 ==== 91450+0+0 (secure 0 0 0) 0x7f0824010380 con 0x7f0830100cb0 2024-02-19T19:03:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.016+0000 7f082dffb700 1 --2- 172.21.15.114:0/1862077851 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f081806d660 0x7f081806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.017+0000 7f082dffb700 1 -- 172.21.15.114:0/1862077851 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f082408cd10 con 0x7f0830100cb0 2024-02-19T19:03:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.017+0000 7f0834976700 1 --2- 172.21.15.114:0/1862077851 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f081806d660 0x7f081806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:09.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.017+0000 7f0834976700 1 --2- 172.21.15.114:0/1862077851 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f081806d660 0x7f081806fb20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f08200059d0 tx=0x7f0820005960 comp rx=0 tx=0).ready entity=mgr.14744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:09.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.021+0000 7f082dffb700 1 -- 172.21.15.114:0/1862077851 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f082405a410 con 0x7f0830100cb0 2024-02-19T19:03:09.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.249+0000 7f0836bda700 1 -- 172.21.15.114:0/1862077851 --> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0830068d50 con 0x7f081806d660 2024-02-19T19:03:09.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.250+0000 7f082dffb700 1 -- 172.21.15.114:0/1862077851 <== mgr.14744 v2:172.21.15.114:6800/1408766633 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f0830068d50 con 0x7f081806d660 2024-02-19T19:03:09.270 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:03:09.270 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:03:09.270 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-02-19T19:03:09.270 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:03:09.270 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [ 2024-02-19T19:03:09.270 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr" 2024-02-19T19:03:09.270 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-02-19T19:03:09.270 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "2/23 daemons upgraded", 2024-02-19T19:03:09.270 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "", 2024-02-19T19:03:09.271 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-02-19T19:03:09.271 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:03:09.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.253+0000 7f0836bda700 1 -- 172.21.15.114:0/1862077851 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f081806d660 msgr2=0x7f081806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:09.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.253+0000 7f0836bda700 1 --2- 172.21.15.114:0/1862077851 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f081806d660 0x7f081806fb20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f08200059d0 tx=0x7f0820005960 comp rx=0 tx=0).stop 2024-02-19T19:03:09.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.253+0000 7f0836bda700 1 -- 172.21.15.114:0/1862077851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0830100cb0 msgr2=0x7f0830195820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:09.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.253+0000 7f0836bda700 1 --2- 172.21.15.114:0/1862077851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0830100cb0 0x7f0830195820 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f082400b4b0 tx=0x7f082400b870 comp rx=0 tx=0).stop 2024-02-19T19:03:09.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.253+0000 7f0836bda700 1 -- 172.21.15.114:0/1862077851 shutdown_connections 2024-02-19T19:03:09.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.253+0000 7f0836bda700 1 --2- 172.21.15.114:0/1862077851 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f081806d660 0x7f081806fb20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:09.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.253+0000 7f0836bda700 1 --2- 172.21.15.114:0/1862077851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08300ffab0 0x7f08301952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:09.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.253+0000 7f0836bda700 1 --2- 172.21.15.114:0/1862077851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0830100cb0 0x7f0830195820 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:09.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.253+0000 7f0836bda700 1 -- 172.21.15.114:0/1862077851 >> 172.21.15.114:0/1862077851 conn(0x7f08300fb030 msgr2=0x7f0830103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:09.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.253+0000 7f0836bda700 1 -- 172.21.15.114:0/1862077851 shutdown_connections 2024-02-19T19:03:09.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.253+0000 7f0836bda700 1 -- 172.21.15.114:0/1862077851 wait complete. 2024-02-19T19:03:09.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.360+0000 7f5ceedd0700 1 -- 172.21.15.114:0/612442668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce8100cd0 msgr2=0x7f5ce8101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:09.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.360+0000 7f5ceedd0700 1 --2- 172.21.15.114:0/612442668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce8100cd0 0x7f5ce8101130 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009910 tx=0x7f5cd4009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:09.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.360+0000 7f5ceedd0700 1 -- 172.21.15.114:0/612442668 shutdown_connections 2024-02-19T19:03:09.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.360+0000 7f5ceedd0700 1 --2- 172.21.15.114:0/612442668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce8100cd0 0x7f5ce8101130 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:09.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.360+0000 7f5ceedd0700 1 --2- 172.21.15.114:0/612442668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ce80ffad0 0x7f5ce80ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:09.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.360+0000 7f5ceedd0700 1 -- 172.21.15.114:0/612442668 >> 172.21.15.114:0/612442668 conn(0x7f5ce80fb050 msgr2=0x7f5ce80fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:09.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.360+0000 7f5ceedd0700 1 -- 172.21.15.114:0/612442668 shutdown_connections 2024-02-19T19:03:09.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.360+0000 7f5ceedd0700 1 -- 172.21.15.114:0/612442668 wait complete. 2024-02-19T19:03:09.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.361+0000 7f5ceedd0700 1 Processor -- start 2024-02-19T19:03:09.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.361+0000 7f5ceedd0700 1 -- start start 2024-02-19T19:03:09.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.361+0000 7f5ceedd0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80ffad0 0x7f5ce8195280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:09.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.361+0000 7f5ceedd0700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ce8100cd0 0x7f5ce81957c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:09.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.361+0000 7f5ceedd0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce8195de0 con 0x7f5ce80ffad0 2024-02-19T19:03:09.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.361+0000 7f5ceedd0700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce8195f20 con 0x7f5ce8100cd0 2024-02-19T19:03:09.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.361+0000 7f5cecb6c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80ffad0 0x7f5ce8195280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:09.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.361+0000 7f5cecb6c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80ffad0 0x7f5ce8195280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52684/0 (socket says 172.21.15.114:52684) 2024-02-19T19:03:09.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.361+0000 7f5cecb6c700 1 -- 172.21.15.114:0/1984430692 learned_addr learned my addr 172.21.15.114:0/1984430692 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:03:09.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.362+0000 7f5ce7fff700 1 --2- 172.21.15.114:0/1984430692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ce8100cd0 0x7f5ce81957c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:09.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.362+0000 7f5cecb6c700 1 -- 172.21.15.114:0/1984430692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ce8100cd0 msgr2=0x7f5ce81957c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:09.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.362+0000 7f5cecb6c700 1 --2- 172.21.15.114:0/1984430692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ce8100cd0 0x7f5ce81957c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:09.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.362+0000 7f5cecb6c700 1 -- 172.21.15.114:0/1984430692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd40095f0 con 0x7f5ce80ffad0 2024-02-19T19:03:09.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.362+0000 7f5ce7fff700 1 --2- 172.21.15.114:0/1984430692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ce8100cd0 0x7f5ce81957c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:03:09.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.362+0000 7f5cecb6c700 1 --2- 172.21.15.114:0/1984430692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80ffad0 0x7f5ce8195280 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc00d900 tx=0x7f5cdc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:09.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.362+0000 7f5ce5ffb700 1 -- 172.21.15.114:0/1984430692 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cdc0041f0 con 0x7f5ce80ffad0 2024-02-19T19:03:09.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.362+0000 7f5ce5ffb700 1 -- 172.21.15.114:0/1984430692 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5cdc004350 con 0x7f5ce80ffad0 2024-02-19T19:03:09.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.362+0000 7f5ceedd0700 1 -- 172.21.15.114:0/1984430692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce819a9d0 con 0x7f5ce80ffad0 2024-02-19T19:03:09.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.362+0000 7f5ce5ffb700 1 -- 172.21.15.114:0/1984430692 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cdc003da0 con 0x7f5ce80ffad0 2024-02-19T19:03:09.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.362+0000 7f5ceedd0700 1 -- 172.21.15.114:0/1984430692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce819af20 con 0x7f5ce80ffad0 2024-02-19T19:03:09.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.363+0000 7f5ceedd0700 1 -- 172.21.15.114:0/1984430692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce8108630 con 0x7f5ce80ffad0 2024-02-19T19:03:09.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.364+0000 7f5ce5ffb700 1 -- 172.21.15.114:0/1984430692 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 29) v1 ==== 91450+0+0 (secure 0 0 0) 0x7f5cdc009730 con 0x7f5ce80ffad0 2024-02-19T19:03:09.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.364+0000 7f5ce5ffb700 1 --2- 172.21.15.114:0/1984430692 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f5cd806d660 0x7f5cd806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:09.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.364+0000 7f5ce5ffb700 1 -- 172.21.15.114:0/1984430692 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f5cdc08d340 con 0x7f5ce80ffad0 2024-02-19T19:03:09.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.364+0000 7f5ce7fff700 1 --2- 172.21.15.114:0/1984430692 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f5cd806d660 0x7f5cd806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:09.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.365+0000 7f5ce7fff700 1 --2- 172.21.15.114:0/1984430692 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f5cd806d660 0x7f5cd806fb20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4005b40 tx=0x7f5cd4005a90 comp rx=0 tx=0).ready entity=mgr.14744 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:09.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.369+0000 7f5ce5ffb700 1 -- 172.21.15.114:0/1984430692 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182869 (secure 0 0 0) 0x7f5cdc05a930 con 0x7f5ce80ffad0 2024-02-19T19:03:09.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:09 smithi114 ceph-mon[29441]: from='client.24475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:09.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:09 smithi114 ceph-mon[29441]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:09.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:09 smithi114 ceph-mon[29441]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:09.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:09 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/631191033' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:09.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:09 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/2113181495' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:03:09.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.654+0000 7f5ceedd0700 1 -- 172.21.15.114:0/1984430692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5ce819b5f0 con 0x7f5ce80ffad0 2024-02-19T19:03:09.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.655+0000 7f5ce5ffb700 1 -- 172.21.15.114:0/1984430692 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f5cdc05a4c0 con 0x7f5ce80ffad0 2024-02-19T19:03:09.674 INFO:teuthology.orchestra.run.smithi114.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:03:09.674 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:03:09.675 INFO:teuthology.orchestra.run.smithi114.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:03:09.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.657+0000 7f5ceedd0700 1 -- 172.21.15.114:0/1984430692 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f5cd806d660 msgr2=0x7f5cd806fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:09.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.657+0000 7f5ceedd0700 1 --2- 172.21.15.114:0/1984430692 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f5cd806d660 0x7f5cd806fb20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4005b40 tx=0x7f5cd4005a90 comp rx=0 tx=0).stop 2024-02-19T19:03:09.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.657+0000 7f5ceedd0700 1 -- 172.21.15.114:0/1984430692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80ffad0 msgr2=0x7f5ce8195280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:09.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.657+0000 7f5ceedd0700 1 --2- 172.21.15.114:0/1984430692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80ffad0 0x7f5ce8195280 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc00d900 tx=0x7f5cdc00dcc0 comp rx=0 tx=0).stop 2024-02-19T19:03:09.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.657+0000 7f5ceedd0700 1 -- 172.21.15.114:0/1984430692 shutdown_connections 2024-02-19T19:03:09.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.657+0000 7f5ceedd0700 1 --2- 172.21.15.114:0/1984430692 >> [v2:172.21.15.114:6800/1408766633,v1:172.21.15.114:6801/1408766633] conn(0x7f5cd806d660 0x7f5cd806fb20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:09.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.657+0000 7f5ceedd0700 1 --2- 172.21.15.114:0/1984430692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ce80ffad0 0x7f5ce8195280 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:09.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.657+0000 7f5ceedd0700 1 --2- 172.21.15.114:0/1984430692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ce8100cd0 0x7f5ce81957c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:09.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.657+0000 7f5ceedd0700 1 -- 172.21.15.114:0/1984430692 >> 172.21.15.114:0/1984430692 conn(0x7f5ce80fb050 msgr2=0x7f5ce8103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:09.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.657+0000 7f5ceedd0700 1 -- 172.21.15.114:0/1984430692 shutdown_connections 2024-02-19T19:03:09.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:09.657+0000 7f5ceedd0700 1 -- 172.21.15.114:0/1984430692 wait complete. 2024-02-19T19:03:09.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:09 smithi174 ceph-mon[41061]: from='client.24475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:09.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:09 smithi174 ceph-mon[41061]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:09.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:09 smithi174 ceph-mon[41061]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:09.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:09 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/631191033' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:09.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:09 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/2113181495' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:03:10.700 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:10 smithi114 ceph-mon[29441]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:10.700 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:10 smithi114 ceph-mon[29441]: pgmap v10: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:03:10.700 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:10 smithi114 ceph-mon[29441]: from='client.? 172.21.15.114:0/1984430692' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:03:10.700 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:10 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:10.722 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:10 smithi174 ceph-mon[41061]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:10.722 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:10 smithi174 ceph-mon[41061]: pgmap v10: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:03:10.722 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:10 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1984430692' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:03:10.722 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:10 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:12.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:12 smithi114 ceph-mon[29441]: pgmap v11: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-19T19:03:12.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:12 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:12.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:12 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:12.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:12 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:03:12.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:12 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:12.862 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:12 smithi174 ceph-mon[41061]: pgmap v11: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-19T19:03:12.862 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:12 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:12.862 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:12 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:12.862 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:12 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:03:12.862 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:12 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:13.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:13 smithi114 ceph-mon[29441]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:03:13.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:13 smithi114 ceph-mon[29441]: pgmap v12: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:03:13.889 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:13 smithi174 ceph-mon[41061]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:03:13.889 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:13 smithi174 ceph-mon[41061]: pgmap v12: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:03:14.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:14 smithi114 ceph-mon[29441]: Upgrade: Updating mgr.smithi174.lydvnj 2024-02-19T19:03:14.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:14 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi174.lydvnj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:03:14.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:14 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:03:14.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:14 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:03:14.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:14 smithi114 ceph-mon[29441]: Deploying daemon mgr.smithi174.lydvnj on smithi174 2024-02-19T19:03:14.746 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:14 smithi174 ceph-mon[41061]: Upgrade: Updating mgr.smithi174.lydvnj 2024-02-19T19:03:14.746 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:14 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi174.lydvnj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:03:14.746 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:14 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:03:14.746 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:14 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:03:14.746 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:14 smithi174 ceph-mon[41061]: Deploying daemon mgr.smithi174.lydvnj on smithi174 2024-02-19T19:03:15.609 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:15 smithi174 ceph-mon[41061]: pgmap v13: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:03:15.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:15 smithi114 ceph-mon[29441]: pgmap v13: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:03:18.487 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:18 smithi174 ceph-mon[41061]: pgmap v14: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:03:18.487 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:18 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:18.487 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:18 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:18.487 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:18 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:18.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:18 smithi114 ceph-mon[29441]: pgmap v14: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:03:18.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:18 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:18.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:18 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:18.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:18 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:20.597 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:20 smithi114 ceph-mon[29441]: pgmap v15: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:03:20.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:20 smithi174 ceph-mon[41061]: pgmap v15: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:03:22.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:21 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:22.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:21 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:22.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:21 smithi114 ceph-mon[29441]: pgmap v16: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:03:22.327 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:21 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:22.327 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:21 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:22.327 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:21 smithi174 ceph-mon[41061]: pgmap v16: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:03:23.362 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:23 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:23.363 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:23 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:23.475 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:23 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:23.475 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:23 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:24.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:24 smithi174 ceph-mon[41061]: pgmap v17: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:03:24.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:24 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:24.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:24 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:24.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:24 smithi114 ceph-mon[29441]: pgmap v17: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:03:24.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:24 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:24.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:24 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:25.393 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:25 smithi174 ceph-mon[41061]: Standby manager daemon smithi174.lydvnj restarted 2024-02-19T19:03:25.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:25 smithi174 ceph-mon[41061]: Standby manager daemon smithi174.lydvnj started 2024-02-19T19:03:25.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:25 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/450406209' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/crt"}]: dispatch 2024-02-19T19:03:25.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:25 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/450406209' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:03:25.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:25 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/450406209' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/key"}]: dispatch 2024-02-19T19:03:25.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:25 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/450406209' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:03:25.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:25 smithi114 ceph-mon[29441]: Standby manager daemon smithi174.lydvnj restarted 2024-02-19T19:03:25.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:25 smithi114 ceph-mon[29441]: Standby manager daemon smithi174.lydvnj started 2024-02-19T19:03:25.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:25 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.174:0/450406209' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/crt"}]: dispatch 2024-02-19T19:03:25.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:25 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.174:0/450406209' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:03:25.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:25 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.174:0/450406209' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/key"}]: dispatch 2024-02-19T19:03:25.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:25 smithi114 ceph-mon[29441]: from='mgr.? 172.21.15.174:0/450406209' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:03:26.516 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:26 smithi114 ceph-mon[29441]: mgrmap e30: smithi114.pcvtrk(active, since 29s), standbys: smithi174.lydvnj 2024-02-19T19:03:26.516 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:26 smithi114 ceph-mon[29441]: pgmap v18: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:03:26.516 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:26 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:26.516 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:26 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:26.516 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:26 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:26.516 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:26 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:03:26.516 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:26 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:03:26.516 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:26 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:26.516 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:26 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:26.516 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:26 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:26.516 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:26 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:26.517 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:26 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi114.pcvtrk"}]: dispatch 2024-02-19T19:03:26.517 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:26 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi114.pcvtrk"}]': finished 2024-02-19T19:03:26.517 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:26 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi174.lydvnj"}]: dispatch 2024-02-19T19:03:26.517 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:26 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi174.lydvnj"}]': finished 2024-02-19T19:03:26.517 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:26 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-19T19:03:26.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:26 smithi174 ceph-mon[41061]: mgrmap e30: smithi114.pcvtrk(active, since 29s), standbys: smithi174.lydvnj 2024-02-19T19:03:26.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:26 smithi174 ceph-mon[41061]: pgmap v18: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:03:26.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:26 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:26.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:26 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:26.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:26 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:26.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:26 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:03:26.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:26 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:03:26.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:26 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:26.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:26 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:26.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:26 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:26.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:26 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:26.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:26 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi114.pcvtrk"}]: dispatch 2024-02-19T19:03:26.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:26 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi114.pcvtrk"}]': finished 2024-02-19T19:03:26.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:26 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi174.lydvnj"}]: dispatch 2024-02-19T19:03:26.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:26 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi174.lydvnj"}]': finished 2024-02-19T19:03:26.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:26 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-19T19:03:27.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:27 smithi114 ceph-mon[29441]: Upgrade: Setting container_image for all mgr 2024-02-19T19:03:27.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:27 smithi174 ceph-mon[41061]: Upgrade: Setting container_image for all mgr 2024-02-19T19:03:28.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:28 smithi114 ceph-mon[29441]: pgmap v19: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:03:28.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:28 smithi114 ceph-mon[29441]: Upgrade: Updating mon.smithi114 2024-02-19T19:03:28.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:28 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:28.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:28 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T19:03:28.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:28 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T19:03:28.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:28 smithi114 ceph-mon[29441]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:03:28.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:28 smithi174 ceph-mon[41061]: pgmap v19: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:03:28.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:28 smithi174 ceph-mon[41061]: Upgrade: Updating mon.smithi114 2024-02-19T19:03:28.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:28 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:28.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:28 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T19:03:28.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:28 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T19:03:28.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:28 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:03:29.561 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:29 smithi114 systemd[1]: Stopping Ceph mon.smithi114 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T19:03:29.561 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:29 smithi114 ceph-mon[29441]: Deploying daemon mon.smithi114 on smithi114 2024-02-19T19:03:29.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:29 smithi174 ceph-mon[41061]: Deploying daemon mon.smithi114 on smithi114 2024-02-19T19:03:29.963 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[29437]: 2024-02-19T19:03:29.541+0000 7f35cd51d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi114 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T19:03:29.963 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[29437]: 2024-02-19T19:03:29.541+0000 7f35cd51d700 -1 mon.smithi114@0(leader) e2 *** Got Signal Terminated *** 2024-02-19T19:03:29.963 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:29 smithi114 podman[92805]: 2024-02-19 19:03:29.633854277 +0000 UTC m=+0.206481833 container died 41468f811fc071a67d1e0ce6a9934c9a386bce3049acd8ce14d8d948c6908255 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214) 2024-02-19T19:03:30.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:30 smithi114 podman[92805]: 2024-02-19 19:03:30.051396454 +0000 UTC m=+0.624024010 container cleanup 41468f811fc071a67d1e0ce6a9934c9a386bce3049acd8ce14d8d948c6908255 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True) 2024-02-19T19:03:30.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:30 smithi114 bash[92805]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114 2024-02-19T19:03:30.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:30 smithi114 podman[92830]: 2024-02-19 19:03:30.286204051 +0000 UTC m=+0.656436744 container remove 41468f811fc071a67d1e0ce6a9934c9a386bce3049acd8ce14d8d948c6908255 (image=quay.io/ceph/ceph:v18.2.1, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, ceph=True, io.buildah.version=1.29.1) 2024-02-19T19:03:30.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:30 smithi114 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@mon.smithi114.service: Deactivated successfully. 2024-02-19T19:03:30.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:30 smithi114 systemd[1]: Stopped Ceph mon.smithi114 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T19:03:30.557 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:30 smithi114 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@mon.smithi114.service: Consumed 11.507s CPU time. 2024-02-19T19:03:30.961 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:30 smithi114 systemd[1]: Starting Ceph mon.smithi114 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T19:03:31.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:30 smithi114 podman[92964]: 2024-02-19 19:03:30.845272948 +0000 UTC m=+0.018432999 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:03:31.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:30 smithi114 podman[92964]: 2024-02-19 19:03:30.962516173 +0000 UTC m=+0.135676280 container create e51f650f9be9c107f0a3e8b1342c8856ab70c7dc1a090e8598d81e6574275a33 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-19T19:03:31.217 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 podman[92964]: 2024-02-19 19:03:31.198627787 +0000 UTC m=+0.371787836 container init e51f650f9be9c107f0a3e8b1342c8856ab70c7dc1a090e8598d81e6574275a33 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:03:31.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 podman[92964]: 2024-02-19 19:03:31.20186442 +0000 UTC m=+0.375024469 container start e51f650f9be9c107f0a3e8b1342c8856ab70c7dc1a090e8598d81e6574275a33 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:03:31.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: set uid:gid to 167:167 (ceph:ceph) 2024-02-19T19:03:31.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev), process ceph-mon, pid 2 2024-02-19T19:03:31.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: pidfile_write: ignore empty --pid-file 2024-02-19T19:03:31.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: load: jerasure load: lrc 2024-02-19T19:03:31.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: RocksDB version: 7.9.2 2024-02-19T19:03:31.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Git sha 0 2024-02-19T19:03:31.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Compile date 2024-02-15 18:05:23 2024-02-19T19:03:31.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: DB SUMMARY 2024-02-19T19:03:31.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: DB Session ID: Q62C01XEABNOB8KXU79J 2024-02-19T19:03:31.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: CURRENT file: CURRENT 2024-02-19T19:03:31.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: IDENTITY file: IDENTITY 2024-02-19T19:03:31.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1517 Bytes 2024-02-19T19:03:31.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi114/store.db dir, Total Num: 1, files: 000032.sst 2024-02-19T19:03:31.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi114/store.db: 000030.log size: 2459984 ; 2024-02-19T19:03:31.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.error_if_exists: 0 2024-02-19T19:03:31.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.create_if_missing: 0 2024-02-19T19:03:31.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.paranoid_checks: 1 2024-02-19T19:03:31.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-19T19:03:31.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-19T19:03:31.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-19T19:03:31.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.env: 0x561efe7baae0 2024-02-19T19:03:31.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.fs: PosixFileSystem 2024-02-19T19:03:31.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.info_log: 0x561eff363760 2024-02-19T19:03:31.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_file_opening_threads: 16 2024-02-19T19:03:31.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.statistics: (nil) 2024-02-19T19:03:31.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.use_fsync: 0 2024-02-19T19:03:31.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_log_file_size: 0 2024-02-19T19:03:31.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T19:03:31.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T19:03:31.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.keep_log_file_num: 1000 2024-02-19T19:03:31.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.recycle_log_file_num: 0 2024-02-19T19:03:31.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.allow_fallocate: 1 2024-02-19T19:03:31.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.allow_mmap_reads: 0 2024-02-19T19:03:31.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.allow_mmap_writes: 0 2024-02-19T19:03:31.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.use_direct_reads: 0 2024-02-19T19:03:31.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T19:03:31.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.create_missing_column_families: 0 2024-02-19T19:03:31.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.db_log_dir: 2024-02-19T19:03:31.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.wal_dir: 2024-02-19T19:03:31.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T19:03:31.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T19:03:31.543 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T19:03:31.544 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T19:03:31.544 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T19:03:31.544 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T19:03:31.544 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.advise_random_on_open: 1 2024-02-19T19:03:31.544 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.db_write_buffer_size: 0 2024-02-19T19:03:31.544 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.write_buffer_manager: 0x561eff337400 2024-02-19T19:03:31.544 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T19:03:31.544 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T19:03:31.544 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T19:03:31.544 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.rate_limiter: (nil) 2024-02-19T19:03:31.544 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T19:03:31.544 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.wal_recovery_mode: 2 2024-02-19T19:03:31.545 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.enable_thread_tracking: 0 2024-02-19T19:03:31.545 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.enable_pipelined_write: 0 2024-02-19T19:03:31.545 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.unordered_write: 0 2024-02-19T19:03:31.545 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T19:03:31.545 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T19:03:31.545 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T19:03:31.545 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T19:03:31.545 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.row_cache: None 2024-02-19T19:03:31.545 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.wal_filter: None 2024-02-19T19:03:31.545 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T19:03:31.545 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.allow_ingest_behind: 0 2024-02-19T19:03:31.545 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.two_write_queues: 0 2024-02-19T19:03:31.545 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.manual_wal_flush: 0 2024-02-19T19:03:31.545 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.wal_compression: 0 2024-02-19T19:03:31.546 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.atomic_flush: 0 2024-02-19T19:03:31.546 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T19:03:31.546 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T19:03:31.546 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T19:03:31.546 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.log_readahead_size: 0 2024-02-19T19:03:31.546 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-19T19:03:31.546 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.best_efforts_recovery: 0 2024-02-19T19:03:31.546 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-19T19:03:31.546 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-19T19:03:31.546 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.allow_data_in_errors: 0 2024-02-19T19:03:31.546 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.db_host_id: __hostname__ 2024-02-19T19:03:31.546 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-19T19:03:31.546 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_background_jobs: 2 2024-02-19T19:03:31.546 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_background_compactions: -1 2024-02-19T19:03:31.547 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_subcompactions: 1 2024-02-19T19:03:31.547 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T19:03:31.547 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T19:03:31.547 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T19:03:31.547 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_total_wal_size: 0 2024-02-19T19:03:31.547 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T19:03:31.547 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T19:03:31.547 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T19:03:31.547 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T19:03:31.547 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_open_files: -1 2024-02-19T19:03:31.547 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.bytes_per_sync: 0 2024-02-19T19:03:31.547 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T19:03:31.547 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T19:03:31.547 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compaction_readahead_size: 0 2024-02-19T19:03:31.547 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_background_flushes: -1 2024-02-19T19:03:31.548 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Compression algorithms supported: 2024-02-19T19:03:31.548 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T19:03:31.548 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: kZSTD supported: 0 2024-02-19T19:03:31.548 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: kXpressCompression supported: 0 2024-02-19T19:03:31.548 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: kLZ4HCCompression supported: 1 2024-02-19T19:03:31.548 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: kZlibCompression supported: 1 2024-02-19T19:03:31.548 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: kSnappyCompression supported: 1 2024-02-19T19:03:31.548 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: kLZ4Compression supported: 1 2024-02-19T19:03:31.548 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: kBZip2Compression supported: 0 2024-02-19T19:03:31.548 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T19:03:31.548 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-19T19:03:31.548 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi114/store.db/MANIFEST-000015 2024-02-19T19:03:31.548 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-19T19:03:31.548 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T19:03:31.549 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.merge_operator: 2024-02-19T19:03:31.549 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compaction_filter: None 2024-02-19T19:03:31.549 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compaction_filter_factory: None 2024-02-19T19:03:31.549 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.sst_partitioner_factory: None 2024-02-19T19:03:31.549 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T19:03:31.549 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T19:03:31.549 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561eff363880) 2024-02-19T19:03:31.549 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-02-19T19:03:31.549 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T19:03:31.549 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T19:03:31.549 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-02-19T19:03:31.549 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: index_type: 0 2024-02-19T19:03:31.549 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: data_block_index_type: 0 2024-02-19T19:03:31.550 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: index_shortening: 1 2024-02-19T19:03:31.550 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-19T19:03:31.550 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: checksum: 4 2024-02-19T19:03:31.550 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: no_block_cache: 0 2024-02-19T19:03:31.550 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_cache: 0x561eff367090 2024-02-19T19:03:31.550 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-02-19T19:03:31.550 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_cache_options: 2024-02-19T19:03:31.550 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: capacity : 536870912 2024-02-19T19:03:31.550 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: num_shard_bits : 4 2024-02-19T19:03:31.550 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: strict_capacity_limit : 0 2024-02-19T19:03:31.550 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-02-19T19:03:31.550 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_cache_compressed: (nil) 2024-02-19T19:03:31.550 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: persistent_cache: (nil) 2024-02-19T19:03:31.551 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_size: 4096 2024-02-19T19:03:31.551 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_size_deviation: 10 2024-02-19T19:03:31.551 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_restart_interval: 16 2024-02-19T19:03:31.551 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: index_block_restart_interval: 1 2024-02-19T19:03:31.551 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: metadata_block_size: 4096 2024-02-19T19:03:31.551 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: partition_filters: 0 2024-02-19T19:03:31.551 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: use_delta_encoding: 1 2024-02-19T19:03:31.551 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: filter_policy: bloomfilter 2024-02-19T19:03:31.551 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: whole_key_filtering: 1 2024-02-19T19:03:31.551 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: verify_compression: 0 2024-02-19T19:03:31.551 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-02-19T19:03:31.551 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: format_version: 5 2024-02-19T19:03:31.551 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: enable_index_compression: 1 2024-02-19T19:03:31.551 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: block_align: 0 2024-02-19T19:03:31.552 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: max_auto_readahead_size: 262144 2024-02-19T19:03:31.552 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: prepopulate_block_cache: 0 2024-02-19T19:03:31.552 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: initial_auto_readahead_size: 8192 2024-02-19T19:03:31.552 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: num_file_reads_for_auto_readahead: 2 2024-02-19T19:03:31.552 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.write_buffer_size: 33554432 2024-02-19T19:03:31.552 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_write_buffer_number: 2 2024-02-19T19:03:31.552 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compression: NoCompression 2024-02-19T19:03:31.552 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.bottommost_compression: Disabled 2024-02-19T19:03:31.552 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.prefix_extractor: nullptr 2024-02-19T19:03:31.552 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T19:03:31.552 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.num_levels: 7 2024-02-19T19:03:31.552 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T19:03:31.552 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T19:03:31.553 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T19:03:31.553 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T19:03:31.553 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T19:03:31.553 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T19:03:31.553 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T19:03:31.553 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T19:03:31.553 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-19T19:03:31.553 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T19:03:31.553 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-19T19:03:31.553 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-19T19:03:31.553 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T19:03:31.553 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compression_opts.level: 32767 2024-02-19T19:03:31.553 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compression_opts.strategy: 0 2024-02-19T19:03:31.553 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T19:03:31.553 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T19:03:31.554 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-19T19:03:31.554 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-19T19:03:31.554 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compression_opts.enabled: false 2024-02-19T19:03:31.554 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-19T19:03:31.554 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T19:03:31.554 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T19:03:31.554 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T19:03:31.554 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.target_file_size_base: 67108864 2024-02-19T19:03:31.554 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T19:03:31.554 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T19:03:31.554 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T19:03:31.554 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T19:03:31.554 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T19:03:31.554 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T19:03:31.555 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T19:03:31.555 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T19:03:31.555 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T19:03:31.555 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T19:03:31.555 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T19:03:31.555 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T19:03:31.555 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T19:03:31.555 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-19T19:03:31.555 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.arena_block_size: 1048576 2024-02-19T19:03:31.555 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T19:03:31.555 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T19:03:31.555 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.disable_auto_compactions: 0 2024-02-19T19:03:31.555 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T19:03:31.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T19:03:31.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T19:03:31.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T19:03:31.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T19:03:31.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T19:03:31.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T19:03:31.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T19:03:31.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T19:03:31.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T19:03:31.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.table_properties_collectors: 2024-02-19T19:03:31.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.inplace_update_support: 0 2024-02-19T19:03:31.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T19:03:31.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T19:03:31.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T19:03:31.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T19:03:31.557 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.bloom_locality: 0 2024-02-19T19:03:31.557 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.max_successive_merges: 0 2024-02-19T19:03:31.557 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T19:03:31.557 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.paranoid_file_checks: 0 2024-02-19T19:03:31.557 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.force_consistency_checks: 1 2024-02-19T19:03:31.557 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.report_bg_io_stats: 0 2024-02-19T19:03:31.557 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.ttl: 2592000 2024-02-19T19:03:31.557 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T19:03:31.557 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-19T19:03:31.557 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-19T19:03:31.557 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.enable_blob_files: false 2024-02-19T19:03:31.557 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.min_blob_size: 0 2024-02-19T19:03:31.557 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.blob_file_size: 268435456 2024-02-19T19:03:31.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-19T19:03:31.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-19T19:03:31.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-19T19:03:31.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-19T19:03:31.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-19T19:03:31.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.blob_file_starting_level: 0 2024-02-19T19:03:31.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-19T19:03:31.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-19T19:03:31.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi114/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 34, last_sequence is 12048, log_number is 30,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 30 2024-02-19T19:03:31.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 30 2024-02-19T19:03:31.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6047413c-9651-48e9-8fb4-2546bf462c87 2024-02-19T19:03:31.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708369411231490, "job": 1, "event": "recovery_started", "wal_files": [30]} 2024-02-19T19:03:31.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #30 mode 2 2024-02-19T19:03:31.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708369411238546, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 35, "file_size": 2227022, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12049, "largest_seqno": 13055, "table_properties": {"data_size": 2221951, "index_size": 2900, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1157, "raw_key_size": 10114, "raw_average_key_size": 23, "raw_value_size": 2212546, "raw_average_value_size": 5063, "num_data_blocks": 133, "num_entries": 437, "num_filter_entries": 437, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1708369411, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6047413c-9651-48e9-8fb4-2546bf462c87", "db_session_id": "Q62C01XEABNOB8KXU79J", "orig_file_number": 35, "seqno_to_time_mapping": "N/A"}} 2024-02-19T19:03:31.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708369411238620, "job": 1, "event": "recovery_finished"} 2024-02-19T19:03:31.559 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: [db/version_set.cc:5047] Creating manifest 37 2024-02-19T19:03:31.559 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-19T19:03:31.559 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi114/store.db/000030.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-19T19:03:31.559 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561eff456000 2024-02-19T19:03:31.559 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: DB pointer 0x561eff442000 2024-02-19T19:03:31.559 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-19T19:03:31.559 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-19T19:03:31.559 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ** DB Stats ** 2024-02-19T19:03:31.559 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T19:03:31.559 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T19:03:31.559 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T19:03:31.559 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T19:03:31.559 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T19:03:31.560 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T19:03:31.560 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T19:03:31.560 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-02-19T19:03:31.560 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ** Compaction Stats [default] ** 2024-02-19T19:03:31.560 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-19T19:03:31.560 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-19T19:03:31.560 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: L0 1/0 2.12 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 606.1 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-19T19:03:31.560 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: L6 1/0 4.76 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-19T19:03:31.560 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Sum 2/0 6.88 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 606.1 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-19T19:03:31.560 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 606.1 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-19T19:03:31.560 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-02-19T19:03:31.560 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ** Compaction Stats [default] ** 2024-02-19T19:03:31.560 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-19T19:03:31.561 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T19:03:31.561 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 606.1 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-19T19:03:31.561 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-02-19T19:03:31.561 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-19T19:03:31.561 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-02-19T19:03:31.561 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T19:03:31.561 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-02-19T19:03:31.561 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T19:03:31.561 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T19:03:31.561 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T19:03:31.561 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-19T19:03:31.561 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Cumulative compaction: 0.00 GB write, 186.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T19:03:31.561 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Interval compaction: 0.00 GB write, 186.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T19:03:31.561 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T19:03:31.561 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Block cache BinnedLRUCache@0x561eff367090#2 capacity: 512.00 MB usage: 4.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-02-19T19:03:31.562 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.28 KB,0.000244379%) IndexBlock(1,3.08 KB,0.000587106%) Misc(1,0.00 KB,0%) 2024-02-19T19:03:31.562 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-02-19T19:03:31.562 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-19T19:03:31.562 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: starting mon.smithi114 rank 0 at public addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] at bind addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi114 fsid be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T19:03:31.562 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: mon.smithi114@-1(???) e2 preinit fsid be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T19:03:31.562 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: mon.smithi114@-1(???).mds e18 new map 2024-02-19T19:03:31.562 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: mon.smithi114@-1(???).mds e18 print_map 2024-02-19T19:03:31.562 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: e18 2024-02-19T19:03:31.562 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:03:31.562 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:03:31.562 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: legacy client fscid: 1 2024-02-19T19:03:31.562 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-02-19T19:03:31.562 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: Filesystem 'cephfs' (1) 2024-02-19T19:03:31.562 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: fs_name cephfs 2024-02-19T19:03:31.562 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: epoch 18 2024-02-19T19:03:31.563 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:03:31.563 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:03:31.563 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: modified 2024-02-19T19:00:21.203922+0000 2024-02-19T19:03:31.563 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: tableserver 0 2024-02-19T19:03:31.563 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: root 0 2024-02-19T19:03:31.563 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: session_timeout 60 2024-02-19T19:03:31.563 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: session_autoclose 300 2024-02-19T19:03:31.563 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: max_file_size 1099511627776 2024-02-19T19:03:31.563 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: max_xattr_size 65536 2024-02-19T19:03:31.563 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: required_client_features {} 2024-02-19T19:03:31.563 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: last_failure 0 2024-02-19T19:03:31.563 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: last_failure_osd_epoch 47 2024-02-19T19:03:31.563 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:03:31.563 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: max_mds 2 2024-02-19T19:03:31.564 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: in 0,1 2024-02-19T19:03:31.564 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: up {0=14530,1=24269} 2024-02-19T19:03:31.564 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: failed 2024-02-19T19:03:31.564 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: damaged 2024-02-19T19:03:31.564 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: stopped 2024-02-19T19:03:31.564 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: data_pools [3] 2024-02-19T19:03:31.564 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: metadata_pool 2 2024-02-19T19:03:31.564 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: inline_data enabled 2024-02-19T19:03:31.564 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: balancer 2024-02-19T19:03:31.564 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: bal_rank_mask -1 2024-02-19T19:03:31.564 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: standby_count_wanted 1 2024-02-19T19:03:31.564 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: [mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:31.564 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: [mds.cephfs.smithi114.vpoeop{0:14540} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:31.565 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: [mds.cephfs.smithi174.znhuvd{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:31.565 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: [mds.cephfs.smithi174.stnfnf{1:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:31.565 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-02-19T19:03:31.565 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout: 2024-02-19T19:03:31.565 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: mon.smithi114@-1(???).osd e49 crush map has features 3314933000852226048, adjusting msgr requires 2024-02-19T19:03:31.565 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: mon.smithi114@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-02-19T19:03:31.565 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: mon.smithi114@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-02-19T19:03:31.565 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: mon.smithi114@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-02-19T19:03:31.565 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 ceph-mon[92983]: mon.smithi114@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-02-19T19:03:31.565 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 bash[92964]: e51f650f9be9c107f0a3e8b1342c8856ab70c7dc1a090e8598d81e6574275a33 2024-02-19T19:03:31.565 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:31 smithi114 systemd[1]: Started Ceph mon.smithi114 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T19:03:33.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:32 smithi114 ceph-mon[92983]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T19:03:33.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:32 smithi114 ceph-mon[92983]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T19:03:33.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:32 smithi114 ceph-mon[92983]: pgmap v21: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:03:33.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:32 smithi114 ceph-mon[92983]: mon.smithi114 calling monitor election 2024-02-19T19:03:33.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:32 smithi114 ceph-mon[92983]: mon.smithi114 is new leader, mons smithi114,smithi174 in quorum (ranks 0,1) 2024-02-19T19:03:33.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:32 smithi114 ceph-mon[92983]: monmap e2: 2 mons at {smithi114=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],smithi174=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-19T19:03:33.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:32 smithi114 ceph-mon[92983]: fsmap cephfs:2 {0=cephfs.smithi114.tcwpod=up:active,1=cephfs.smithi174.znhuvd=up:active} 2 up:standby-replay 2024-02-19T19:03:33.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:32 smithi114 ceph-mon[92983]: osdmap e49: 6 total, 6 up, 6 in 2024-02-19T19:03:33.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:32 smithi114 ceph-mon[92983]: mgrmap e30: smithi114.pcvtrk(active, since 37s), standbys: smithi174.lydvnj 2024-02-19T19:03:33.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:32 smithi114 ceph-mon[92983]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:03:33.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:32 smithi114 ceph-mon[92983]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:03:33.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:32 smithi114 ceph-mon[92983]: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:03:33.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:32 smithi114 ceph-mon[92983]: from='mgr.14744 ' entity='' 2024-02-19T19:03:33.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:32 smithi114 ceph-mon[92983]: mgrmap e31: smithi114.pcvtrk(active, since 37s), standbys: smithi174.lydvnj 2024-02-19T19:03:33.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:32 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T19:03:33.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:32 smithi174 ceph-mon[41061]: from='mgr.14744 172.21.15.114:0/1742488423' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T19:03:33.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:32 smithi174 ceph-mon[41061]: pgmap v21: 65 pgs: 65 active+clean; 592 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:03:33.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:32 smithi174 ceph-mon[41061]: mon.smithi114 calling monitor election 2024-02-19T19:03:33.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:32 smithi174 ceph-mon[41061]: mon.smithi114 is new leader, mons smithi114,smithi174 in quorum (ranks 0,1) 2024-02-19T19:03:33.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:32 smithi174 ceph-mon[41061]: monmap e2: 2 mons at {smithi114=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],smithi174=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-19T19:03:33.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:32 smithi174 ceph-mon[41061]: fsmap cephfs:2 {0=cephfs.smithi114.tcwpod=up:active,1=cephfs.smithi174.znhuvd=up:active} 2 up:standby-replay 2024-02-19T19:03:33.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:32 smithi174 ceph-mon[41061]: osdmap e49: 6 total, 6 up, 6 in 2024-02-19T19:03:33.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:32 smithi174 ceph-mon[41061]: mgrmap e30: smithi114.pcvtrk(active, since 37s), standbys: smithi174.lydvnj 2024-02-19T19:03:33.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:32 smithi174 ceph-mon[41061]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:03:33.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:32 smithi174 ceph-mon[41061]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:03:33.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:32 smithi174 ceph-mon[41061]: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:03:33.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:32 smithi174 ceph-mon[41061]: from='mgr.14744 ' entity='' 2024-02-19T19:03:33.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:32 smithi174 ceph-mon[41061]: mgrmap e31: smithi114.pcvtrk(active, since 37s), standbys: smithi174.lydvnj 2024-02-19T19:03:39.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.777+0000 7f6fe9bf8700 1 -- 172.21.15.114:0/3987427911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fe41010d0 msgr2=0x7f6fe41014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:39.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.777+0000 7f6fe9bf8700 1 --2- 172.21.15.114:0/3987427911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fe41010d0 0x7f6fe41014f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4009910 tx=0x7f6fd4009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:39.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.777+0000 7f6fe9bf8700 1 -- 172.21.15.114:0/3987427911 shutdown_connections 2024-02-19T19:03:39.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.777+0000 7f6fe9bf8700 1 --2- 172.21.15.114:0/3987427911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe40ffd30 0x7f6fe41001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:39.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.777+0000 7f6fe9bf8700 1 --2- 172.21.15.114:0/3987427911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fe41010d0 0x7f6fe41014f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:39.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.777+0000 7f6fe9bf8700 1 -- 172.21.15.114:0/3987427911 >> 172.21.15.114:0/3987427911 conn(0x7f6fe40fa7c0 msgr2=0x7f6fe40fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:39.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.777+0000 7f6fe9bf8700 1 -- 172.21.15.114:0/3987427911 shutdown_connections 2024-02-19T19:03:39.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.777+0000 7f6fe9bf8700 1 -- 172.21.15.114:0/3987427911 wait complete. 2024-02-19T19:03:39.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.778+0000 7f6fe9bf8700 1 Processor -- start 2024-02-19T19:03:39.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.778+0000 7f6fe9bf8700 1 -- start start 2024-02-19T19:03:39.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.778+0000 7f6fe9bf8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fe40ffd30 0x7f6fe4195280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:39.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.778+0000 7f6fe9bf8700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe41010d0 0x7f6fe41957c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:39.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.778+0000 7f6fe9bf8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4195de0 con 0x7f6fe40ffd30 2024-02-19T19:03:39.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.778+0000 7f6fe9bf8700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4195f20 con 0x7f6fe41010d0 2024-02-19T19:03:39.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.778+0000 7f6fe37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fe40ffd30 0x7f6fe4195280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:39.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.778+0000 7f6fe37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fe40ffd30 0x7f6fe4195280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50304/0 (socket says 172.21.15.114:50304) 2024-02-19T19:03:39.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.779+0000 7f6fe37fe700 1 -- 172.21.15.114:0/3955984351 learned_addr learned my addr 172.21.15.114:0/3955984351 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:03:39.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.779+0000 7f6fe2ffd700 1 --2- 172.21.15.114:0/3955984351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe41010d0 0x7f6fe41957c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:39.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.779+0000 7f6fe2ffd700 1 -- 172.21.15.114:0/3955984351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fe40ffd30 msgr2=0x7f6fe4195280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:39.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.779+0000 7f6fe2ffd700 1 --2- 172.21.15.114:0/3955984351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fe40ffd30 0x7f6fe4195280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:39.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.779+0000 7f6fe2ffd700 1 -- 172.21.15.114:0/3955984351 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd40095f0 con 0x7f6fe41010d0 2024-02-19T19:03:39.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.779+0000 7f6fe37fe700 1 --2- 172.21.15.114:0/3955984351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fe40ffd30 0x7f6fe4195280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:03:39.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.779+0000 7f6fe2ffd700 1 --2- 172.21.15.114:0/3955984351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe41010d0 0x7f6fe41957c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8009ad0 tx=0x7f6fd8009de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:39.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.779+0000 7f6fe0ff9700 1 -- 172.21.15.114:0/3955984351 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fd8014070 con 0x7f6fe41010d0 2024-02-19T19:03:39.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.779+0000 7f6fe9bf8700 1 -- 172.21.15.114:0/3955984351 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe40fec80 con 0x7f6fe41010d0 2024-02-19T19:03:39.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.779+0000 7f6fe9bf8700 1 -- 172.21.15.114:0/3955984351 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe40ff1d0 con 0x7f6fe41010d0 2024-02-19T19:03:39.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.779+0000 7f6fe0ff9700 1 -- 172.21.15.114:0/3955984351 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6fd8007cf0 con 0x7f6fe41010d0 2024-02-19T19:03:39.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.779+0000 7f6fe0ff9700 1 -- 172.21.15.114:0/3955984351 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fd801aea0 con 0x7f6fe41010d0 2024-02-19T19:03:39.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.781+0000 7f6fe9bf8700 1 -- 172.21.15.114:0/3955984351 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd0005320 con 0x7f6fe41010d0 2024-02-19T19:03:39.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.781+0000 7f6fe0ff9700 1 -- 172.21.15.114:0/3955984351 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 33) v1 ==== 91110+0+0 (secure 0 0 0) 0x7f6fd8018070 con 0x7f6fe41010d0 2024-02-19T19:03:39.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.781+0000 7f6fe0ff9700 1 -- 172.21.15.114:0/3955984351 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6fd8090160 con 0x7f6fe41010d0 2024-02-19T19:03:39.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:39.786+0000 7f6fe0ff9700 1 -- 172.21.15.114:0/3955984351 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6fd805adf0 con 0x7f6fe41010d0 2024-02-19T19:03:39.893 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:39 smithi174 ceph-mon[41061]: Standby manager daemon smithi174.lydvnj restarted 2024-02-19T19:03:39.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:39 smithi174 ceph-mon[41061]: Standby manager daemon smithi174.lydvnj started 2024-02-19T19:03:39.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:39 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/178535369' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/crt"}]: dispatch 2024-02-19T19:03:39.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:39 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/178535369' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:03:39.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:39 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/178535369' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/key"}]: dispatch 2024-02-19T19:03:39.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:39 smithi174 ceph-mon[41061]: from='mgr.? 172.21.15.174:0/178535369' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:03:39.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:39 smithi114 ceph-mon[92983]: Standby manager daemon smithi174.lydvnj restarted 2024-02-19T19:03:39.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:39 smithi114 ceph-mon[92983]: Standby manager daemon smithi174.lydvnj started 2024-02-19T19:03:39.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:39 smithi114 ceph-mon[92983]: from='mgr.? 172.21.15.174:0/178535369' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/crt"}]: dispatch 2024-02-19T19:03:39.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:39 smithi114 ceph-mon[92983]: from='mgr.? 172.21.15.174:0/178535369' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:03:39.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:39 smithi114 ceph-mon[92983]: from='mgr.? 172.21.15.174:0/178535369' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi174.lydvnj/key"}]: dispatch 2024-02-19T19:03:39.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:39 smithi114 ceph-mon[92983]: from='mgr.? 172.21.15.174:0/178535369' entity='mgr.smithi174.lydvnj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:03:40.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.721+0000 7f6fe0ff9700 1 -- 172.21.15.114:0/3955984351 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mgrmap(e 34) v1 ==== 91237+0+0 (secure 0 0 0) 0x7f6fd805a8a0 con 0x7f6fe41010d0 2024-02-19T19:03:40.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.722+0000 7f6fe0ff9700 1 --2- 172.21.15.114:0/3955984351 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6fcc072a60 0x7f6fcc074e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:40.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.722+0000 7f6fe0ff9700 1 -- 172.21.15.114:0/3955984351 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fcc00fd90 con 0x7f6fcc072a60 2024-02-19T19:03:40.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.722+0000 7f6fe37fe700 1 --2- 172.21.15.114:0/3955984351 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6fcc072a60 0x7f6fcc074e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:40.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.723+0000 7f6fe37fe700 1 --2- 172.21.15.114:0/3955984351 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6fcc072a60 0x7f6fcc074e50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400b5c0 tx=0x7f6fd401c040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:40.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.725+0000 7f6fe0ff9700 1 -- 172.21.15.114:0/3955984351 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f6fd400b7d0 con 0x7f6fcc072a60 2024-02-19T19:03:40.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.727+0000 7f6fe9bf8700 1 -- 172.21.15.114:0/3955984351 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6fcc072a60 msgr2=0x7f6fcc074e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:40.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.727+0000 7f6fe9bf8700 1 --2- 172.21.15.114:0/3955984351 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6fcc072a60 0x7f6fcc074e50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400b5c0 tx=0x7f6fd401c040 comp rx=0 tx=0).stop 2024-02-19T19:03:40.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.727+0000 7f6fe9bf8700 1 -- 172.21.15.114:0/3955984351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe41010d0 msgr2=0x7f6fe41957c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:40.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.727+0000 7f6fe9bf8700 1 --2- 172.21.15.114:0/3955984351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe41010d0 0x7f6fe41957c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8009ad0 tx=0x7f6fd8009de0 comp rx=0 tx=0).stop 2024-02-19T19:03:40.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.727+0000 7f6fe9bf8700 1 -- 172.21.15.114:0/3955984351 shutdown_connections 2024-02-19T19:03:40.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.727+0000 7f6fe9bf8700 1 --2- 172.21.15.114:0/3955984351 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6fcc072a60 0x7f6fcc074e50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:40.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.727+0000 7f6fe9bf8700 1 --2- 172.21.15.114:0/3955984351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fe40ffd30 0x7f6fe4195280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:40.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.727+0000 7f6fe9bf8700 1 --2- 172.21.15.114:0/3955984351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe41010d0 0x7f6fe41957c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:40.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.728+0000 7f6fe9bf8700 1 -- 172.21.15.114:0/3955984351 >> 172.21.15.114:0/3955984351 conn(0x7f6fe40fa7c0 msgr2=0x7f6fe4103f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:40.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.728+0000 7f6fe9bf8700 1 -- 172.21.15.114:0/3955984351 shutdown_connections 2024-02-19T19:03:40.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.728+0000 7f6fe9bf8700 1 -- 172.21.15.114:0/3955984351 wait complete. 2024-02-19T19:03:40.757 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-02-19T19:03:40.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.834+0000 7fc71f06d700 1 -- 172.21.15.114:0/3340999322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7180ffac0 msgr2=0x7fc7180ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:40.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.834+0000 7fc71f06d700 1 --2- 172.21.15.114:0/3340999322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7180ffac0 0x7fc7180ffee0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc70c009910 tx=0x7fc70c009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:40.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.834+0000 7fc71f06d700 1 -- 172.21.15.114:0/3340999322 shutdown_connections 2024-02-19T19:03:40.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.834+0000 7fc71f06d700 1 --2- 172.21.15.114:0/3340999322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc718100cc0 0x7fc718101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:40.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.834+0000 7fc71f06d700 1 --2- 172.21.15.114:0/3340999322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7180ffac0 0x7fc7180ffee0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:40.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.834+0000 7fc71f06d700 1 -- 172.21.15.114:0/3340999322 >> 172.21.15.114:0/3340999322 conn(0x7fc7180fb060 msgr2=0x7fc7180fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:40.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.834+0000 7fc71f06d700 1 -- 172.21.15.114:0/3340999322 shutdown_connections 2024-02-19T19:03:40.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.834+0000 7fc71f06d700 1 -- 172.21.15.114:0/3340999322 wait complete. 2024-02-19T19:03:40.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.835+0000 7fc71f06d700 1 Processor -- start 2024-02-19T19:03:40.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.835+0000 7fc71f06d700 1 -- start start 2024-02-19T19:03:40.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.835+0000 7fc71f06d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7180ffac0 0x7fc7181963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:40.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.835+0000 7fc71f06d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc718100cc0 0x7fc7181968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:40.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.835+0000 7fc71f06d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc718196f00 con 0x7fc718100cc0 2024-02-19T19:03:40.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.835+0000 7fc71f06d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc718197040 con 0x7fc7180ffac0 2024-02-19T19:03:40.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.835+0000 7fc717fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc718100cc0 0x7fc7181968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:40.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.835+0000 7fc717fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc718100cc0 0x7fc7181968e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50382/0 (socket says 172.21.15.114:50382) 2024-02-19T19:03:40.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.835+0000 7fc717fff700 1 -- 172.21.15.114:0/1282357715 learned_addr learned my addr 172.21.15.114:0/1282357715 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:03:40.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.835+0000 7fc71ce09700 1 --2- 172.21.15.114:0/1282357715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7180ffac0 0x7fc7181963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:40.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.835+0000 7fc717fff700 1 -- 172.21.15.114:0/1282357715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7180ffac0 msgr2=0x7fc7181963a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:40.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.835+0000 7fc717fff700 1 --2- 172.21.15.114:0/1282357715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7180ffac0 0x7fc7181963a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:40.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.835+0000 7fc717fff700 1 -- 172.21.15.114:0/1282357715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c0095f0 con 0x7fc718100cc0 2024-02-19T19:03:40.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.835+0000 7fc71ce09700 1 --2- 172.21.15.114:0/1282357715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7180ffac0 0x7fc7181963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:03:40.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.835+0000 7fc717fff700 1 --2- 172.21.15.114:0/1282357715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc718100cc0 0x7fc7181968e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc70400e920 tx=0x7fc70400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:40.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.836+0000 7fc715ffb700 1 -- 172.21.15.114:0/1282357715 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc70400c9d0 con 0x7fc718100cc0 2024-02-19T19:03:40.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.836+0000 7fc71f06d700 1 -- 172.21.15.114:0/1282357715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71819baf0 con 0x7fc718100cc0 2024-02-19T19:03:40.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.836+0000 7fc715ffb700 1 -- 172.21.15.114:0/1282357715 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc704010430 con 0x7fc718100cc0 2024-02-19T19:03:40.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.836+0000 7fc715ffb700 1 -- 172.21.15.114:0/1282357715 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc704018650 con 0x7fc718100cc0 2024-02-19T19:03:40.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.836+0000 7fc71f06d700 1 -- 172.21.15.114:0/1282357715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71819c040 con 0x7fc718100cc0 2024-02-19T19:03:40.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.837+0000 7fc71f06d700 1 -- 172.21.15.114:0/1282357715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc718108620 con 0x7fc718100cc0 2024-02-19T19:03:40.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.837+0000 7fc715ffb700 1 -- 172.21.15.114:0/1282357715 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 34) v1 ==== 91245+0+0 (secure 0 0 0) 0x7fc7040105a0 con 0x7fc718100cc0 2024-02-19T19:03:40.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.838+0000 7fc715ffb700 1 --2- 172.21.15.114:0/1282357715 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc70806d5c0 0x7fc70806fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:40.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.838+0000 7fc715ffb700 1 -- 172.21.15.114:0/1282357715 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc704014070 con 0x7fc718100cc0 2024-02-19T19:03:40.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.838+0000 7fc71ce09700 1 --2- 172.21.15.114:0/1282357715 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc70806d5c0 0x7fc70806fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:40.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.838+0000 7fc71ce09700 1 --2- 172.21.15.114:0/1282357715 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc70806d5c0 0x7fc70806fa80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00b6f0 tx=0x7fc70c005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:40.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:40.843+0000 7fc715ffb700 1 -- 172.21.15.114:0/1282357715 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc704052d20 con 0x7fc718100cc0 2024-02-19T19:03:41.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: mgrmap e32: smithi114.pcvtrk(active, since 44s), standbys: smithi174.lydvnj 2024-02-19T19:03:41.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: Active manager daemon smithi114.pcvtrk restarted 2024-02-19T19:03:41.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: Activating manager daemon smithi114.pcvtrk 2024-02-19T19:03:41.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: osdmap e50: 6 total, 6 up, 6 in 2024-02-19T19:03:41.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: mgrmap e33: smithi114.pcvtrk(active, starting, since 0.00723334s), standbys: smithi174.lydvnj 2024-02-19T19:03:41.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T19:03:41.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T19:03:41.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.tcwpod"}]: dispatch 2024-02-19T19:03:41.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.vpoeop"}]: dispatch 2024-02-19T19:03:41.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.znhuvd"}]: dispatch 2024-02-19T19:03:41.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.stnfnf"}]: dispatch 2024-02-19T19:03:41.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr metadata", "who": "smithi114.pcvtrk", "id": "smithi114.pcvtrk"}]: dispatch 2024-02-19T19:03:41.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr metadata", "who": "smithi174.lydvnj", "id": "smithi174.lydvnj"}]: dispatch 2024-02-19T19:03:41.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:03:41.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:03:41.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:03:41.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:03:41.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:03:41.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:03:41.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T19:03:41.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T19:03:41.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T19:03:41.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: Manager daemon smithi114.pcvtrk is now available 2024-02-19T19:03:41.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:41.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:41.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:41.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:03:41.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/trash_purge_schedule"}]: dispatch 2024-02-19T19:03:41.066 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: mgrmap e32: smithi114.pcvtrk(active, since 44s), standbys: smithi174.lydvnj 2024-02-19T19:03:41.066 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: Active manager daemon smithi114.pcvtrk restarted 2024-02-19T19:03:41.066 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: Activating manager daemon smithi114.pcvtrk 2024-02-19T19:03:41.066 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: osdmap e50: 6 total, 6 up, 6 in 2024-02-19T19:03:41.066 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: mgrmap e33: smithi114.pcvtrk(active, starting, since 0.00723334s), standbys: smithi174.lydvnj 2024-02-19T19:03:41.066 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T19:03:41.066 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T19:03:41.066 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.tcwpod"}]: dispatch 2024-02-19T19:03:41.066 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.vpoeop"}]: dispatch 2024-02-19T19:03:41.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.znhuvd"}]: dispatch 2024-02-19T19:03:41.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.stnfnf"}]: dispatch 2024-02-19T19:03:41.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr metadata", "who": "smithi114.pcvtrk", "id": "smithi114.pcvtrk"}]: dispatch 2024-02-19T19:03:41.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr metadata", "who": "smithi174.lydvnj", "id": "smithi174.lydvnj"}]: dispatch 2024-02-19T19:03:41.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:03:41.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:03:41.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:03:41.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:03:41.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:03:41.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:03:41.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T19:03:41.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T19:03:41.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T19:03:41.067 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: Manager daemon smithi114.pcvtrk is now available 2024-02-19T19:03:41.068 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:41.068 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:41.068 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:41.068 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:03:41.068 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:40 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi114.pcvtrk/trash_purge_schedule"}]: dispatch 2024-02-19T19:03:41.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.069+0000 7fc71f06d700 1 -- 172.21.15.114:0/1282357715 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7180694c0 con 0x7fc70806d5c0 2024-02-19T19:03:41.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.070+0000 7fc715ffb700 1 -- 172.21.15.114:0/1282357715 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7fc7180694c0 con 0x7fc70806d5c0 2024-02-19T19:03:41.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.072+0000 7fc71f06d700 1 -- 172.21.15.114:0/1282357715 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc70806d5c0 msgr2=0x7fc70806fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:41.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.073+0000 7fc71f06d700 1 --2- 172.21.15.114:0/1282357715 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc70806d5c0 0x7fc70806fa80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00b6f0 tx=0x7fc70c005940 comp rx=0 tx=0).stop 2024-02-19T19:03:41.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.073+0000 7fc71f06d700 1 -- 172.21.15.114:0/1282357715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc718100cc0 msgr2=0x7fc7181968e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:41.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.073+0000 7fc71f06d700 1 --2- 172.21.15.114:0/1282357715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc718100cc0 0x7fc7181968e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc70400e920 tx=0x7fc70400ec30 comp rx=0 tx=0).stop 2024-02-19T19:03:41.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.073+0000 7fc71f06d700 1 -- 172.21.15.114:0/1282357715 shutdown_connections 2024-02-19T19:03:41.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.073+0000 7fc71f06d700 1 --2- 172.21.15.114:0/1282357715 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc70806d5c0 0x7fc70806fa80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.073+0000 7fc71f06d700 1 --2- 172.21.15.114:0/1282357715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7180ffac0 0x7fc7181963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.073+0000 7fc71f06d700 1 --2- 172.21.15.114:0/1282357715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc718100cc0 0x7fc7181968e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.073+0000 7fc71f06d700 1 -- 172.21.15.114:0/1282357715 >> 172.21.15.114:0/1282357715 conn(0x7fc7180fb060 msgr2=0x7fc718068db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:41.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.073+0000 7fc71f06d700 1 -- 172.21.15.114:0/1282357715 shutdown_connections 2024-02-19T19:03:41.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.073+0000 7fc71f06d700 1 -- 172.21.15.114:0/1282357715 wait complete. 2024-02-19T19:03:41.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.179+0000 7fc206270700 1 -- 172.21.15.114:0/4247844785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc200100be0 msgr2=0x7fc200101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:41.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.179+0000 7fc206270700 1 --2- 172.21.15.114:0/4247844785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc200100be0 0x7fc200101000 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0009910 tx=0x7fc1f0009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:41.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.179+0000 7fc206270700 1 -- 172.21.15.114:0/4247844785 shutdown_connections 2024-02-19T19:03:41.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.179+0000 7fc206270700 1 --2- 172.21.15.114:0/4247844785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc200101de0 0x7fc200102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.179+0000 7fc206270700 1 --2- 172.21.15.114:0/4247844785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc200100be0 0x7fc200101000 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.179+0000 7fc206270700 1 -- 172.21.15.114:0/4247844785 >> 172.21.15.114:0/4247844785 conn(0x7fc2000fc140 msgr2=0x7fc2000fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:41.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.179+0000 7fc206270700 1 -- 172.21.15.114:0/4247844785 shutdown_connections 2024-02-19T19:03:41.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.179+0000 7fc206270700 1 -- 172.21.15.114:0/4247844785 wait complete. 2024-02-19T19:03:41.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.179+0000 7fc206270700 1 Processor -- start 2024-02-19T19:03:41.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.179+0000 7fc206270700 1 -- start start 2024-02-19T19:03:41.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.180+0000 7fc206270700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc200100be0 0x7fc200196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:41.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.180+0000 7fc206270700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc200101de0 0x7fc2001968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:41.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.180+0000 7fc206270700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc200196ef0 con 0x7fc200101de0 2024-02-19T19:03:41.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.180+0000 7fc206270700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc200197030 con 0x7fc200100be0 2024-02-19T19:03:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.180+0000 7fc1ff7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc200101de0 0x7fc2001968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.180+0000 7fc1ff7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc200101de0 0x7fc2001968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50406/0 (socket says 172.21.15.114:50406) 2024-02-19T19:03:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.180+0000 7fc1ff7fe700 1 -- 172.21.15.114:0/1995164724 learned_addr learned my addr 172.21.15.114:0/1995164724 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:03:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.180+0000 7fc1fffff700 1 --2- 172.21.15.114:0/1995164724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc200100be0 0x7fc200196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.180+0000 7fc1ff7fe700 1 -- 172.21.15.114:0/1995164724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc200100be0 msgr2=0x7fc200196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.180+0000 7fc1ff7fe700 1 --2- 172.21.15.114:0/1995164724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc200100be0 0x7fc200196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.180+0000 7fc1ff7fe700 1 -- 172.21.15.114:0/1995164724 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f00095f0 con 0x7fc200101de0 2024-02-19T19:03:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.180+0000 7fc1ff7fe700 1 --2- 172.21.15.114:0/1995164724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc200101de0 0x7fc2001968d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc1f400d8d0 tx=0x7fc1f400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.180+0000 7fc1fd7fa700 1 -- 172.21.15.114:0/1995164724 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1f40098a0 con 0x7fc200101de0 2024-02-19T19:03:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.180+0000 7fc1fd7fa700 1 -- 172.21.15.114:0/1995164724 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc1f4010460 con 0x7fc200101de0 2024-02-19T19:03:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.180+0000 7fc1fd7fa700 1 -- 172.21.15.114:0/1995164724 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1f400f5f0 con 0x7fc200101de0 2024-02-19T19:03:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.180+0000 7fc206270700 1 -- 172.21.15.114:0/1995164724 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc20019bae0 con 0x7fc200101de0 2024-02-19T19:03:41.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.181+0000 7fc206270700 1 -- 172.21.15.114:0/1995164724 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20019c030 con 0x7fc200101de0 2024-02-19T19:03:41.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.182+0000 7fc206270700 1 -- 172.21.15.114:0/1995164724 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc200109740 con 0x7fc200101de0 2024-02-19T19:03:41.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.182+0000 7fc1fd7fa700 1 -- 172.21.15.114:0/1995164724 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 34) v1 ==== 91245+0+0 (secure 0 0 0) 0x7fc1f4009a00 con 0x7fc200101de0 2024-02-19T19:03:41.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.182+0000 7fc1fd7fa700 1 --2- 172.21.15.114:0/1995164724 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc1e806d5c0 0x7fc1e806fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:41.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.182+0000 7fc1fd7fa700 1 -- 172.21.15.114:0/1995164724 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc1f408cf20 con 0x7fc200101de0 2024-02-19T19:03:41.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.182+0000 7fc1fffff700 1 --2- 172.21.15.114:0/1995164724 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc1e806d5c0 0x7fc1e806fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:41.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.183+0000 7fc1fffff700 1 --2- 172.21.15.114:0/1995164724 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc1e806d5c0 0x7fc1e806fa80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0006010 tx=0x7fc1f000b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:41.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.187+0000 7fc1fd7fa700 1 -- 172.21.15.114:0/1995164724 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc1f4056980 con 0x7fc200101de0 2024-02-19T19:03:41.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.398+0000 7fc206270700 1 -- 172.21.15.114:0/1995164724 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc200106730 con 0x7fc1e806d5c0 2024-02-19T19:03:41.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.410+0000 7fc1fd7fa700 1 -- 172.21.15.114:0/1995164724 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fc200106730 con 0x7fc1e806d5c0 2024-02-19T19:03:41.429 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:03:41.429 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (7m) 17s ago 10m 26.4M - 0.25.0 c8568f914cd2 30444b676ca4 2024-02-19T19:03:41.429 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (10m) 17s ago 10m 7818k - 18.2.1 94eadd990e0a 4918cd87e312 2024-02-19T19:03:41.430 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi174 smithi174 running (5m) 20s ago 8m 7721k - 18.2.1 94eadd990e0a 516c34431b93 2024-02-19T19:03:41.430 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (10m) 17s ago 10m 6660k - 18.2.1 94eadd990e0a ecfcd98907ba 2024-02-19T19:03:41.430 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi174 smithi174 running (8m) 20s ago 8m 6660k - 18.2.1 94eadd990e0a de88985f5a07 2024-02-19T19:03:41.430 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (7m) 17s ago 9m 85.7M - 9.4.7 954c08fa6188 478bd533e827 2024-02-19T19:03:41.430 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tcwpod smithi114 running (3m) 17s ago 3m 26.8M - 18.2.1 94eadd990e0a 420175777cf4 2024-02-19T19:03:41.430 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.vpoeop smithi114 running (3m) 17s ago 3m 21.0M - 18.2.1 94eadd990e0a f81a79e53ead 2024-02-19T19:03:41.430 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.stnfnf smithi174 running (3m) 20s ago 3m 22.1M - 18.2.1 94eadd990e0a 3df35954a492 2024-02-19T19:03:41.430 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.znhuvd smithi174 running (3m) 20s ago 3m 21.6M - 18.2.1 94eadd990e0a 2d2267494ae1 2024-02-19T19:03:41.430 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.pcvtrk smithi114 *:8443,9283,8765 running (57s) 17s ago 11m 491M - 19.0.0-1450-gf78a58c0 f081f847070a 05b5c06122b1 2024-02-19T19:03:41.430 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi174.lydvnj smithi174 *:8443,9283,8765 running (23s) 20s ago 8m 68.1M - 19.0.0-1450-gf78a58c0 f081f847070a f962b1af5aed 2024-02-19T19:03:41.430 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (12m) 17s ago 12m 52.5M 2048M 18.2.1 94eadd990e0a 41468f811fc0 2024-02-19T19:03:41.430 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi174 smithi174 running (8m) 20s ago 8m 42.5M 2048M 18.2.1 94eadd990e0a 70dd7bf69fd9 2024-02-19T19:03:41.430 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (10m) 17s ago 10m 14.4M - 1.5.0 0da6a335fe13 b6b258074cce 2024-02-19T19:03:41.430 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi174 smithi174 *:9100 running (8m) 20s ago 8m 14.8M - 1.5.0 0da6a335fe13 d0b4fdf67b22 2024-02-19T19:03:41.430 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (7m) 17s ago 7m 48.7M 1216M 18.2.1 94eadd990e0a 990afc982b6c 2024-02-19T19:03:41.430 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (6m) 17s ago 6m 49.0M 1216M 18.2.1 94eadd990e0a 932e3ad37db5 2024-02-19T19:03:41.431 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (6m) 17s ago 6m 50.7M 1216M 18.2.1 94eadd990e0a 12953988af86 2024-02-19T19:03:41.431 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi174 running (5m) 20s ago 5m 48.1M 2240M 18.2.1 94eadd990e0a 979f45cabbce 2024-02-19T19:03:41.431 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi174 running (5m) 20s ago 5m 49.0M 2240M 18.2.1 94eadd990e0a 8b8c3745b8c5 2024-02-19T19:03:41.431 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi174 running (5m) 20s ago 5m 51.4M 2240M 18.2.1 94eadd990e0a 5771a995ed75 2024-02-19T19:03:41.431 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (29s) 17s ago 8m 41.1M - 2.43.0 a07b618ecd1d a36922c7c9a1 2024-02-19T19:03:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.412+0000 7fc206270700 1 -- 172.21.15.114:0/1995164724 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc1e806d5c0 msgr2=0x7fc1e806fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.412+0000 7fc206270700 1 --2- 172.21.15.114:0/1995164724 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc1e806d5c0 0x7fc1e806fa80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0006010 tx=0x7fc1f000b540 comp rx=0 tx=0).stop 2024-02-19T19:03:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.412+0000 7fc206270700 1 -- 172.21.15.114:0/1995164724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc200101de0 msgr2=0x7fc2001968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.412+0000 7fc206270700 1 --2- 172.21.15.114:0/1995164724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc200101de0 0x7fc2001968d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc1f400d8d0 tx=0x7fc1f400dbe0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.412+0000 7fc206270700 1 -- 172.21.15.114:0/1995164724 shutdown_connections 2024-02-19T19:03:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.412+0000 7fc206270700 1 --2- 172.21.15.114:0/1995164724 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc1e806d5c0 0x7fc1e806fa80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.412+0000 7fc206270700 1 --2- 172.21.15.114:0/1995164724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc200100be0 0x7fc200196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.412+0000 7fc206270700 1 --2- 172.21.15.114:0/1995164724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc200101de0 0x7fc2001968d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.412+0000 7fc206270700 1 -- 172.21.15.114:0/1995164724 >> 172.21.15.114:0/1995164724 conn(0x7fc2000fc140 msgr2=0x7fc200105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.412+0000 7fc206270700 1 -- 172.21.15.114:0/1995164724 shutdown_connections 2024-02-19T19:03:41.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.412+0000 7fc206270700 1 -- 172.21.15.114:0/1995164724 wait complete. 2024-02-19T19:03:41.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.527+0000 7ff19c4f0700 1 -- 172.21.15.114:0/1590169544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1940fec20 msgr2=0x7ff1940ff080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:41.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.527+0000 7ff19c4f0700 1 --2- 172.21.15.114:0/1590169544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1940fec20 0x7ff1940ff080 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff184009910 tx=0x7ff184009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.527+0000 7ff19c4f0700 1 -- 172.21.15.114:0/1590169544 shutdown_connections 2024-02-19T19:03:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.527+0000 7ff19c4f0700 1 --2- 172.21.15.114:0/1590169544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1940fec20 0x7ff1940ff080 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.527+0000 7ff19c4f0700 1 --2- 172.21.15.114:0/1590169544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1941021a0 0x7ff1941025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.527+0000 7ff19c4f0700 1 -- 172.21.15.114:0/1590169544 >> 172.21.15.114:0/1590169544 conn(0x7ff1940fa7c0 msgr2=0x7ff1940fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.527+0000 7ff19c4f0700 1 -- 172.21.15.114:0/1590169544 shutdown_connections 2024-02-19T19:03:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.527+0000 7ff19c4f0700 1 -- 172.21.15.114:0/1590169544 wait complete. 2024-02-19T19:03:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.528+0000 7ff19c4f0700 1 Processor -- start 2024-02-19T19:03:41.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.528+0000 7ff19c4f0700 1 -- start start 2024-02-19T19:03:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.528+0000 7ff19c4f0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1940fec20 0x7ff194196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.528+0000 7ff19c4f0700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1941021a0 0x7ff194196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.528+0000 7ff19c4f0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff194197070 con 0x7ff1940fec20 2024-02-19T19:03:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.528+0000 7ff19c4f0700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1941971b0 con 0x7ff1941021a0 2024-02-19T19:03:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.528+0000 7ff19a28c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1940fec20 0x7ff194196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.528+0000 7ff19a28c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1940fec20 0x7ff194196510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50430/0 (socket says 172.21.15.114:50430) 2024-02-19T19:03:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.528+0000 7ff19a28c700 1 -- 172.21.15.114:0/177802722 learned_addr learned my addr 172.21.15.114:0/177802722 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:03:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.528+0000 7ff199a8b700 1 --2- 172.21.15.114:0/177802722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1941021a0 0x7ff194196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.529+0000 7ff19a28c700 1 -- 172.21.15.114:0/177802722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1941021a0 msgr2=0x7ff194196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.529+0000 7ff19a28c700 1 --2- 172.21.15.114:0/177802722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1941021a0 0x7ff194196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.529+0000 7ff19a28c700 1 -- 172.21.15.114:0/177802722 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1840095f0 con 0x7ff1940fec20 2024-02-19T19:03:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.529+0000 7ff19a28c700 1 --2- 172.21.15.114:0/177802722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1940fec20 0x7ff194196510 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff19000b480 tx=0x7ff19000b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:41.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.529+0000 7ff18b7fe700 1 -- 172.21.15.114:0/177802722 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff190011800 con 0x7ff1940fec20 2024-02-19T19:03:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.529+0000 7ff19c4f0700 1 -- 172.21.15.114:0/177802722 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19419bc60 con 0x7ff1940fec20 2024-02-19T19:03:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.529+0000 7ff18b7fe700 1 -- 172.21.15.114:0/177802722 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff190011e10 con 0x7ff1940fec20 2024-02-19T19:03:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.529+0000 7ff18b7fe700 1 -- 172.21.15.114:0/177802722 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff190010ea0 con 0x7ff1940fec20 2024-02-19T19:03:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.529+0000 7ff19c4f0700 1 -- 172.21.15.114:0/177802722 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19419c1b0 con 0x7ff1940fec20 2024-02-19T19:03:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.530+0000 7ff19c4f0700 1 -- 172.21.15.114:0/177802722 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1941097d0 con 0x7ff1940fec20 2024-02-19T19:03:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.530+0000 7ff18b7fe700 1 -- 172.21.15.114:0/177802722 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 34) v1 ==== 91245+0+0 (secure 0 0 0) 0x7ff190010380 con 0x7ff1940fec20 2024-02-19T19:03:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.531+0000 7ff18b7fe700 1 --2- 172.21.15.114:0/177802722 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff18006d5c0 0x7ff18006fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.531+0000 7ff18b7fe700 1 -- 172.21.15.114:0/177802722 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff19008cbb0 con 0x7ff1940fec20 2024-02-19T19:03:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.531+0000 7ff199a8b700 1 --2- 172.21.15.114:0/177802722 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff18006d5c0 0x7ff18006fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.531+0000 7ff199a8b700 1 --2- 172.21.15.114:0/177802722 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff18006d5c0 0x7ff18006fa80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff184006010 tx=0x7ff18400b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:41.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.536+0000 7ff18b7fe700 1 -- 172.21.15.114:0/177802722 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff190056610 con 0x7ff1940fec20 2024-02-19T19:03:41.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.833+0000 7ff19c4f0700 1 -- 172.21.15.114:0/177802722 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff194109990 con 0x7ff1940fec20 2024-02-19T19:03:41.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.834+0000 7ff18b7fe700 1 -- 172.21.15.114:0/177802722 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+779 (secure 0 0 0) 0x7ff190015320 con 0x7ff1940fec20 2024-02-19T19:03:41.854 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:03:41.854 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-02-19T19:03:41.854 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-02-19T19:03:41.854 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 1 2024-02-19T19:03:41.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:03:41.854 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-02-19T19:03:41.854 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:03:41.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:03:41.855 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-02-19T19:03:41.855 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-02-19T19:03:41.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:03:41.855 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-02-19T19:03:41.855 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:03:41.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:03:41.855 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-02-19T19:03:41.855 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 11, 2024-02-19T19:03:41.855 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 3 2024-02-19T19:03:41.855 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-02-19T19:03:41.855 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:03:41.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.836+0000 7ff19c4f0700 1 -- 172.21.15.114:0/177802722 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff18006d5c0 msgr2=0x7ff18006fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:41.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.837+0000 7ff19c4f0700 1 --2- 172.21.15.114:0/177802722 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff18006d5c0 0x7ff18006fa80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff184006010 tx=0x7ff18400b540 comp rx=0 tx=0).stop 2024-02-19T19:03:41.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.837+0000 7ff19c4f0700 1 -- 172.21.15.114:0/177802722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1940fec20 msgr2=0x7ff194196510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:41.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.837+0000 7ff19c4f0700 1 --2- 172.21.15.114:0/177802722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1940fec20 0x7ff194196510 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff19000b480 tx=0x7ff19000b790 comp rx=0 tx=0).stop 2024-02-19T19:03:41.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.837+0000 7ff19c4f0700 1 -- 172.21.15.114:0/177802722 shutdown_connections 2024-02-19T19:03:41.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.837+0000 7ff19c4f0700 1 --2- 172.21.15.114:0/177802722 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff18006d5c0 0x7ff18006fa80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.837+0000 7ff19c4f0700 1 --2- 172.21.15.114:0/177802722 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1940fec20 0x7ff194196510 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.837+0000 7ff19c4f0700 1 --2- 172.21.15.114:0/177802722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1941021a0 0x7ff194196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.837+0000 7ff19c4f0700 1 -- 172.21.15.114:0/177802722 >> 172.21.15.114:0/177802722 conn(0x7ff1940fa7c0 msgr2=0x7ff194105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:41.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.837+0000 7ff19c4f0700 1 -- 172.21.15.114:0/177802722 shutdown_connections 2024-02-19T19:03:41.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.837+0000 7ff19c4f0700 1 -- 172.21.15.114:0/177802722 wait complete. 2024-02-19T19:03:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.944+0000 7fc960a6c700 1 -- 172.21.15.114:0/595921782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc95c0feec0 msgr2=0x7fc95c0ff2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.944+0000 7fc960a6c700 1 --2- 172.21.15.114:0/595921782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc95c0feec0 0x7fc95c0ff2e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc94c009910 tx=0x7fc94c009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.944+0000 7fc960a6c700 1 -- 172.21.15.114:0/595921782 shutdown_connections 2024-02-19T19:03:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.944+0000 7fc960a6c700 1 --2- 172.21.15.114:0/595921782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc95c101f40 0x7fc95c1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.944+0000 7fc960a6c700 1 --2- 172.21.15.114:0/595921782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc95c0feec0 0x7fc95c0ff2e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.944+0000 7fc960a6c700 1 -- 172.21.15.114:0/595921782 >> 172.21.15.114:0/595921782 conn(0x7fc95c0fa800 msgr2=0x7fc95c0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.944+0000 7fc960a6c700 1 -- 172.21.15.114:0/595921782 shutdown_connections 2024-02-19T19:03:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.944+0000 7fc960a6c700 1 -- 172.21.15.114:0/595921782 wait complete. 2024-02-19T19:03:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.945+0000 7fc960a6c700 1 Processor -- start 2024-02-19T19:03:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.945+0000 7fc960a6c700 1 -- start start 2024-02-19T19:03:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.945+0000 7fc960a6c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc95c0feec0 0x7fc95c1018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.945+0000 7fc960a6c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc95c101f40 0x7fc95c0ffef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.945+0000 7fc960a6c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c100430 con 0x7fc95c101f40 2024-02-19T19:03:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.945+0000 7fc960a6c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c100570 con 0x7fc95c0feec0 2024-02-19T19:03:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.945+0000 7fc959d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc95c101f40 0x7fc95c0ffef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.945+0000 7fc95a59c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc95c0feec0 0x7fc95c1018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.945+0000 7fc959d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc95c101f40 0x7fc95c0ffef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50478/0 (socket says 172.21.15.114:50478) 2024-02-19T19:03:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.945+0000 7fc959d9b700 1 -- 172.21.15.114:0/3392890398 learned_addr learned my addr 172.21.15.114:0/3392890398 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:03:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.946+0000 7fc959d9b700 1 -- 172.21.15.114:0/3392890398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc95c0feec0 msgr2=0x7fc95c1018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.946+0000 7fc959d9b700 1 --2- 172.21.15.114:0/3392890398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc95c0feec0 0x7fc95c1018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.946+0000 7fc959d9b700 1 -- 172.21.15.114:0/3392890398 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94c0095f0 con 0x7fc95c101f40 2024-02-19T19:03:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.946+0000 7fc95a59c700 1 --2- 172.21.15.114:0/3392890398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc95c0feec0 0x7fc95c1018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:03:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.946+0000 7fc959d9b700 1 --2- 172.21.15.114:0/3392890398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc95c101f40 0x7fc95c0ffef0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc95000c930 tx=0x7fc95000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.946+0000 7fc94b7fe700 1 -- 172.21.15.114:0/3392890398 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc950007ad0 con 0x7fc95c101f40 2024-02-19T19:03:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.946+0000 7fc94b7fe700 1 -- 172.21.15.114:0/3392890398 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc95000ce80 con 0x7fc95c101f40 2024-02-19T19:03:41.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.946+0000 7fc960a6c700 1 -- 172.21.15.114:0/3392890398 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95c100850 con 0x7fc95c101f40 2024-02-19T19:03:41.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.946+0000 7fc94b7fe700 1 -- 172.21.15.114:0/3392890398 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc950018630 con 0x7fc95c101f40 2024-02-19T19:03:41.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.946+0000 7fc960a6c700 1 -- 172.21.15.114:0/3392890398 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95c19e3f0 con 0x7fc95c101f40 2024-02-19T19:03:41.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.947+0000 7fc960a6c700 1 -- 172.21.15.114:0/3392890398 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc95c109760 con 0x7fc95c101f40 2024-02-19T19:03:41.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.947+0000 7fc94b7fe700 1 -- 172.21.15.114:0/3392890398 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 34) v1 ==== 91245+0+0 (secure 0 0 0) 0x7fc950007c30 con 0x7fc95c101f40 2024-02-19T19:03:41.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.948+0000 7fc94b7fe700 1 --2- 172.21.15.114:0/3392890398 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc94406d5c0 0x7fc94406fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:41.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.948+0000 7fc94b7fe700 1 -- 172.21.15.114:0/3392890398 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fc95008d020 con 0x7fc95c101f40 2024-02-19T19:03:41.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.948+0000 7fc95a59c700 1 --2- 172.21.15.114:0/3392890398 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc94406d5c0 0x7fc94406fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:41.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.948+0000 7fc95a59c700 1 --2- 172.21.15.114:0/3392890398 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc94406d5c0 0x7fc94406fa80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc94c006010 tx=0x7fc94c00b560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:41.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:41.953+0000 7fc94b7fe700 1 -- 172.21.15.114:0/3392890398 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc950056a80 con 0x7fc95c101f40 2024-02-19T19:03:42.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:41 smithi114 ceph-mon[92983]: [19/Feb/2024:19:03:40] ENGINE Bus STARTING 2024-02-19T19:03:42.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:41 smithi114 ceph-mon[92983]: mgrmap e34: smithi114.pcvtrk(active, since 1.00984s), standbys: smithi174.lydvnj 2024-02-19T19:03:42.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:41 smithi114 ceph-mon[92983]: from='client.24507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:42.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:41 smithi114 ceph-mon[92983]: pgmap v3: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:03:42.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:41 smithi174 ceph-mon[41061]: [19/Feb/2024:19:03:40] ENGINE Bus STARTING 2024-02-19T19:03:42.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:41 smithi174 ceph-mon[41061]: mgrmap e34: smithi114.pcvtrk(active, since 1.00984s), standbys: smithi174.lydvnj 2024-02-19T19:03:42.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:41 smithi174 ceph-mon[41061]: from='client.24507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:42.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:41 smithi174 ceph-mon[41061]: pgmap v3: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:03:42.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.197+0000 7fc960a6c700 1 -- 172.21.15.114:0/3392890398 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc95c19e9a0 con 0x7fc95c101f40 2024-02-19T19:03:42.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.197+0000 7fc94b7fe700 1 -- 172.21.15.114:0/3392890398 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fc95c19e9a0 con 0x7fc95c101f40 2024-02-19T19:03:42.217 INFO:teuthology.orchestra.run.smithi114.stdout:e18 2024-02-19T19:03:42.217 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:03:42.217 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:03:42.217 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-02-19T19:03:42.218 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:03:42.218 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:03:42.218 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-02-19T19:03:42.218 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 18 2024-02-19T19:03:42.218 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:03:42.218 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:03:42.218 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-02-19T19:00:21.203922+0000 2024-02-19T19:03:42.218 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-02-19T19:03:42.218 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-02-19T19:03:42.218 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-02-19T19:03:42.218 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-02-19T19:03:42.218 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-02-19T19:03:42.218 INFO:teuthology.orchestra.run.smithi114.stdout:max_xattr_size 65536 2024-02-19T19:03:42.218 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-02-19T19:03:42.219 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-02-19T19:03:42.219 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 47 2024-02-19T19:03:42.219 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:03:42.219 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 2 2024-02-19T19:03:42.219 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1 2024-02-19T19:03:42.219 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=14530,1=24269} 2024-02-19T19:03:42.219 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-02-19T19:03:42.219 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-02-19T19:03:42.219 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-02-19T19:03:42.219 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-02-19T19:03:42.219 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-02-19T19:03:42.219 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data enabled 2024-02-19T19:03:42.219 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-02-19T19:03:42.219 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-02-19T19:03:42.219 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 1 2024-02-19T19:03:42.220 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:42.220 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.vpoeop{0:14540} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:42.220 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.znhuvd{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:42.220 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.stnfnf{1:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:42.220 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:03:42.220 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:03:42.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.200+0000 7fc960a6c700 1 -- 172.21.15.114:0/3392890398 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc94406d5c0 msgr2=0x7fc94406fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:42.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.200+0000 7fc960a6c700 1 --2- 172.21.15.114:0/3392890398 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc94406d5c0 0x7fc94406fa80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc94c006010 tx=0x7fc94c00b560 comp rx=0 tx=0).stop 2024-02-19T19:03:42.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.200+0000 7fc960a6c700 1 -- 172.21.15.114:0/3392890398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc95c101f40 msgr2=0x7fc95c0ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:42.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.200+0000 7fc960a6c700 1 --2- 172.21.15.114:0/3392890398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc95c101f40 0x7fc95c0ffef0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc95000c930 tx=0x7fc95000cc40 comp rx=0 tx=0).stop 2024-02-19T19:03:42.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.200+0000 7fc960a6c700 1 -- 172.21.15.114:0/3392890398 shutdown_connections 2024-02-19T19:03:42.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.200+0000 7fc960a6c700 1 --2- 172.21.15.114:0/3392890398 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc94406d5c0 0x7fc94406fa80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:42.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.200+0000 7fc960a6c700 1 --2- 172.21.15.114:0/3392890398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc95c0feec0 0x7fc95c1018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:42.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.200+0000 7fc960a6c700 1 --2- 172.21.15.114:0/3392890398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc95c101f40 0x7fc95c0ffef0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:42.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.200+0000 7fc960a6c700 1 -- 172.21.15.114:0/3392890398 >> 172.21.15.114:0/3392890398 conn(0x7fc95c0fa800 msgr2=0x7fc95c105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:42.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.200+0000 7fc960a6c700 1 -- 172.21.15.114:0/3392890398 shutdown_connections 2024-02-19T19:03:42.221 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.200+0000 7fc960a6c700 1 -- 172.21.15.114:0/3392890398 wait complete. 2024-02-19T19:03:42.221 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 18 2024-02-19T19:03:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.323+0000 7f82bced3700 1 -- 172.21.15.114:0/819096774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b80ffab0 msgr2=0x7f82b80ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.323+0000 7f82bced3700 1 --2- 172.21.15.114:0/819096774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b80ffab0 0x7f82b80ffed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f82a8009910 tx=0x7f82a8009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.323+0000 7f82bced3700 1 -- 172.21.15.114:0/819096774 shutdown_connections 2024-02-19T19:03:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.323+0000 7f82bced3700 1 --2- 172.21.15.114:0/819096774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82b8100cb0 0x7f82b8101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.323+0000 7f82bced3700 1 --2- 172.21.15.114:0/819096774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b80ffab0 0x7f82b80ffed0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:42.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.323+0000 7f82bced3700 1 -- 172.21.15.114:0/819096774 >> 172.21.15.114:0/819096774 conn(0x7f82b80fb050 msgr2=0x7f82b80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.323+0000 7f82bced3700 1 -- 172.21.15.114:0/819096774 shutdown_connections 2024-02-19T19:03:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.323+0000 7f82bced3700 1 -- 172.21.15.114:0/819096774 wait complete. 2024-02-19T19:03:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.324+0000 7f82bced3700 1 Processor -- start 2024-02-19T19:03:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.324+0000 7f82bced3700 1 -- start start 2024-02-19T19:03:42.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.324+0000 7f82bced3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b80ffab0 0x7f82b8069820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.324+0000 7f82bced3700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82b8100cb0 0x7f82b8198eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.324+0000 7f82bced3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b8199480 con 0x7f82b80ffab0 2024-02-19T19:03:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.324+0000 7f82bced3700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b81995f0 con 0x7f82b8100cb0 2024-02-19T19:03:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.325+0000 7f82b659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b80ffab0 0x7f82b8069820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.325+0000 7f82b5d9b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82b8100cb0 0x7f82b8198eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.325+0000 7f82b5d9b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82b8100cb0 0x7f82b8198eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.114:57084/0 (socket says 172.21.15.114:57084) 2024-02-19T19:03:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.325+0000 7f82b5d9b700 1 -- 172.21.15.114:0/3302711881 learned_addr learned my addr 172.21.15.114:0/3302711881 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:03:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.325+0000 7f82b659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b80ffab0 0x7f82b8069820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50488/0 (socket says 172.21.15.114:50488) 2024-02-19T19:03:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.325+0000 7f82b659c700 1 -- 172.21.15.114:0/3302711881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82b8100cb0 msgr2=0x7f82b8198eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.325+0000 7f82b659c700 1 --2- 172.21.15.114:0/3302711881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82b8100cb0 0x7f82b8198eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.325+0000 7f82b659c700 1 -- 172.21.15.114:0/3302711881 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a80095f0 con 0x7f82b80ffab0 2024-02-19T19:03:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.325+0000 7f82b5d9b700 1 --2- 172.21.15.114:0/3302711881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82b8100cb0 0x7f82b8198eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:03:42.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.325+0000 7f82b659c700 1 --2- 172.21.15.114:0/3302711881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b80ffab0 0x7f82b8069820 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f82a8000c00 tx=0x7f82a800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:42.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.325+0000 7f82a77fe700 1 -- 172.21.15.114:0/3302711881 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82a801d070 con 0x7f82b80ffab0 2024-02-19T19:03:42.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.325+0000 7f82a77fe700 1 -- 172.21.15.114:0/3302711881 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f82a8005dc0 con 0x7f82b80ffab0 2024-02-19T19:03:42.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.325+0000 7f82a77fe700 1 -- 172.21.15.114:0/3302711881 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82a8017400 con 0x7f82b80ffab0 2024-02-19T19:03:42.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.325+0000 7f82bced3700 1 -- 172.21.15.114:0/3302711881 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b8199870 con 0x7f82b80ffab0 2024-02-19T19:03:42.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.326+0000 7f82bced3700 1 -- 172.21.15.114:0/3302711881 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b8199ca0 con 0x7f82b80ffab0 2024-02-19T19:03:42.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.327+0000 7f82bced3700 1 -- 172.21.15.114:0/3302711881 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b8108610 con 0x7f82b80ffab0 2024-02-19T19:03:42.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.327+0000 7f82a77fe700 1 -- 172.21.15.114:0/3302711881 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 34) v1 ==== 91245+0+0 (secure 0 0 0) 0x7f82a8003810 con 0x7f82b80ffab0 2024-02-19T19:03:42.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.327+0000 7f82a77fe700 1 --2- 172.21.15.114:0/3302711881 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f82a006d5c0 0x7f82a006fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:42.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.328+0000 7f82a77fe700 1 -- 172.21.15.114:0/3302711881 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f82a808df00 con 0x7f82b80ffab0 2024-02-19T19:03:42.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.328+0000 7f82b5d9b700 1 --2- 172.21.15.114:0/3302711881 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f82a006d5c0 0x7f82a006fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:42.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.328+0000 7f82b5d9b700 1 --2- 172.21.15.114:0/3302711881 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f82a006d5c0 0x7f82a006fa80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f82b8069d80 tx=0x7f82ac009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:42.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.333+0000 7f82a77fe700 1 -- 172.21.15.114:0/3302711881 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f82a8057960 con 0x7f82b80ffab0 2024-02-19T19:03:42.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.555+0000 7f82bced3700 1 -- 172.21.15.114:0/3302711881 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f82b8105600 con 0x7f82a006d5c0 2024-02-19T19:03:42.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.556+0000 7f82a77fe700 1 -- 172.21.15.114:0/3302711881 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f82b8105600 con 0x7f82a006d5c0 2024-02-19T19:03:42.576 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:03:42.576 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:03:42.576 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-02-19T19:03:42.576 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:03:42.576 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [ 2024-02-19T19:03:42.576 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr" 2024-02-19T19:03:42.577 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-02-19T19:03:42.577 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "2/23 daemons upgraded", 2024-02-19T19:03:42.577 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "", 2024-02-19T19:03:42.577 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-02-19T19:03:42.577 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:03:42.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.560+0000 7f82bced3700 1 -- 172.21.15.114:0/3302711881 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f82a006d5c0 msgr2=0x7f82a006fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:42.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.560+0000 7f82bced3700 1 --2- 172.21.15.114:0/3302711881 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f82a006d5c0 0x7f82a006fa80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f82b8069d80 tx=0x7f82ac009260 comp rx=0 tx=0).stop 2024-02-19T19:03:42.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.560+0000 7f82bced3700 1 -- 172.21.15.114:0/3302711881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b80ffab0 msgr2=0x7f82b8069820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:42.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.560+0000 7f82bced3700 1 --2- 172.21.15.114:0/3302711881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b80ffab0 0x7f82b8069820 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f82a8000c00 tx=0x7f82a800f690 comp rx=0 tx=0).stop 2024-02-19T19:03:42.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.560+0000 7f82bced3700 1 -- 172.21.15.114:0/3302711881 shutdown_connections 2024-02-19T19:03:42.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.560+0000 7f82bced3700 1 --2- 172.21.15.114:0/3302711881 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f82a006d5c0 0x7f82a006fa80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:42.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.560+0000 7f82bced3700 1 --2- 172.21.15.114:0/3302711881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82b80ffab0 0x7f82b8069820 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:42.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.560+0000 7f82bced3700 1 --2- 172.21.15.114:0/3302711881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82b8100cb0 0x7f82b8198eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:42.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.560+0000 7f82bced3700 1 -- 172.21.15.114:0/3302711881 >> 172.21.15.114:0/3302711881 conn(0x7f82b80fb050 msgr2=0x7f82b8103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:42.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.560+0000 7f82bced3700 1 -- 172.21.15.114:0/3302711881 shutdown_connections 2024-02-19T19:03:42.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.560+0000 7f82bced3700 1 -- 172.21.15.114:0/3302711881 wait complete. 2024-02-19T19:03:42.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.668+0000 7ff0f7c8e700 1 -- 172.21.15.114:0/1261499422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f00ffab0 msgr2=0x7ff0f00ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.668+0000 7ff0f7c8e700 1 --2- 172.21.15.114:0/1261499422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f00ffab0 0x7ff0f00ffed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8009910 tx=0x7ff0e8009c20 comp rx=0 tx=0).stop 2024-02-19T19:03:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.668+0000 7ff0f7c8e700 1 -- 172.21.15.114:0/1261499422 shutdown_connections 2024-02-19T19:03:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.668+0000 7ff0f7c8e700 1 --2- 172.21.15.114:0/1261499422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0f0100cb0 0x7ff0f0101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.668+0000 7ff0f7c8e700 1 --2- 172.21.15.114:0/1261499422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f00ffab0 0x7ff0f00ffed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.668+0000 7ff0f7c8e700 1 -- 172.21.15.114:0/1261499422 >> 172.21.15.114:0/1261499422 conn(0x7ff0f00fb030 msgr2=0x7ff0f00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.668+0000 7ff0f7c8e700 1 -- 172.21.15.114:0/1261499422 shutdown_connections 2024-02-19T19:03:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.668+0000 7ff0f7c8e700 1 -- 172.21.15.114:0/1261499422 wait complete. 2024-02-19T19:03:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.669+0000 7ff0f7c8e700 1 Processor -- start 2024-02-19T19:03:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.669+0000 7ff0f7c8e700 1 -- start start 2024-02-19T19:03:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.669+0000 7ff0f7c8e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0f00ffab0 0x7ff0f0103a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.669+0000 7ff0f7c8e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f0100cb0 0x7ff0f0102090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.669+0000 7ff0f7c8e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f01025d0 con 0x7ff0f0100cb0 2024-02-19T19:03:42.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.669+0000 7ff0f7c8e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f0102710 con 0x7ff0f00ffab0 2024-02-19T19:03:42.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.669+0000 7ff0f5229700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f0100cb0 0x7ff0f0102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:42.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.669+0000 7ff0f5229700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f0100cb0 0x7ff0f0102090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50508/0 (socket says 172.21.15.114:50508) 2024-02-19T19:03:42.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.669+0000 7ff0f5229700 1 -- 172.21.15.114:0/1879216467 learned_addr learned my addr 172.21.15.114:0/1879216467 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:03:42.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.670+0000 7ff0f5a2a700 1 --2- 172.21.15.114:0/1879216467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0f00ffab0 0x7ff0f0103a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:42.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.670+0000 7ff0f5229700 1 -- 172.21.15.114:0/1879216467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0f00ffab0 msgr2=0x7ff0f0103a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:42.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.670+0000 7ff0f5229700 1 --2- 172.21.15.114:0/1879216467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0f00ffab0 0x7ff0f0103a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:42.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.670+0000 7ff0f5229700 1 -- 172.21.15.114:0/1879216467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e80095f0 con 0x7ff0f0100cb0 2024-02-19T19:03:42.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.670+0000 7ff0f5229700 1 --2- 172.21.15.114:0/1879216467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f0100cb0 0x7ff0f0102090 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400d8d0 tx=0x7ff0e400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:42.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.670+0000 7ff0eeffd700 1 -- 172.21.15.114:0/1879216467 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0e4009940 con 0x7ff0f0100cb0 2024-02-19T19:03:42.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.670+0000 7ff0f7c8e700 1 -- 172.21.15.114:0/1879216467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f01029f0 con 0x7ff0f0100cb0 2024-02-19T19:03:42.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.670+0000 7ff0eeffd700 1 -- 172.21.15.114:0/1879216467 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff0e4010460 con 0x7ff0f0100cb0 2024-02-19T19:03:42.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.670+0000 7ff0eeffd700 1 -- 172.21.15.114:0/1879216467 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0e400f5f0 con 0x7ff0f0100cb0 2024-02-19T19:03:42.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.670+0000 7ff0f7c8e700 1 -- 172.21.15.114:0/1879216467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f019e380 con 0x7ff0f0100cb0 2024-02-19T19:03:42.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.671+0000 7ff0f7c8e700 1 -- 172.21.15.114:0/1879216467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f0108610 con 0x7ff0f0100cb0 2024-02-19T19:03:42.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.671+0000 7ff0eeffd700 1 -- 172.21.15.114:0/1879216467 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 34) v1 ==== 91245+0+0 (secure 0 0 0) 0x7ff0e40105d0 con 0x7ff0f0100cb0 2024-02-19T19:03:42.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.672+0000 7ff0eeffd700 1 --2- 172.21.15.114:0/1879216467 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff0d806d5c0 0x7ff0d806fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:42.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.672+0000 7ff0eeffd700 1 -- 172.21.15.114:0/1879216467 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7ff0e408caa0 con 0x7ff0f0100cb0 2024-02-19T19:03:42.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.672+0000 7ff0f5a2a700 1 --2- 172.21.15.114:0/1879216467 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff0d806d5c0 0x7ff0d806fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:42.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.672+0000 7ff0f5a2a700 1 --2- 172.21.15.114:0/1879216467 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff0d806d5c0 0x7ff0d806fa80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8006010 tx=0x7ff0e800b580 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:42.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.677+0000 7ff0eeffd700 1 -- 172.21.15.114:0/1879216467 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff0e4056500 con 0x7ff0f0100cb0 2024-02-19T19:03:42.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.728+0000 7ff0eeffd700 1 -- 172.21.15.114:0/1879216467 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 35) v1 ==== 91541+0+0 (secure 0 0 0) 0x7ff0e40568d0 con 0x7ff0f0100cb0 2024-02-19T19:03:42.847 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:42 smithi114 ceph-mon[92983]: [19/Feb/2024:19:03:40] ENGINE Serving on https://172.21.15.114:7150 2024-02-19T19:03:42.847 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:42 smithi114 ceph-mon[92983]: [19/Feb/2024:19:03:40] ENGINE Serving on http://172.21.15.114:8765 2024-02-19T19:03:42.847 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:42 smithi114 ceph-mon[92983]: [19/Feb/2024:19:03:40] ENGINE Bus STARTED 2024-02-19T19:03:42.847 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:42 smithi114 ceph-mon[92983]: from='client.34128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:42.848 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:42 smithi114 ceph-mon[92983]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:42.848 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:42 smithi114 ceph-mon[92983]: pgmap v4: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:03:42.848 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:42 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/177802722' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:42.848 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:42 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/3392890398' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:03:42.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.965+0000 7ff0f7c8e700 1 -- 172.21.15.114:0/1879216467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff0f0108800 con 0x7ff0f0100cb0 2024-02-19T19:03:42.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.965+0000 7ff0eeffd700 1 -- 172.21.15.114:0/1879216467 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7ff0e408a2d0 con 0x7ff0f0100cb0 2024-02-19T19:03:42.985 INFO:teuthology.orchestra.run.smithi114.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:03:42.985 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:03:42.985 INFO:teuthology.orchestra.run.smithi114.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:03:42.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.967+0000 7ff0f7c8e700 1 -- 172.21.15.114:0/1879216467 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff0d806d5c0 msgr2=0x7ff0d806fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:42.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.967+0000 7ff0f7c8e700 1 --2- 172.21.15.114:0/1879216467 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff0d806d5c0 0x7ff0d806fa80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8006010 tx=0x7ff0e800b580 comp rx=0 tx=0).stop 2024-02-19T19:03:42.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.967+0000 7ff0f7c8e700 1 -- 172.21.15.114:0/1879216467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f0100cb0 msgr2=0x7ff0f0102090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:42.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.967+0000 7ff0f7c8e700 1 --2- 172.21.15.114:0/1879216467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f0100cb0 0x7ff0f0102090 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400d8d0 tx=0x7ff0e400dc90 comp rx=0 tx=0).stop 2024-02-19T19:03:42.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.967+0000 7ff0f7c8e700 1 -- 172.21.15.114:0/1879216467 shutdown_connections 2024-02-19T19:03:42.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.967+0000 7ff0f7c8e700 1 --2- 172.21.15.114:0/1879216467 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff0d806d5c0 0x7ff0d806fa80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:42.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.967+0000 7ff0f7c8e700 1 --2- 172.21.15.114:0/1879216467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0f00ffab0 0x7ff0f0103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:42.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.967+0000 7ff0f7c8e700 1 --2- 172.21.15.114:0/1879216467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0f0100cb0 0x7ff0f0102090 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:42.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.967+0000 7ff0f7c8e700 1 -- 172.21.15.114:0/1879216467 >> 172.21.15.114:0/1879216467 conn(0x7ff0f00fb030 msgr2=0x7ff0f00683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:42.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.967+0000 7ff0f7c8e700 1 -- 172.21.15.114:0/1879216467 shutdown_connections 2024-02-19T19:03:42.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:03:42.967+0000 7ff0f7c8e700 1 -- 172.21.15.114:0/1879216467 wait complete. 2024-02-19T19:03:43.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:42 smithi174 ceph-mon[41061]: [19/Feb/2024:19:03:40] ENGINE Serving on https://172.21.15.114:7150 2024-02-19T19:03:43.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:42 smithi174 ceph-mon[41061]: [19/Feb/2024:19:03:40] ENGINE Serving on http://172.21.15.114:8765 2024-02-19T19:03:43.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:42 smithi174 ceph-mon[41061]: [19/Feb/2024:19:03:40] ENGINE Bus STARTED 2024-02-19T19:03:43.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:42 smithi174 ceph-mon[41061]: from='client.34128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:43.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:42 smithi174 ceph-mon[41061]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:43.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:42 smithi174 ceph-mon[41061]: pgmap v4: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:03:43.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:42 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/177802722' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:43.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:42 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/3392890398' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:03:44.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:43 smithi114 ceph-mon[92983]: from='client.34148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:44.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:43 smithi114 ceph-mon[92983]: mgrmap e35: smithi114.pcvtrk(active, since 3s), standbys: smithi174.lydvnj 2024-02-19T19:03:44.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:43 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/1879216467' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:03:44.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:43 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:44.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:43 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:44.143 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:43 smithi174 ceph-mon[41061]: from='client.34148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:44.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:43 smithi174 ceph-mon[41061]: mgrmap e35: smithi114.pcvtrk(active, since 3s), standbys: smithi174.lydvnj 2024-02-19T19:03:44.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:43 smithi174 ceph-mon[41061]: from='client.? 172.21.15.114:0/1879216467' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:03:44.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:43 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:44.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:43 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:44.917 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:44 smithi114 ceph-mon[92983]: pgmap v5: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:03:44.917 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:44 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:44.917 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:44 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:45.040 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:44 smithi174 ceph-mon[41061]: pgmap v5: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-19T19:03:45.040 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:44 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:45.040 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:44 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:46.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:45 smithi114 ceph-mon[92983]: mgrmap e36: smithi114.pcvtrk(active, since 5s), standbys: smithi174.lydvnj 2024-02-19T19:03:46.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:45 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:46.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:45 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:46.056 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:45 smithi174 ceph-mon[41061]: mgrmap e36: smithi114.pcvtrk(active, since 5s), standbys: smithi174.lydvnj 2024-02-19T19:03:46.056 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:45 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:46.056 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:45 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:47.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:46 smithi114 ceph-mon[92983]: pgmap v6: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 43 KiB/s rd, 0 B/s wr, 26 op/s 2024-02-19T19:03:47.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:46 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:47.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:46 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:47.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:46 smithi174 ceph-mon[41061]: pgmap v6: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 43 KiB/s rd, 0 B/s wr, 26 op/s 2024-02-19T19:03:47.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:46 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:47.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:46 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:49.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:48 smithi114 ceph-mon[92983]: pgmap v7: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-02-19T19:03:49.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:48 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:49.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:48 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:49.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:48 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:03:49.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:48 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:03:49.059 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:48 smithi174 ceph-mon[41061]: pgmap v7: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-02-19T19:03:49.060 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:48 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:49.060 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:48 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:49.060 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:48 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:03:49.060 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:48 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:03:49.957 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:49 smithi174 ceph-mon[41061]: Updating smithi114:/etc/ceph/ceph.conf 2024-02-19T19:03:49.957 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:49 smithi174 ceph-mon[41061]: Updating smithi174:/etc/ceph/ceph.conf 2024-02-19T19:03:49.958 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:49 smithi174 ceph-mon[41061]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:03:49.958 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:49 smithi174 ceph-mon[41061]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:03:49.958 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:49 smithi174 ceph-mon[41061]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:03:49.958 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:49 smithi174 ceph-mon[41061]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:03:49.958 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:49 smithi174 ceph-mon[41061]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:03:49.958 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:49 smithi174 ceph-mon[41061]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:03:49.958 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:49 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:49.958 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:49 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:49.958 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:49 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:49.958 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:49 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:49.958 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:49 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:49.958 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:49 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:49.958 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:49 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:49.958 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:49 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-19T19:03:50.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:49 smithi114 ceph-mon[92983]: Updating smithi114:/etc/ceph/ceph.conf 2024-02-19T19:03:50.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:49 smithi114 ceph-mon[92983]: Updating smithi174:/etc/ceph/ceph.conf 2024-02-19T19:03:50.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:49 smithi114 ceph-mon[92983]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:03:50.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:49 smithi114 ceph-mon[92983]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:03:50.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:49 smithi114 ceph-mon[92983]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:03:50.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:49 smithi114 ceph-mon[92983]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:03:50.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:49 smithi114 ceph-mon[92983]: Updating smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:03:50.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:49 smithi114 ceph-mon[92983]: Updating smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:03:50.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:49 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:50.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:49 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:50.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:49 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:50.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:49 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:50.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:49 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:50.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:49 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:50.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:49 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:50.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:49 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-19T19:03:51.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:50 smithi114 ceph-mon[92983]: pgmap v8: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-02-19T19:03:51.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:50 smithi174 ceph-mon[41061]: pgmap v8: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-02-19T19:03:52.056 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:51 smithi174 ceph-mon[41061]: Upgrade: Updating mon.smithi174 2024-02-19T19:03:52.057 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:51 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:52.057 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:51 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T19:03:52.057 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:51 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T19:03:52.057 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:51 smithi174 ceph-mon[41061]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:03:52.057 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:51 smithi174 ceph-mon[41061]: Deploying daemon mon.smithi174 on smithi174 2024-02-19T19:03:52.057 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:51 smithi174 ceph-mon[41061]: pgmap v9: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:03:52.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:51 smithi114 ceph-mon[92983]: Upgrade: Updating mon.smithi174 2024-02-19T19:03:52.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:51 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:52.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:51 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T19:03:52.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:51 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T19:03:52.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:51 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:03:52.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:51 smithi114 ceph-mon[92983]: Deploying daemon mon.smithi174 on smithi174 2024-02-19T19:03:52.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:51 smithi114 ceph-mon[92983]: pgmap v9: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:03:52.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:52 smithi174 systemd[1]: Stopping Ceph mon.smithi174 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T19:03:53.151 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:52 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi174[41049]: 2024-02-19T19:03:52.876+0000 7f1808acb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi174 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T19:03:53.151 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:52 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi174[41049]: 2024-02-19T19:03:52.876+0000 7f1808acb700 -1 mon.smithi174@1(peon) e2 *** Got Signal Terminated *** 2024-02-19T19:03:53.464 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:53 smithi174 podman[87114]: 2024-02-19 19:03:53.132276355 +0000 UTC m=+0.376760814 container died 70dd7bf69fd96add53ce87c186fe201334b327ce7670fe8543f403bfc2a040bf (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi174, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:03:53.789 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:53 smithi174 podman[87114]: 2024-02-19 19:03:53.535832463 +0000 UTC m=+0.780316921 container cleanup 70dd7bf69fd96add53ce87c186fe201334b327ce7670fe8543f403bfc2a040bf (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi174, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:03:53.789 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:53 smithi174 bash[87114]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi174 2024-02-19T19:03:54.098 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:53 smithi174 podman[87128]: 2024-02-19 19:03:53.77050108 +0000 UTC m=+0.638576562 container remove 70dd7bf69fd96add53ce87c186fe201334b327ce7670fe8543f403bfc2a040bf (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi174, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-19T19:03:54.098 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:53 smithi174 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@mon.smithi174.service: Deactivated successfully. 2024-02-19T19:03:54.098 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:53 smithi174 systemd[1]: Stopped Ceph mon.smithi174 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T19:03:54.098 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:53 smithi174 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@mon.smithi174.service: Consumed 5.811s CPU time. 2024-02-19T19:03:54.348 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 systemd[1]: Starting Ceph mon.smithi174 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T19:03:54.349 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 podman[87260]: 2024-02-19 19:03:54.21965846 +0000 UTC m=+0.017910874 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:03:54.349 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 podman[87260]: 2024-02-19 19:03:54.329911448 +0000 UTC m=+0.128163850 container create e811581f05fb85e91a16941d16f8364f16bae9a2976ba051ac8c42e541f0fc58 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi174, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214) 2024-02-19T19:03:54.611 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 podman[87260]: 2024-02-19 19:03:54.566013233 +0000 UTC m=+0.364265636 container init e811581f05fb85e91a16941d16f8364f16bae9a2976ba051ac8c42e541f0fc58 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi174, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:03:54.611 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 podman[87260]: 2024-02-19 19:03:54.569102407 +0000 UTC m=+0.367354809 container start e811581f05fb85e91a16941d16f8364f16bae9a2976ba051ac8c42e541f0fc58 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi174, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T19:03:54.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: set uid:gid to 167:167 (ceph:ceph) 2024-02-19T19:03:54.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev), process ceph-mon, pid 2 2024-02-19T19:03:54.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: pidfile_write: ignore empty --pid-file 2024-02-19T19:03:54.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: load: jerasure load: lrc 2024-02-19T19:03:54.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: RocksDB version: 7.9.2 2024-02-19T19:03:54.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Git sha 0 2024-02-19T19:03:54.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Compile date 2024-02-15 18:05:23 2024-02-19T19:03:54.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: DB SUMMARY 2024-02-19T19:03:54.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: DB Session ID: HVAX2FJ4B87XYJVQX0I8 2024-02-19T19:03:54.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: CURRENT file: CURRENT 2024-02-19T19:03:54.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: IDENTITY file: IDENTITY 2024-02-19T19:03:54.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1411 Bytes 2024-02-19T19:03:54.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi174/store.db dir, Total Num: 1, files: 000027.sst 2024-02-19T19:03:54.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi174/store.db: 000025.log size: 6556317 ; 2024-02-19T19:03:54.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.error_if_exists: 0 2024-02-19T19:03:54.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.create_if_missing: 0 2024-02-19T19:03:54.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.paranoid_checks: 1 2024-02-19T19:03:54.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-19T19:03:54.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-19T19:03:54.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-19T19:03:54.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.env: 0x5598578e8ae0 2024-02-19T19:03:54.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.fs: PosixFileSystem 2024-02-19T19:03:54.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.info_log: 0x559859bdf760 2024-02-19T19:03:54.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_file_opening_threads: 16 2024-02-19T19:03:54.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.statistics: (nil) 2024-02-19T19:03:54.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.use_fsync: 0 2024-02-19T19:03:54.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_log_file_size: 0 2024-02-19T19:03:54.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T19:03:54.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T19:03:54.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.keep_log_file_num: 1000 2024-02-19T19:03:54.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.recycle_log_file_num: 0 2024-02-19T19:03:54.897 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.allow_fallocate: 1 2024-02-19T19:03:54.897 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.allow_mmap_reads: 0 2024-02-19T19:03:54.897 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.allow_mmap_writes: 0 2024-02-19T19:03:54.897 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.use_direct_reads: 0 2024-02-19T19:03:54.897 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T19:03:54.897 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.create_missing_column_families: 0 2024-02-19T19:03:54.897 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.db_log_dir: 2024-02-19T19:03:54.897 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.wal_dir: 2024-02-19T19:03:54.897 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T19:03:54.899 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T19:03:54.899 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T19:03:54.899 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T19:03:54.899 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T19:03:54.899 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T19:03:54.899 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.advise_random_on_open: 1 2024-02-19T19:03:54.899 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.db_write_buffer_size: 0 2024-02-19T19:03:54.899 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.write_buffer_manager: 0x559859bb3400 2024-02-19T19:03:54.899 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T19:03:54.899 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T19:03:54.899 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T19:03:54.899 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.rate_limiter: (nil) 2024-02-19T19:03:54.899 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T19:03:54.900 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.wal_recovery_mode: 2 2024-02-19T19:03:54.900 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.enable_thread_tracking: 0 2024-02-19T19:03:54.900 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.enable_pipelined_write: 0 2024-02-19T19:03:54.900 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.unordered_write: 0 2024-02-19T19:03:54.900 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T19:03:54.900 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T19:03:54.900 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T19:03:54.900 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T19:03:54.900 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.row_cache: None 2024-02-19T19:03:54.900 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.wal_filter: None 2024-02-19T19:03:54.900 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T19:03:54.900 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.allow_ingest_behind: 0 2024-02-19T19:03:54.900 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.two_write_queues: 0 2024-02-19T19:03:54.901 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.manual_wal_flush: 0 2024-02-19T19:03:54.901 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.wal_compression: 0 2024-02-19T19:03:54.901 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.atomic_flush: 0 2024-02-19T19:03:54.901 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T19:03:54.901 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T19:03:54.901 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T19:03:54.901 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.log_readahead_size: 0 2024-02-19T19:03:54.901 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-19T19:03:54.901 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.best_efforts_recovery: 0 2024-02-19T19:03:54.901 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-19T19:03:54.901 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-19T19:03:54.901 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.allow_data_in_errors: 0 2024-02-19T19:03:54.901 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.db_host_id: __hostname__ 2024-02-19T19:03:54.901 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-19T19:03:54.901 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_background_jobs: 2 2024-02-19T19:03:54.902 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_background_compactions: -1 2024-02-19T19:03:54.902 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_subcompactions: 1 2024-02-19T19:03:54.902 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T19:03:54.902 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T19:03:54.902 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T19:03:54.902 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_total_wal_size: 0 2024-02-19T19:03:54.902 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T19:03:54.902 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T19:03:54.902 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T19:03:54.902 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T19:03:54.902 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_open_files: -1 2024-02-19T19:03:54.902 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.bytes_per_sync: 0 2024-02-19T19:03:54.902 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T19:03:54.902 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T19:03:54.902 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compaction_readahead_size: 0 2024-02-19T19:03:54.903 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_background_flushes: -1 2024-02-19T19:03:54.903 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Compression algorithms supported: 2024-02-19T19:03:54.903 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T19:03:54.903 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: kZSTD supported: 0 2024-02-19T19:03:54.903 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: kXpressCompression supported: 0 2024-02-19T19:03:54.903 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: kLZ4HCCompression supported: 1 2024-02-19T19:03:54.903 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: kZlibCompression supported: 1 2024-02-19T19:03:54.903 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: kSnappyCompression supported: 1 2024-02-19T19:03:54.903 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: kLZ4Compression supported: 1 2024-02-19T19:03:54.903 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: kBZip2Compression supported: 0 2024-02-19T19:03:54.903 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T19:03:54.903 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-19T19:03:54.903 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi174/store.db/MANIFEST-000010 2024-02-19T19:03:54.903 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-19T19:03:54.903 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T19:03:54.904 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.merge_operator: 2024-02-19T19:03:54.904 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compaction_filter: None 2024-02-19T19:03:54.904 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compaction_filter_factory: None 2024-02-19T19:03:54.904 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.sst_partitioner_factory: None 2024-02-19T19:03:54.904 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T19:03:54.904 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T19:03:54.904 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559859bdf880) 2024-02-19T19:03:54.905 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: cache_index_and_filter_blocks: 1 2024-02-19T19:03:54.905 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T19:03:54.905 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T19:03:54.905 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: pin_top_level_index_and_filter: 1 2024-02-19T19:03:54.905 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: index_type: 0 2024-02-19T19:03:54.905 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: data_block_index_type: 0 2024-02-19T19:03:54.905 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: index_shortening: 1 2024-02-19T19:03:54.905 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-19T19:03:54.905 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: checksum: 4 2024-02-19T19:03:54.905 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: no_block_cache: 0 2024-02-19T19:03:54.906 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: block_cache: 0x559859be3090 2024-02-19T19:03:54.906 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: block_cache_name: BinnedLRUCache 2024-02-19T19:03:54.906 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: block_cache_options: 2024-02-19T19:03:54.906 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: capacity : 536870912 2024-02-19T19:03:54.906 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: num_shard_bits : 4 2024-02-19T19:03:54.906 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: strict_capacity_limit : 0 2024-02-19T19:03:54.906 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: high_pri_pool_ratio: 0.000 2024-02-19T19:03:54.906 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: block_cache_compressed: (nil) 2024-02-19T19:03:54.906 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: persistent_cache: (nil) 2024-02-19T19:03:54.906 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: block_size: 4096 2024-02-19T19:03:54.906 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: block_size_deviation: 10 2024-02-19T19:03:54.906 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: block_restart_interval: 16 2024-02-19T19:03:54.906 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: index_block_restart_interval: 1 2024-02-19T19:03:54.906 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: metadata_block_size: 4096 2024-02-19T19:03:54.906 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: partition_filters: 0 2024-02-19T19:03:54.907 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: use_delta_encoding: 1 2024-02-19T19:03:54.907 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: filter_policy: bloomfilter 2024-02-19T19:03:54.907 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: whole_key_filtering: 1 2024-02-19T19:03:54.907 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: verify_compression: 0 2024-02-19T19:03:54.907 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: read_amp_bytes_per_bit: 0 2024-02-19T19:03:54.907 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: format_version: 5 2024-02-19T19:03:54.907 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: enable_index_compression: 1 2024-02-19T19:03:54.907 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: block_align: 0 2024-02-19T19:03:54.907 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: max_auto_readahead_size: 262144 2024-02-19T19:03:54.907 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: prepopulate_block_cache: 0 2024-02-19T19:03:54.907 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: initial_auto_readahead_size: 8192 2024-02-19T19:03:54.907 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: num_file_reads_for_auto_readahead: 2 2024-02-19T19:03:54.907 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.write_buffer_size: 33554432 2024-02-19T19:03:54.908 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_write_buffer_number: 2 2024-02-19T19:03:54.908 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compression: NoCompression 2024-02-19T19:03:54.908 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.bottommost_compression: Disabled 2024-02-19T19:03:54.908 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.prefix_extractor: nullptr 2024-02-19T19:03:54.908 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T19:03:54.908 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.num_levels: 7 2024-02-19T19:03:54.908 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T19:03:54.908 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T19:03:54.908 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T19:03:54.908 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T19:03:54.908 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T19:03:54.908 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T19:03:54.908 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T19:03:54.909 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T19:03:54.909 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-19T19:03:54.909 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T19:03:54.909 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-19T19:03:54.909 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-19T19:03:54.909 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T19:03:54.909 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compression_opts.level: 32767 2024-02-19T19:03:54.909 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compression_opts.strategy: 0 2024-02-19T19:03:54.909 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T19:03:54.909 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T19:03:54.909 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-19T19:03:54.909 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-19T19:03:54.909 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compression_opts.enabled: false 2024-02-19T19:03:54.909 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-19T19:03:54.909 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T19:03:54.910 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T19:03:54.910 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T19:03:54.910 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.target_file_size_base: 67108864 2024-02-19T19:03:54.910 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T19:03:54.910 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T19:03:54.910 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T19:03:54.910 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T19:03:54.910 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T19:03:54.910 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T19:03:54.910 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T19:03:54.910 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T19:03:54.910 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T19:03:54.910 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T19:03:54.910 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T19:03:54.910 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T19:03:54.911 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T19:03:54.911 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-19T19:03:54.911 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.arena_block_size: 1048576 2024-02-19T19:03:54.911 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T19:03:54.911 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T19:03:54.911 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.disable_auto_compactions: 0 2024-02-19T19:03:54.911 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T19:03:54.911 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T19:03:54.911 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T19:03:54.911 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T19:03:54.911 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T19:03:54.911 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T19:03:54.911 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T19:03:54.911 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T19:03:54.912 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T19:03:54.912 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T19:03:54.912 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.table_properties_collectors: 2024-02-19T19:03:54.912 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.inplace_update_support: 0 2024-02-19T19:03:54.912 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T19:03:54.912 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T19:03:54.912 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T19:03:54.912 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T19:03:54.912 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.bloom_locality: 0 2024-02-19T19:03:54.912 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.max_successive_merges: 0 2024-02-19T19:03:54.912 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T19:03:54.912 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.paranoid_file_checks: 0 2024-02-19T19:03:54.912 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.force_consistency_checks: 1 2024-02-19T19:03:54.913 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.report_bg_io_stats: 0 2024-02-19T19:03:54.913 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.ttl: 2592000 2024-02-19T19:03:54.913 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T19:03:54.913 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-19T19:03:54.913 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-19T19:03:54.913 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.enable_blob_files: false 2024-02-19T19:03:54.913 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.min_blob_size: 0 2024-02-19T19:03:54.913 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.blob_file_size: 268435456 2024-02-19T19:03:54.913 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-19T19:03:54.913 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-19T19:03:54.913 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-19T19:03:54.913 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-19T19:03:54.913 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-19T19:03:54.913 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.blob_file_starting_level: 0 2024-02-19T19:03:54.914 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-19T19:03:54.914 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-19T19:03:54.914 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi174/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 29, last_sequence is 11662, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 25 2024-02-19T19:03:54.914 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-02-19T19:03:54.914 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fcaa4927-5ace-4f3c-a090-68d2f97f0518 2024-02-19T19:03:54.914 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708369434598855, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-02-19T19:03:54.914 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-02-19T19:03:54.914 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708369434614316, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 3931864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11667, "largest_seqno": 13263, "table_properties": {"data_size": 3924958, "index_size": 4223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1669, "raw_key_size": 14858, "raw_average_key_size": 23, "raw_value_size": 3911325, "raw_average_value_size": 6101, "num_data_blocks": 192, "num_entries": 641, "num_filter_entries": 641, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1708369434, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fcaa4927-5ace-4f3c-a090-68d2f97f0518", "db_session_id": "HVAX2FJ4B87XYJVQX0I8", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-02-19T19:03:54.914 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708369434614428, "job": 1, "event": "recovery_finished"} 2024-02-19T19:03:54.914 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-02-19T19:03:54.914 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-19T19:03:54.914 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi174/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-19T19:03:54.914 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559859cd2000 2024-02-19T19:03:54.914 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: DB pointer 0x559859cbe000 2024-02-19T19:03:54.915 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-19T19:03:54.915 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-19T19:03:54.915 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: ** DB Stats ** 2024-02-19T19:03:54.915 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T19:03:54.915 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T19:03:54.915 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T19:03:54.915 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T19:03:54.915 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T19:03:54.915 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T19:03:54.915 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T19:03:54.915 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: 2024-02-19T19:03:54.915 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: ** Compaction Stats [default] ** 2024-02-19T19:03:54.915 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-19T19:03:54.915 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-19T19:03:54.916 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: L0 1/0 3.75 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 531.6 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-02-19T19:03:54.916 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: L6 1/0 4.76 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-19T19:03:54.916 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Sum 2/0 8.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 531.6 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-02-19T19:03:54.916 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 531.6 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-02-19T19:03:54.916 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: 2024-02-19T19:03:54.916 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: ** Compaction Stats [default] ** 2024-02-19T19:03:54.916 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-19T19:03:54.916 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T19:03:54.916 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 531.6 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-02-19T19:03:54.916 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: 2024-02-19T19:03:54.916 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-19T19:03:54.916 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: 2024-02-19T19:03:54.916 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T19:03:54.916 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-02-19T19:03:54.917 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T19:03:54.917 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T19:03:54.917 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T19:03:54.917 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-19T19:03:54.917 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Cumulative compaction: 0.00 GB write, 207.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T19:03:54.917 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Interval compaction: 0.00 GB write, 207.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T19:03:54.917 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T19:03:54.917 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Block cache BinnedLRUCache@0x559859be3090#2 capacity: 512.00 MB usage: 6.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-02-19T19:03:54.917 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-02-19T19:03:54.917 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: 2024-02-19T19:03:54.917 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-19T19:03:54.917 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: starting mon.smithi174 rank 1 at public addrs [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] at bind addrs [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi174 fsid be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T19:03:54.917 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: mon.smithi174@-1(???) e2 preinit fsid be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T19:03:54.917 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: mon.smithi174@-1(???).mds e18 new map 2024-02-19T19:03:54.918 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: mon.smithi174@-1(???).mds e18 print_map 2024-02-19T19:03:54.918 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: e18 2024-02-19T19:03:54.918 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:03:54.918 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:03:54.918 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: legacy client fscid: 1 2024-02-19T19:03:54.918 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: 2024-02-19T19:03:54.918 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: Filesystem 'cephfs' (1) 2024-02-19T19:03:54.918 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: fs_name cephfs 2024-02-19T19:03:54.918 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: epoch 18 2024-02-19T19:03:54.918 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:03:54.918 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:03:54.918 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: modified 2024-02-19T19:00:21.203922+0000 2024-02-19T19:03:54.918 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: tableserver 0 2024-02-19T19:03:54.918 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: root 0 2024-02-19T19:03:54.918 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: session_timeout 60 2024-02-19T19:03:54.919 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: session_autoclose 300 2024-02-19T19:03:54.919 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: max_file_size 1099511627776 2024-02-19T19:03:54.919 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: max_xattr_size 65536 2024-02-19T19:03:54.919 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: required_client_features {} 2024-02-19T19:03:54.919 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: last_failure 0 2024-02-19T19:03:54.919 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: last_failure_osd_epoch 47 2024-02-19T19:03:54.919 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:03:54.919 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: max_mds 2 2024-02-19T19:03:54.919 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: in 0,1 2024-02-19T19:03:54.919 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: up {0=14530,1=24269} 2024-02-19T19:03:54.919 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: failed 2024-02-19T19:03:54.919 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: damaged 2024-02-19T19:03:54.919 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: stopped 2024-02-19T19:03:54.919 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: data_pools [3] 2024-02-19T19:03:54.919 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: metadata_pool 2 2024-02-19T19:03:54.920 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: inline_data enabled 2024-02-19T19:03:54.920 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: balancer 2024-02-19T19:03:54.920 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: bal_rank_mask -1 2024-02-19T19:03:54.920 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: standby_count_wanted 1 2024-02-19T19:03:54.920 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: [mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:54.920 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: [mds.cephfs.smithi114.vpoeop{0:14540} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:54.920 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: [mds.cephfs.smithi174.znhuvd{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:54.920 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: [mds.cephfs.smithi174.stnfnf{1:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:03:54.920 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: 2024-02-19T19:03:54.920 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout: 2024-02-19T19:03:54.920 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: mon.smithi174@-1(???).osd e50 crush map has features 3314933000852226048, adjusting msgr requires 2024-02-19T19:03:54.920 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: mon.smithi174@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-02-19T19:03:54.920 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: mon.smithi174@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-02-19T19:03:54.920 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: mon.smithi174@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-02-19T19:03:54.921 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 ceph-mon[87279]: mon.smithi174@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-02-19T19:03:54.921 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 bash[87260]: e811581f05fb85e91a16941d16f8364f16bae9a2976ba051ac8c42e541f0fc58 2024-02-19T19:03:54.921 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:54 smithi174 systemd[1]: Started Ceph mon.smithi174 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T19:03:56.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T19:03:56.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:56 smithi174 ceph-mon[87279]: mon.smithi174 calling monitor election 2024-02-19T19:03:56.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:56 smithi174 ceph-mon[87279]: mon.smithi114 calling monitor election 2024-02-19T19:03:56.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T19:03:56.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:56 smithi174 ceph-mon[87279]: mon.smithi114 is new leader, mons smithi114,smithi174 in quorum (ranks 0,1) 2024-02-19T19:03:56.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:56 smithi174 ceph-mon[87279]: monmap e3: 2 mons at {smithi114=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],smithi174=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-19T19:03:56.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:56 smithi174 ceph-mon[87279]: fsmap cephfs:2 {0=cephfs.smithi114.tcwpod=up:active,1=cephfs.smithi174.znhuvd=up:active} 2 up:standby-replay 2024-02-19T19:03:56.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:56 smithi174 ceph-mon[87279]: osdmap e50: 6 total, 6 up, 6 in 2024-02-19T19:03:56.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:56 smithi174 ceph-mon[87279]: mgrmap e36: smithi114.pcvtrk(active, since 15s), standbys: smithi174.lydvnj 2024-02-19T19:03:56.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:56 smithi174 ceph-mon[87279]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:03:56.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:56 smithi174 ceph-mon[87279]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:03:56.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:56 smithi174 ceph-mon[87279]: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:03:56.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:56.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:56.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:56.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:56 smithi174 ceph-mon[87279]: pgmap v11: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:03:56.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi114"}]: dispatch 2024-02-19T19:03:56.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:56 smithi114 ceph-mon[92983]: mon.smithi174 calling monitor election 2024-02-19T19:03:56.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:56 smithi114 ceph-mon[92983]: mon.smithi114 calling monitor election 2024-02-19T19:03:56.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mon metadata", "id": "smithi174"}]: dispatch 2024-02-19T19:03:56.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:56 smithi114 ceph-mon[92983]: mon.smithi114 is new leader, mons smithi114,smithi174 in quorum (ranks 0,1) 2024-02-19T19:03:56.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:56 smithi114 ceph-mon[92983]: monmap e3: 2 mons at {smithi114=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],smithi174=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-19T19:03:56.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:56 smithi114 ceph-mon[92983]: fsmap cephfs:2 {0=cephfs.smithi114.tcwpod=up:active,1=cephfs.smithi174.znhuvd=up:active} 2 up:standby-replay 2024-02-19T19:03:56.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:56 smithi114 ceph-mon[92983]: osdmap e50: 6 total, 6 up, 6 in 2024-02-19T19:03:56.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:56 smithi114 ceph-mon[92983]: mgrmap e36: smithi114.pcvtrk(active, since 15s), standbys: smithi174.lydvnj 2024-02-19T19:03:56.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:56 smithi114 ceph-mon[92983]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:03:56.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:56 smithi114 ceph-mon[92983]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:03:56.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:56 smithi114 ceph-mon[92983]: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:03:56.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:56.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:56.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:56.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:56 smithi114 ceph-mon[92983]: pgmap v11: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-19T19:03:58.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:57 smithi174 ceph-mon[87279]: pgmap v12: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:03:58.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:57 smithi114 ceph-mon[92983]: pgmap v12: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:03:59.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:59 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:59.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:03:59 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:59.643 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:59 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:03:59.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:03:59 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:00.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:00 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:00.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:00 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:00.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:00 smithi174 ceph-mon[87279]: pgmap v13: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:00.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:00 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:00.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:00 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:00.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:00 smithi114 ceph-mon[92983]: pgmap v13: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:01.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:01 smithi174 ceph-mon[87279]: pgmap v14: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:02.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:01 smithi114 ceph-mon[92983]: pgmap v14: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:04.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:04.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:04.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:04.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:04:04.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:04.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:04 smithi174 ceph-mon[87279]: Reconfiguring mon.smithi114 (monmap changed)... 2024-02-19T19:04:04.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T19:04:04.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T19:04:04.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:04.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:04 smithi174 ceph-mon[87279]: Reconfiguring daemon mon.smithi114 on smithi114 2024-02-19T19:04:04.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:04 smithi174 ceph-mon[87279]: pgmap v15: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:04.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:04.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:04.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:04.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:04:04.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:04.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:04 smithi114 ceph-mon[92983]: Reconfiguring mon.smithi114 (monmap changed)... 2024-02-19T19:04:04.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T19:04:04.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T19:04:04.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:04.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:04 smithi114 ceph-mon[92983]: Reconfiguring daemon mon.smithi114 on smithi114 2024-02-19T19:04:04.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:04 smithi114 ceph-mon[92983]: pgmap v15: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:06.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:06 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:06.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:06 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:06.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:06 smithi114 ceph-mon[92983]: Reconfiguring mgr.smithi114.pcvtrk (monmap changed)... 2024-02-19T19:04:06.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:06 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi114.pcvtrk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:04:06.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:06 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:04:06.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:06 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:06.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:06 smithi114 ceph-mon[92983]: Reconfiguring daemon mgr.smithi114.pcvtrk on smithi114 2024-02-19T19:04:06.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:06 smithi114 ceph-mon[92983]: pgmap v16: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:06.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:06 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:06.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:06 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:06.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:06 smithi174 ceph-mon[87279]: Reconfiguring mgr.smithi114.pcvtrk (monmap changed)... 2024-02-19T19:04:06.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:06 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi114.pcvtrk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:04:06.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:06 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:04:06.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:06 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:06.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:06 smithi174 ceph-mon[87279]: Reconfiguring daemon mgr.smithi114.pcvtrk on smithi114 2024-02-19T19:04:06.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:06 smithi174 ceph-mon[87279]: pgmap v16: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:08.225 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:07 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:08.225 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:07 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:08.225 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:07 smithi114 ceph-mon[92983]: Reconfiguring ceph-exporter.smithi114 (monmap changed)... 2024-02-19T19:04:08.225 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:07 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi114", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:04:08.225 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:07 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi114", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:04:08.225 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:07 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi114", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-02-19T19:04:08.225 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:07 smithi114 ceph-mon[92983]: Unable to update caps for client.ceph-exporter.smithi114 2024-02-19T19:04:08.225 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:07 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi114"}]: dispatch 2024-02-19T19:04:08.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:07 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:08.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:07 smithi114 ceph-mon[92983]: Reconfiguring daemon ceph-exporter.smithi114 on smithi114 2024-02-19T19:04:08.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:07 smithi114 ceph-mon[92983]: pgmap v17: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:08.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:07 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:08.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:07 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:08.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:07 smithi174 ceph-mon[87279]: Reconfiguring ceph-exporter.smithi114 (monmap changed)... 2024-02-19T19:04:08.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:07 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi114", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:04:08.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:07 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi114", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:04:08.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:07 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi114", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-02-19T19:04:08.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:07 smithi174 ceph-mon[87279]: Unable to update caps for client.ceph-exporter.smithi114 2024-02-19T19:04:08.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:07 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi114"}]: dispatch 2024-02-19T19:04:08.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:07 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:08.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:07 smithi174 ceph-mon[87279]: Reconfiguring daemon ceph-exporter.smithi114 on smithi114 2024-02-19T19:04:08.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:07 smithi174 ceph-mon[87279]: pgmap v17: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:10.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:09 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:10.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:09 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:10.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:09 smithi114 ceph-mon[92983]: Reconfiguring crash.smithi114 (monmap changed)... 2024-02-19T19:04:10.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:09 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi114", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T19:04:10.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:09 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:10.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:09 smithi114 ceph-mon[92983]: Reconfiguring daemon crash.smithi114 on smithi114 2024-02-19T19:04:10.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:09 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:10.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:09 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:10.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:09 smithi174 ceph-mon[87279]: Reconfiguring crash.smithi114 (monmap changed)... 2024-02-19T19:04:10.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:09 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi114", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T19:04:10.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:09 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:10.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:09 smithi174 ceph-mon[87279]: Reconfiguring daemon crash.smithi114 on smithi114 2024-02-19T19:04:11.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:10 smithi114 ceph-mon[92983]: pgmap v18: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:11.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:10 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:04:11.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:10 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:11.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:10 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:11.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:10 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-19T19:04:11.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:10 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:11.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:10 smithi174 ceph-mon[87279]: pgmap v18: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:11.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:10 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:04:11.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:10 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:11.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:10 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:11.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:10 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-19T19:04:11.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:10 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:11.908 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:11 smithi114 ceph-mon[92983]: Reconfiguring osd.0 (monmap changed)... 2024-02-19T19:04:11.908 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:11 smithi114 ceph-mon[92983]: Reconfiguring daemon osd.0 on smithi114 2024-02-19T19:04:12.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:11 smithi174 ceph-mon[87279]: Reconfiguring osd.0 (monmap changed)... 2024-02-19T19:04:12.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:11 smithi174 ceph-mon[87279]: Reconfiguring daemon osd.0 on smithi114 2024-02-19T19:04:13.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:12 smithi114 ceph-mon[92983]: pgmap v19: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:13.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:13.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:13.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-19T19:04:13.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:13.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.076+0000 7fb7258a5700 1 -- 172.21.15.114:0/1378024429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb720101710 msgr2=0x7fb720103b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:13.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.076+0000 7fb7258a5700 1 --2- 172.21.15.114:0/1378024429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb720101710 0x7fb720103b00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb710009910 tx=0x7fb710009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:13.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.076+0000 7fb7258a5700 1 -- 172.21.15.114:0/1378024429 shutdown_connections 2024-02-19T19:04:13.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.076+0000 7fb7258a5700 1 --2- 172.21.15.114:0/1378024429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb720104040 0x7fb720106430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:13.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.076+0000 7fb7258a5700 1 --2- 172.21.15.114:0/1378024429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb720101710 0x7fb720103b00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:13.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.076+0000 7fb7258a5700 1 -- 172.21.15.114:0/1378024429 >> 172.21.15.114:0/1378024429 conn(0x7fb7200fb080 msgr2=0x7fb7200fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:13.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.076+0000 7fb7258a5700 1 -- 172.21.15.114:0/1378024429 shutdown_connections 2024-02-19T19:04:13.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.076+0000 7fb7258a5700 1 -- 172.21.15.114:0/1378024429 wait complete. 2024-02-19T19:04:13.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.077+0000 7fb7258a5700 1 Processor -- start 2024-02-19T19:04:13.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.077+0000 7fb7258a5700 1 -- start start 2024-02-19T19:04:13.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.077+0000 7fb7258a5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb720101710 0x7fb7201962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:13.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.077+0000 7fb7258a5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb720104040 0x7fb720196830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:13.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.077+0000 7fb7258a5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb720196e50 con 0x7fb720104040 2024-02-19T19:04:13.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.077+0000 7fb7258a5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb720196f90 con 0x7fb720101710 2024-02-19T19:04:13.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.077+0000 7fb71e7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb720104040 0x7fb720196830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:13.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.077+0000 7fb71e7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb720104040 0x7fb720196830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60106/0 (socket says 172.21.15.114:60106) 2024-02-19T19:04:13.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.077+0000 7fb71e7fc700 1 -- 172.21.15.114:0/2386108746 learned_addr learned my addr 172.21.15.114:0/2386108746 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:04:13.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.077+0000 7fb71effd700 1 --2- 172.21.15.114:0/2386108746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb720101710 0x7fb7201962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:13.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.077+0000 7fb71e7fc700 1 -- 172.21.15.114:0/2386108746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb720101710 msgr2=0x7fb7201962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:13.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.077+0000 7fb71e7fc700 1 --2- 172.21.15.114:0/2386108746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb720101710 0x7fb7201962f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:13.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.077+0000 7fb71e7fc700 1 -- 172.21.15.114:0/2386108746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7100095f0 con 0x7fb720104040 2024-02-19T19:04:13.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.078+0000 7fb71e7fc700 1 --2- 172.21.15.114:0/2386108746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb720104040 0x7fb720196830 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb71400d8d0 tx=0x7fb71400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:13.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.078+0000 7fb7248a3700 1 -- 172.21.15.114:0/2386108746 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7140098a0 con 0x7fb720104040 2024-02-19T19:04:13.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.078+0000 7fb7258a5700 1 -- 172.21.15.114:0/2386108746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72019ba40 con 0x7fb720104040 2024-02-19T19:04:13.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.078+0000 7fb7248a3700 1 -- 172.21.15.114:0/2386108746 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb714010460 con 0x7fb720104040 2024-02-19T19:04:13.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.078+0000 7fb7248a3700 1 -- 172.21.15.114:0/2386108746 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb71400f5f0 con 0x7fb720104040 2024-02-19T19:04:13.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.078+0000 7fb7258a5700 1 -- 172.21.15.114:0/2386108746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72019bf90 con 0x7fb720104040 2024-02-19T19:04:13.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.079+0000 7fb7258a5700 1 -- 172.21.15.114:0/2386108746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb720190560 con 0x7fb720104040 2024-02-19T19:04:13.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.079+0000 7fb7248a3700 1 -- 172.21.15.114:0/2386108746 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 36) v1 ==== 91597+0+0 (secure 0 0 0) 0x7fb714009a00 con 0x7fb720104040 2024-02-19T19:04:13.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.079+0000 7fb7248a3700 1 --2- 172.21.15.114:0/2386108746 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb70806d9d0 0x7fb70806fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:13.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.080+0000 7fb7248a3700 1 -- 172.21.15.114:0/2386108746 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb71408d630 con 0x7fb720104040 2024-02-19T19:04:13.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.080+0000 7fb71effd700 1 --2- 172.21.15.114:0/2386108746 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb70806d9d0 0x7fb70806fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:13.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.080+0000 7fb71effd700 1 --2- 172.21.15.114:0/2386108746 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb70806d9d0 0x7fb70806fe90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb710006010 tx=0x7fb71000b560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:13.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.084+0000 7fb7248a3700 1 -- 172.21.15.114:0/2386108746 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb71405a740 con 0x7fb720104040 2024-02-19T19:04:13.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:12 smithi174 ceph-mon[87279]: pgmap v19: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:13.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:13.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:13.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-19T19:04:13.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:13.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.304+0000 7fb7258a5700 1 -- 172.21.15.114:0/2386108746 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7200611d0 con 0x7fb70806d9d0 2024-02-19T19:04:13.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.305+0000 7fb7248a3700 1 -- 172.21.15.114:0/2386108746 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fb7200611d0 con 0x7fb70806d9d0 2024-02-19T19:04:13.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.307+0000 7fb7258a5700 1 -- 172.21.15.114:0/2386108746 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb70806d9d0 msgr2=0x7fb70806fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:13.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.307+0000 7fb7258a5700 1 --2- 172.21.15.114:0/2386108746 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb70806d9d0 0x7fb70806fe90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb710006010 tx=0x7fb71000b560 comp rx=0 tx=0).stop 2024-02-19T19:04:13.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.307+0000 7fb7258a5700 1 -- 172.21.15.114:0/2386108746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb720104040 msgr2=0x7fb720196830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:13.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.307+0000 7fb7258a5700 1 --2- 172.21.15.114:0/2386108746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb720104040 0x7fb720196830 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb71400d8d0 tx=0x7fb71400dbe0 comp rx=0 tx=0).stop 2024-02-19T19:04:13.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.307+0000 7fb7258a5700 1 -- 172.21.15.114:0/2386108746 shutdown_connections 2024-02-19T19:04:13.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.307+0000 7fb7258a5700 1 --2- 172.21.15.114:0/2386108746 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb70806d9d0 0x7fb70806fe90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:13.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.307+0000 7fb7258a5700 1 --2- 172.21.15.114:0/2386108746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb720101710 0x7fb7201962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:13.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.307+0000 7fb7258a5700 1 --2- 172.21.15.114:0/2386108746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb720104040 0x7fb720196830 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:13.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.307+0000 7fb7258a5700 1 -- 172.21.15.114:0/2386108746 >> 172.21.15.114:0/2386108746 conn(0x7fb7200fb080 msgr2=0x7fb7200fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:13.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.308+0000 7fb7258a5700 1 -- 172.21.15.114:0/2386108746 shutdown_connections 2024-02-19T19:04:13.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.308+0000 7fb7258a5700 1 -- 172.21.15.114:0/2386108746 wait complete. 2024-02-19T19:04:13.337 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-02-19T19:04:13.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.414+0000 7f098141b700 1 -- 172.21.15.114:0/3151051774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f097c101dd0 msgr2=0x7f097c102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:13.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.414+0000 7f098141b700 1 --2- 172.21.15.114:0/3151051774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f097c101dd0 0x7f097c102230 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0970009910 tx=0x7f0970009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:13.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.414+0000 7f098141b700 1 -- 172.21.15.114:0/3151051774 shutdown_connections 2024-02-19T19:04:13.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.414+0000 7f098141b700 1 --2- 172.21.15.114:0/3151051774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f097c101dd0 0x7f097c102230 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:13.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.414+0000 7f098141b700 1 --2- 172.21.15.114:0/3151051774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f097c100bd0 0x7f097c100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:13.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.414+0000 7f098141b700 1 -- 172.21.15.114:0/3151051774 >> 172.21.15.114:0/3151051774 conn(0x7f097c0fc190 msgr2=0x7f097c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:13.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.414+0000 7f098141b700 1 -- 172.21.15.114:0/3151051774 shutdown_connections 2024-02-19T19:04:13.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.414+0000 7f098141b700 1 -- 172.21.15.114:0/3151051774 wait complete. 2024-02-19T19:04:13.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.414+0000 7f098141b700 1 Processor -- start 2024-02-19T19:04:13.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.414+0000 7f098141b700 1 -- start start 2024-02-19T19:04:13.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.415+0000 7f098141b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f097c100bd0 0x7f097c196350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:13.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.415+0000 7f098141b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f097c101dd0 0x7f097c196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:13.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.415+0000 7f098141b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097c196e20 con 0x7f097c101dd0 2024-02-19T19:04:13.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.415+0000 7f098141b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097c196f60 con 0x7f097c100bd0 2024-02-19T19:04:13.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.415+0000 7f097a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f097c101dd0 0x7f097c196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:13.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.415+0000 7f097a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f097c101dd0 0x7f097c196890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60130/0 (socket says 172.21.15.114:60130) 2024-02-19T19:04:13.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.415+0000 7f097a7fc700 1 -- 172.21.15.114:0/1888147433 learned_addr learned my addr 172.21.15.114:0/1888147433 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:04:13.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.415+0000 7f097affd700 1 --2- 172.21.15.114:0/1888147433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f097c100bd0 0x7f097c196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:13.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.415+0000 7f097a7fc700 1 -- 172.21.15.114:0/1888147433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f097c100bd0 msgr2=0x7f097c196350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:13.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.415+0000 7f097a7fc700 1 --2- 172.21.15.114:0/1888147433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f097c100bd0 0x7f097c196350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:13.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.415+0000 7f097a7fc700 1 -- 172.21.15.114:0/1888147433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09700095f0 con 0x7f097c101dd0 2024-02-19T19:04:13.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.415+0000 7f097affd700 1 --2- 172.21.15.114:0/1888147433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f097c100bd0 0x7f097c196350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:04:13.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.415+0000 7f097a7fc700 1 --2- 172.21.15.114:0/1888147433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f097c101dd0 0x7f097c196890 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f097000bd70 tx=0x7f097000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:13.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.415+0000 7f0963fff700 1 -- 172.21.15.114:0/1888147433 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f097001d070 con 0x7f097c101dd0 2024-02-19T19:04:13.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.415+0000 7f0963fff700 1 -- 172.21.15.114:0/1888147433 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0970022470 con 0x7f097c101dd0 2024-02-19T19:04:13.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.415+0000 7f0963fff700 1 -- 172.21.15.114:0/1888147433 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f097000f670 con 0x7f097c101dd0 2024-02-19T19:04:13.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.415+0000 7f098141b700 1 -- 172.21.15.114:0/1888147433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f097c19b9c0 con 0x7f097c101dd0 2024-02-19T19:04:13.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.416+0000 7f098141b700 1 -- 172.21.15.114:0/1888147433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f097c19be80 con 0x7f097c101dd0 2024-02-19T19:04:13.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.416+0000 7f098141b700 1 -- 172.21.15.114:0/1888147433 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f097c109730 con 0x7f097c101dd0 2024-02-19T19:04:13.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.417+0000 7f0963fff700 1 -- 172.21.15.114:0/1888147433 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 36) v1 ==== 91597+0+0 (secure 0 0 0) 0x7f0970022a50 con 0x7f097c101dd0 2024-02-19T19:04:13.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.417+0000 7f0963fff700 1 --2- 172.21.15.114:0/1888147433 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f096406d9e0 0x7f096406fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:13.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.417+0000 7f0963fff700 1 -- 172.21.15.114:0/1888147433 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f097008ee90 con 0x7f097c101dd0 2024-02-19T19:04:13.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.417+0000 7f097affd700 1 --2- 172.21.15.114:0/1888147433 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f096406d9e0 0x7f096406fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:13.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.418+0000 7f097affd700 1 --2- 172.21.15.114:0/1888147433 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f096406d9e0 0x7f096406fea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f097c101c30 tx=0x7f0974009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:13.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.422+0000 7f0963fff700 1 -- 172.21.15.114:0/1888147433 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f097005c1f0 con 0x7f097c101dd0 2024-02-19T19:04:13.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.642+0000 7f098141b700 1 -- 172.21.15.114:0/1888147433 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f097c106720 con 0x7f096406d9e0 2024-02-19T19:04:13.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.643+0000 7f0963fff700 1 -- 172.21.15.114:0/1888147433 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f097c106720 con 0x7f096406d9e0 2024-02-19T19:04:13.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.645+0000 7f098141b700 1 -- 172.21.15.114:0/1888147433 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f096406d9e0 msgr2=0x7f096406fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:13.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.645+0000 7f098141b700 1 --2- 172.21.15.114:0/1888147433 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f096406d9e0 0x7f096406fea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f097c101c30 tx=0x7f0974009040 comp rx=0 tx=0).stop 2024-02-19T19:04:13.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.645+0000 7f098141b700 1 -- 172.21.15.114:0/1888147433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f097c101dd0 msgr2=0x7f097c196890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:13.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.645+0000 7f098141b700 1 --2- 172.21.15.114:0/1888147433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f097c101dd0 0x7f097c196890 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f097000bd70 tx=0x7f097000be50 comp rx=0 tx=0).stop 2024-02-19T19:04:13.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.645+0000 7f098141b700 1 -- 172.21.15.114:0/1888147433 shutdown_connections 2024-02-19T19:04:13.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.645+0000 7f098141b700 1 --2- 172.21.15.114:0/1888147433 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f096406d9e0 0x7f096406fea0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:13.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.645+0000 7f098141b700 1 --2- 172.21.15.114:0/1888147433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f097c100bd0 0x7f097c196350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:13.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.645+0000 7f098141b700 1 --2- 172.21.15.114:0/1888147433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f097c101dd0 0x7f097c196890 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:13.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.645+0000 7f098141b700 1 -- 172.21.15.114:0/1888147433 >> 172.21.15.114:0/1888147433 conn(0x7f097c0fc190 msgr2=0x7f097c105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:13.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.645+0000 7f098141b700 1 -- 172.21.15.114:0/1888147433 shutdown_connections 2024-02-19T19:04:13.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.646+0000 7f098141b700 1 -- 172.21.15.114:0/1888147433 wait complete. 2024-02-19T19:04:13.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.751+0000 7f8898a18700 1 -- 172.21.15.114:0/1457931210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8894100cd0 msgr2=0x7f8894101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:13.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.751+0000 7f8898a18700 1 --2- 172.21.15.114:0/1457931210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8894100cd0 0x7f8894101130 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f887c009910 tx=0x7f887c009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.752+0000 7f8898a18700 1 -- 172.21.15.114:0/1457931210 shutdown_connections 2024-02-19T19:04:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.752+0000 7f8898a18700 1 --2- 172.21.15.114:0/1457931210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8894100cd0 0x7f8894101130 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.752+0000 7f8898a18700 1 --2- 172.21.15.114:0/1457931210 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88940ffad0 0x7f88940ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.752+0000 7f8898a18700 1 -- 172.21.15.114:0/1457931210 >> 172.21.15.114:0/1457931210 conn(0x7f88940fb050 msgr2=0x7f88940fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.752+0000 7f8898a18700 1 -- 172.21.15.114:0/1457931210 shutdown_connections 2024-02-19T19:04:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.752+0000 7f8898a18700 1 -- 172.21.15.114:0/1457931210 wait complete. 2024-02-19T19:04:13.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.752+0000 7f8898a18700 1 Processor -- start 2024-02-19T19:04:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.752+0000 7f8898a18700 1 -- start start 2024-02-19T19:04:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.753+0000 7f8898a18700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88940ffad0 0x7f8894198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.753+0000 7f8898a18700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8894100cd0 0x7f8894198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.753+0000 7f8898a18700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88941990f0 con 0x7f8894100cd0 2024-02-19T19:04:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.753+0000 7f8898a18700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8894199230 con 0x7f88940ffad0 2024-02-19T19:04:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.753+0000 7f8889d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8894100cd0 0x7f8894198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.753+0000 7f8889d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8894100cd0 0x7f8894198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60146/0 (socket says 172.21.15.114:60146) 2024-02-19T19:04:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.753+0000 7f8889d9b700 1 -- 172.21.15.114:0/341522929 learned_addr learned my addr 172.21.15.114:0/341522929 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:04:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.753+0000 7f889259c700 1 --2- 172.21.15.114:0/341522929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88940ffad0 0x7f8894198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.753+0000 7f8889d9b700 1 -- 172.21.15.114:0/341522929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88940ffad0 msgr2=0x7f8894198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.753+0000 7f8889d9b700 1 --2- 172.21.15.114:0/341522929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88940ffad0 0x7f8894198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.753+0000 7f8889d9b700 1 -- 172.21.15.114:0/341522929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c0095f0 con 0x7f8894100cd0 2024-02-19T19:04:13.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.753+0000 7f889259c700 1 --2- 172.21.15.114:0/341522929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88940ffad0 0x7f8894198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:04:13.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.753+0000 7f8889d9b700 1 --2- 172.21.15.114:0/341522929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8894100cd0 0x7f8894198ad0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f887c006010 tx=0x7f887c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:13.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.753+0000 7f888bfff700 1 -- 172.21.15.114:0/341522929 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f887c01d070 con 0x7f8894100cd0 2024-02-19T19:04:13.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.753+0000 7f888bfff700 1 -- 172.21.15.114:0/341522929 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f887c00f460 con 0x7f8894100cd0 2024-02-19T19:04:13.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.753+0000 7f8898a18700 1 -- 172.21.15.114:0/341522929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889419dc80 con 0x7f8894100cd0 2024-02-19T19:04:13.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.753+0000 7f888bfff700 1 -- 172.21.15.114:0/341522929 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f887c017630 con 0x7f8894100cd0 2024-02-19T19:04:13.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.754+0000 7f8898a18700 1 -- 172.21.15.114:0/341522929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889419e170 con 0x7f8894100cd0 2024-02-19T19:04:13.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.754+0000 7f8898a18700 1 -- 172.21.15.114:0/341522929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8894108650 con 0x7f8894100cd0 2024-02-19T19:04:13.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.755+0000 7f888bfff700 1 -- 172.21.15.114:0/341522929 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 36) v1 ==== 91597+0+0 (secure 0 0 0) 0x7f887c00f5d0 con 0x7f8894100cd0 2024-02-19T19:04:13.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.755+0000 7f888bfff700 1 --2- 172.21.15.114:0/341522929 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f888006d9e0 0x7f888006fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:13.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.755+0000 7f889259c700 1 --2- 172.21.15.114:0/341522929 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f888006d9e0 0x7f888006fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:13.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.755+0000 7f888bfff700 1 -- 172.21.15.114:0/341522929 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f887c08ec60 con 0x7f8894100cd0 2024-02-19T19:04:13.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.756+0000 7f889259c700 1 --2- 172.21.15.114:0/341522929 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f888006d9e0 0x7f888006fea0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8884005950 tx=0x7f88840058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:13.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.760+0000 7f888bfff700 1 -- 172.21.15.114:0/341522929 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f887c05bd70 con 0x7f8894100cd0 2024-02-19T19:04:13.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.968+0000 7f8898a18700 1 -- 172.21.15.114:0/341522929 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f88940694a0 con 0x7f888006d9e0 2024-02-19T19:04:13.989 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:13 smithi114 ceph-mon[92983]: Reconfiguring osd.1 (monmap changed)... 2024-02-19T19:04:13.989 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:13 smithi114 ceph-mon[92983]: Reconfiguring daemon osd.1 on smithi114 2024-02-19T19:04:13.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.979+0000 7f888bfff700 1 -- 172.21.15.114:0/341522929 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f88940694a0 con 0x7f888006d9e0 2024-02-19T19:04:13.999 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:04:13.999 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (8m) 28s ago 10m 26.4M - 0.25.0 c8568f914cd2 30444b676ca4 2024-02-19T19:04:13.999 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (11m) 28s ago 11m 7852k - 18.2.1 94eadd990e0a 4918cd87e312 2024-02-19T19:04:13.999 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi174 smithi174 running (5m) 15s ago 8m 7780k - 18.2.1 94eadd990e0a 516c34431b93 2024-02-19T19:04:13.999 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (11m) 28s ago 11m 6660k - 18.2.1 94eadd990e0a ecfcd98907ba 2024-02-19T19:04:13.999 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi174 smithi174 running (8m) 15s ago 8m 6660k - 18.2.1 94eadd990e0a de88985f5a07 2024-02-19T19:04:13.999 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (8m) 28s ago 9m 85.7M - 9.4.7 954c08fa6188 478bd533e827 2024-02-19T19:04:14.000 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tcwpod smithi114 running (4m) 28s ago 4m 26.7M - 18.2.1 94eadd990e0a 420175777cf4 2024-02-19T19:04:14.000 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.vpoeop smithi114 running (4m) 28s ago 4m 22.1M - 18.2.1 94eadd990e0a f81a79e53ead 2024-02-19T19:04:14.000 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.stnfnf smithi174 running (3m) 15s ago 3m 24.0M - 18.2.1 94eadd990e0a 3df35954a492 2024-02-19T19:04:14.000 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.znhuvd smithi174 running (4m) 15s ago 4m 22.7M - 18.2.1 94eadd990e0a 2d2267494ae1 2024-02-19T19:04:14.000 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.pcvtrk smithi114 *:8443,9283,8765 running (89s) 28s ago 12m 470M - 19.0.0-1450-gf78a58c0 f081f847070a 05b5c06122b1 2024-02-19T19:04:14.000 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi174.lydvnj smithi174 *:8443,9283,8765 running (56s) 15s ago 8m 450M - 19.0.0-1450-gf78a58c0 f081f847070a f962b1af5aed 2024-02-19T19:04:14.000 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (43s) 28s ago 12m 29.8M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e51f650f9be9 2024-02-19T19:04:14.000 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi174 smithi174 running (19s) 15s ago 8m 23.8M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e811581f05fb 2024-02-19T19:04:14.000 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (11m) 28s ago 11m 15.0M - 1.5.0 0da6a335fe13 b6b258074cce 2024-02-19T19:04:14.000 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi174 smithi174 *:9100 running (8m) 15s ago 8m 14.7M - 1.5.0 0da6a335fe13 d0b4fdf67b22 2024-02-19T19:04:14.000 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (7m) 28s ago 7m 48.9M 1216M 18.2.1 94eadd990e0a 990afc982b6c 2024-02-19T19:04:14.000 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (7m) 28s ago 7m 49.3M 1216M 18.2.1 94eadd990e0a 932e3ad37db5 2024-02-19T19:04:14.001 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (6m) 28s ago 6m 51.3M 1216M 18.2.1 94eadd990e0a 12953988af86 2024-02-19T19:04:14.001 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi174 running (6m) 15s ago 6m 48.9M 2240M 18.2.1 94eadd990e0a 979f45cabbce 2024-02-19T19:04:14.001 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi174 running (6m) 15s ago 6m 49.5M 2240M 18.2.1 94eadd990e0a 8b8c3745b8c5 2024-02-19T19:04:14.001 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi174 running (5m) 15s ago 5m 51.9M 2240M 18.2.1 94eadd990e0a 5771a995ed75 2024-02-19T19:04:14.001 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (62s) 28s ago 9m 45.0M - 2.43.0 a07b618ecd1d a36922c7c9a1 2024-02-19T19:04:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.982+0000 7f8898a18700 1 -- 172.21.15.114:0/341522929 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f888006d9e0 msgr2=0x7f888006fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.982+0000 7f8898a18700 1 --2- 172.21.15.114:0/341522929 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f888006d9e0 0x7f888006fea0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8884005950 tx=0x7f88840058e0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.982+0000 7f8898a18700 1 -- 172.21.15.114:0/341522929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8894100cd0 msgr2=0x7f8894198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.982+0000 7f8898a18700 1 --2- 172.21.15.114:0/341522929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8894100cd0 0x7f8894198ad0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f887c006010 tx=0x7f887c00be20 comp rx=0 tx=0).stop 2024-02-19T19:04:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.982+0000 7f8898a18700 1 -- 172.21.15.114:0/341522929 shutdown_connections 2024-02-19T19:04:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.982+0000 7f8898a18700 1 --2- 172.21.15.114:0/341522929 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f888006d9e0 0x7f888006fea0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.982+0000 7f8898a18700 1 --2- 172.21.15.114:0/341522929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88940ffad0 0x7f8894198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.982+0000 7f8898a18700 1 --2- 172.21.15.114:0/341522929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8894100cd0 0x7f8894198ad0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.982+0000 7f8898a18700 1 -- 172.21.15.114:0/341522929 >> 172.21.15.114:0/341522929 conn(0x7f88940fb050 msgr2=0x7f8894068d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.982+0000 7f8898a18700 1 -- 172.21.15.114:0/341522929 shutdown_connections 2024-02-19T19:04:14.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:13.982+0000 7f8898a18700 1 -- 172.21.15.114:0/341522929 wait complete. 2024-02-19T19:04:14.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.086+0000 7f56c2bdd700 1 -- 172.21.15.114:0/2444965265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc100bc0 msgr2=0x7f56bc100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:14.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.086+0000 7f56c2bdd700 1 --2- 172.21.15.114:0/2444965265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc100bc0 0x7f56bc100fe0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009910 tx=0x7f56b0009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:14.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.086+0000 7f56c2bdd700 1 -- 172.21.15.114:0/2444965265 shutdown_connections 2024-02-19T19:04:14.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.086+0000 7f56c2bdd700 1 --2- 172.21.15.114:0/2444965265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56bc101dc0 0x7f56bc102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.086+0000 7f56c2bdd700 1 --2- 172.21.15.114:0/2444965265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc100bc0 0x7f56bc100fe0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.086+0000 7f56c2bdd700 1 -- 172.21.15.114:0/2444965265 >> 172.21.15.114:0/2444965265 conn(0x7f56bc0fc140 msgr2=0x7f56bc0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:14.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.086+0000 7f56c2bdd700 1 -- 172.21.15.114:0/2444965265 shutdown_connections 2024-02-19T19:04:14.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.086+0000 7f56c2bdd700 1 -- 172.21.15.114:0/2444965265 wait complete. 2024-02-19T19:04:14.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.087+0000 7f56c2bdd700 1 Processor -- start 2024-02-19T19:04:14.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.087+0000 7f56c2bdd700 1 -- start start 2024-02-19T19:04:14.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.087+0000 7f56c2bdd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56bc100bc0 0x7f56bc074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:14.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.087+0000 7f56c2bdd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc101dc0 0x7f56bc073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:14.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.087+0000 7f56c2bdd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc0736c0 con 0x7f56bc101dc0 2024-02-19T19:04:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.087+0000 7f56c2bdd700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc073800 con 0x7f56bc100bc0 2024-02-19T19:04:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.087+0000 7f56bbfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc101dc0 0x7f56bc073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.087+0000 7f56bbfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc101dc0 0x7f56bc073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60158/0 (socket says 172.21.15.114:60158) 2024-02-19T19:04:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.087+0000 7f56bbfff700 1 -- 172.21.15.114:0/558209544 learned_addr learned my addr 172.21.15.114:0/558209544 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:04:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.087+0000 7f56c0979700 1 --2- 172.21.15.114:0/558209544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56bc100bc0 0x7f56bc074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.087+0000 7f56bbfff700 1 -- 172.21.15.114:0/558209544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56bc100bc0 msgr2=0x7f56bc074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.087+0000 7f56bbfff700 1 --2- 172.21.15.114:0/558209544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56bc100bc0 0x7f56bc074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.087+0000 7f56bbfff700 1 -- 172.21.15.114:0/558209544 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b00095f0 con 0x7f56bc101dc0 2024-02-19T19:04:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.088+0000 7f56bbfff700 1 --2- 172.21.15.114:0/558209544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc101dc0 0x7f56bc073180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f56a800d8d0 tx=0x7f56a800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.088+0000 7f56b9ffb700 1 -- 172.21.15.114:0/558209544 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56a8009940 con 0x7f56bc101dc0 2024-02-19T19:04:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.088+0000 7f56b9ffb700 1 -- 172.21.15.114:0/558209544 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f56a8010460 con 0x7f56bc101dc0 2024-02-19T19:04:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.088+0000 7f56c2bdd700 1 -- 172.21.15.114:0/558209544 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56bc073ae0 con 0x7f56bc101dc0 2024-02-19T19:04:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.088+0000 7f56b9ffb700 1 -- 172.21.15.114:0/558209544 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56a800f5f0 con 0x7f56bc101dc0 2024-02-19T19:04:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.088+0000 7f56c2bdd700 1 -- 172.21.15.114:0/558209544 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56bc19e3f0 con 0x7f56bc101dc0 2024-02-19T19:04:14.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.089+0000 7f56c2bdd700 1 -- 172.21.15.114:0/558209544 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56bc109720 con 0x7f56bc101dc0 2024-02-19T19:04:14.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.089+0000 7f56b9ffb700 1 -- 172.21.15.114:0/558209544 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 36) v1 ==== 91597+0+0 (secure 0 0 0) 0x7f56a80105d0 con 0x7f56bc101dc0 2024-02-19T19:04:14.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.090+0000 7f56b9ffb700 1 --2- 172.21.15.114:0/558209544 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f56ac06d9e0 0x7f56ac06fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:14.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.090+0000 7f56b9ffb700 1 -- 172.21.15.114:0/558209544 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f56a808d660 con 0x7f56bc101dc0 2024-02-19T19:04:14.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.090+0000 7f56c0979700 1 --2- 172.21.15.114:0/558209544 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f56ac06d9e0 0x7f56ac06fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:14.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.090+0000 7f56c0979700 1 --2- 172.21.15.114:0/558209544 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f56ac06d9e0 0x7f56ac06fea0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f56b0006010 tx=0x7f56b0005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:14.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.094+0000 7f56b9ffb700 1 -- 172.21.15.114:0/558209544 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f56a805a7f0 con 0x7f56bc101dc0 2024-02-19T19:04:14.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:13 smithi174 ceph-mon[87279]: Reconfiguring osd.1 (monmap changed)... 2024-02-19T19:04:14.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:13 smithi174 ceph-mon[87279]: Reconfiguring daemon osd.1 on smithi114 2024-02-19T19:04:14.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.383+0000 7f56c2bdd700 1 -- 172.21.15.114:0/558209544 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f56bc1098e0 con 0x7f56bc101dc0 2024-02-19T19:04:14.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.383+0000 7f56b9ffb700 1 -- 172.21.15.114:0/558209544 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+688 (secure 0 0 0) 0x7f56a805a380 con 0x7f56bc101dc0 2024-02-19T19:04:14.403 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:04:14.403 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-02-19T19:04:14.404 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:04:14.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:04:14.404 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-02-19T19:04:14.404 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:04:14.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:04:14.404 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-02-19T19:04:14.404 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-02-19T19:04:14.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:04:14.404 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-02-19T19:04:14.404 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:04:14.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:04:14.404 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-02-19T19:04:14.404 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 10, 2024-02-19T19:04:14.405 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 4 2024-02-19T19:04:14.405 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-02-19T19:04:14.405 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:04:14.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.385+0000 7f56c2bdd700 1 -- 172.21.15.114:0/558209544 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f56ac06d9e0 msgr2=0x7f56ac06fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:14.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.385+0000 7f56c2bdd700 1 --2- 172.21.15.114:0/558209544 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f56ac06d9e0 0x7f56ac06fea0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f56b0006010 tx=0x7f56b0005a90 comp rx=0 tx=0).stop 2024-02-19T19:04:14.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.386+0000 7f56c2bdd700 1 -- 172.21.15.114:0/558209544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc101dc0 msgr2=0x7f56bc073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:14.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.386+0000 7f56c2bdd700 1 --2- 172.21.15.114:0/558209544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc101dc0 0x7f56bc073180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f56a800d8d0 tx=0x7f56a800dc90 comp rx=0 tx=0).stop 2024-02-19T19:04:14.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.386+0000 7f56c2bdd700 1 -- 172.21.15.114:0/558209544 shutdown_connections 2024-02-19T19:04:14.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.386+0000 7f56c2bdd700 1 --2- 172.21.15.114:0/558209544 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f56ac06d9e0 0x7f56ac06fea0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.386+0000 7f56c2bdd700 1 --2- 172.21.15.114:0/558209544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56bc100bc0 0x7f56bc074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.386+0000 7f56c2bdd700 1 --2- 172.21.15.114:0/558209544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56bc101dc0 0x7f56bc073180 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.386+0000 7f56c2bdd700 1 -- 172.21.15.114:0/558209544 >> 172.21.15.114:0/558209544 conn(0x7f56bc0fc140 msgr2=0x7f56bc104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:14.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.386+0000 7f56c2bdd700 1 -- 172.21.15.114:0/558209544 shutdown_connections 2024-02-19T19:04:14.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.386+0000 7f56c2bdd700 1 -- 172.21.15.114:0/558209544 wait complete. 2024-02-19T19:04:14.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.494+0000 7f43b8962700 1 -- 172.21.15.114:0/3084619974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43b4101dd0 msgr2=0x7f43b4102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:14.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.494+0000 7f43b8962700 1 --2- 172.21.15.114:0/3084619974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43b4101dd0 0x7f43b4102230 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f43a4009910 tx=0x7f43a4009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:14.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.494+0000 7f43b8962700 1 -- 172.21.15.114:0/3084619974 shutdown_connections 2024-02-19T19:04:14.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.494+0000 7f43b8962700 1 --2- 172.21.15.114:0/3084619974 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43b4101dd0 0x7f43b4102230 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.494+0000 7f43b8962700 1 --2- 172.21.15.114:0/3084619974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b4100bd0 0x7f43b4100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.494+0000 7f43b8962700 1 -- 172.21.15.114:0/3084619974 >> 172.21.15.114:0/3084619974 conn(0x7f43b40fc190 msgr2=0x7f43b40fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:14.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.494+0000 7f43b8962700 1 -- 172.21.15.114:0/3084619974 shutdown_connections 2024-02-19T19:04:14.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.494+0000 7f43b8962700 1 -- 172.21.15.114:0/3084619974 wait complete. 2024-02-19T19:04:14.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.495+0000 7f43b8962700 1 Processor -- start 2024-02-19T19:04:14.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.495+0000 7f43b8962700 1 -- start start 2024-02-19T19:04:14.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.495+0000 7f43b8962700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b4100bd0 0x7f43b4196350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:14.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.495+0000 7f43b8962700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43b4101dd0 0x7f43b4196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:14.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.496+0000 7f43b8962700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b4196eb0 con 0x7f43b4101dd0 2024-02-19T19:04:14.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.496+0000 7f43b8962700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b4196ff0 con 0x7f43b4100bd0 2024-02-19T19:04:14.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.496+0000 7f43b1d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43b4101dd0 0x7f43b4196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:14.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.496+0000 7f43b1d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43b4101dd0 0x7f43b4196890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60184/0 (socket says 172.21.15.114:60184) 2024-02-19T19:04:14.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.496+0000 7f43b1d9b700 1 -- 172.21.15.114:0/2226272788 learned_addr learned my addr 172.21.15.114:0/2226272788 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:04:14.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.496+0000 7f43b259c700 1 --2- 172.21.15.114:0/2226272788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b4100bd0 0x7f43b4196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:14.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.496+0000 7f43b1d9b700 1 -- 172.21.15.114:0/2226272788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b4100bd0 msgr2=0x7f43b4196350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:14.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.496+0000 7f43b1d9b700 1 --2- 172.21.15.114:0/2226272788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b4100bd0 0x7f43b4196350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.496+0000 7f43b1d9b700 1 -- 172.21.15.114:0/2226272788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a40095f0 con 0x7f43b4101dd0 2024-02-19T19:04:14.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.496+0000 7f43b259c700 1 --2- 172.21.15.114:0/2226272788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b4100bd0 0x7f43b4196350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:04:14.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.496+0000 7f43b1d9b700 1 --2- 172.21.15.114:0/2226272788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43b4101dd0 0x7f43b4196890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f43a4006010 tx=0x7f43a400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:14.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.496+0000 7f43a37fe700 1 -- 172.21.15.114:0/2226272788 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43a401d070 con 0x7f43b4101dd0 2024-02-19T19:04:14.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.496+0000 7f43a37fe700 1 -- 172.21.15.114:0/2226272788 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f43a400f460 con 0x7f43b4101dd0 2024-02-19T19:04:14.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.496+0000 7f43a37fe700 1 -- 172.21.15.114:0/2226272788 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43a4017630 con 0x7f43b4101dd0 2024-02-19T19:04:14.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.497+0000 7f43b8962700 1 -- 172.21.15.114:0/2226272788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b4072fc0 con 0x7f43b4101dd0 2024-02-19T19:04:14.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.497+0000 7f43b8962700 1 -- 172.21.15.114:0/2226272788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b4073460 con 0x7f43b4101dd0 2024-02-19T19:04:14.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.498+0000 7f43b8962700 1 -- 172.21.15.114:0/2226272788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b4109730 con 0x7f43b4101dd0 2024-02-19T19:04:14.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.498+0000 7f43a37fe700 1 -- 172.21.15.114:0/2226272788 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 36) v1 ==== 91597+0+0 (secure 0 0 0) 0x7f43a400f5d0 con 0x7f43b4101dd0 2024-02-19T19:04:14.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.498+0000 7f43a37fe700 1 --2- 172.21.15.114:0/2226272788 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f439c06d9e0 0x7f439c06fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:14.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.498+0000 7f43a37fe700 1 -- 172.21.15.114:0/2226272788 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f43a408ec90 con 0x7f43b4101dd0 2024-02-19T19:04:14.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.499+0000 7f43b259c700 1 --2- 172.21.15.114:0/2226272788 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f439c06d9e0 0x7f439c06fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:14.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.499+0000 7f43b259c700 1 --2- 172.21.15.114:0/2226272788 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f439c06d9e0 0x7f439c06fea0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f43a8005950 tx=0x7f43a80058e0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:14.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.503+0000 7f43a37fe700 1 -- 172.21.15.114:0/2226272788 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f43a405bda0 con 0x7f43b4101dd0 2024-02-19T19:04:14.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.782+0000 7f43b8962700 1 -- 172.21.15.114:0/2226272788 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f43b4073af0 con 0x7f43b4101dd0 2024-02-19T19:04:14.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.783+0000 7f43a37fe700 1 -- 172.21.15.114:0/2226272788 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1871 (secure 0 0 0) 0x7f43a405b930 con 0x7f43b4101dd0 2024-02-19T19:04:14.803 INFO:teuthology.orchestra.run.smithi114.stdout:e18 2024-02-19T19:04:14.803 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:04:14.803 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:04:14.803 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-02-19T19:04:14.803 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:04:14.803 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:04:14.803 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-02-19T19:04:14.803 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 18 2024-02-19T19:04:14.804 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:04:14.804 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:04:14.804 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-02-19T19:00:21.203922+0000 2024-02-19T19:04:14.804 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-02-19T19:04:14.804 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-02-19T19:04:14.804 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-02-19T19:04:14.804 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-02-19T19:04:14.804 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-02-19T19:04:14.804 INFO:teuthology.orchestra.run.smithi114.stdout:max_xattr_size 65536 2024-02-19T19:04:14.804 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-02-19T19:04:14.805 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-02-19T19:04:14.805 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 47 2024-02-19T19:04:14.805 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:04:14.805 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 2 2024-02-19T19:04:14.805 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1 2024-02-19T19:04:14.805 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=14530,1=24269} 2024-02-19T19:04:14.805 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-02-19T19:04:14.805 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-02-19T19:04:14.805 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-02-19T19:04:14.805 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-02-19T19:04:14.805 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-02-19T19:04:14.806 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data enabled 2024-02-19T19:04:14.806 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-02-19T19:04:14.806 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-02-19T19:04:14.806 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 1 2024-02-19T19:04:14.806 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:04:14.806 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.vpoeop{0:14540} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:04:14.806 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.znhuvd{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:04:14.806 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.stnfnf{1:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:04:14.806 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:04:14.807 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:04:14.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.786+0000 7f43b8962700 1 -- 172.21.15.114:0/2226272788 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f439c06d9e0 msgr2=0x7f439c06fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:14.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.786+0000 7f43b8962700 1 --2- 172.21.15.114:0/2226272788 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f439c06d9e0 0x7f439c06fea0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f43a8005950 tx=0x7f43a80058e0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.786+0000 7f43b8962700 1 -- 172.21.15.114:0/2226272788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43b4101dd0 msgr2=0x7f43b4196890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:14.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.786+0000 7f43b8962700 1 --2- 172.21.15.114:0/2226272788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43b4101dd0 0x7f43b4196890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f43a4006010 tx=0x7f43a400be20 comp rx=0 tx=0).stop 2024-02-19T19:04:14.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.786+0000 7f43b8962700 1 -- 172.21.15.114:0/2226272788 shutdown_connections 2024-02-19T19:04:14.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.786+0000 7f43b8962700 1 --2- 172.21.15.114:0/2226272788 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f439c06d9e0 0x7f439c06fea0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.786+0000 7f43b8962700 1 --2- 172.21.15.114:0/2226272788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b4100bd0 0x7f43b4196350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.786+0000 7f43b8962700 1 --2- 172.21.15.114:0/2226272788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43b4101dd0 0x7f43b4196890 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.786+0000 7f43b8962700 1 -- 172.21.15.114:0/2226272788 >> 172.21.15.114:0/2226272788 conn(0x7f43b40fc190 msgr2=0x7f43b4105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:14.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.786+0000 7f43b8962700 1 -- 172.21.15.114:0/2226272788 shutdown_connections 2024-02-19T19:04:14.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.786+0000 7f43b8962700 1 -- 172.21.15.114:0/2226272788 wait complete. 2024-02-19T19:04:14.808 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 18 2024-02-19T19:04:14.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.893+0000 7fd367e9e700 1 -- 172.21.15.114:0/2922839499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd360103fd0 msgr2=0x7fd3601063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:14.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.893+0000 7fd367e9e700 1 --2- 172.21.15.114:0/2922839499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd360103fd0 0x7fd3601063c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd350009910 tx=0x7fd350009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:14.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.893+0000 7fd367e9e700 1 -- 172.21.15.114:0/2922839499 shutdown_connections 2024-02-19T19:04:14.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.893+0000 7fd367e9e700 1 --2- 172.21.15.114:0/2922839499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd360103fd0 0x7fd3601063c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.893+0000 7fd367e9e700 1 --2- 172.21.15.114:0/2922839499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3601016a0 0x7fd360103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.893+0000 7fd367e9e700 1 -- 172.21.15.114:0/2922839499 >> 172.21.15.114:0/2922839499 conn(0x7fd3600fb030 msgr2=0x7fd3600fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:14.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.893+0000 7fd367e9e700 1 -- 172.21.15.114:0/2922839499 shutdown_connections 2024-02-19T19:04:14.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.893+0000 7fd367e9e700 1 -- 172.21.15.114:0/2922839499 wait complete. 2024-02-19T19:04:14.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.894+0000 7fd367e9e700 1 Processor -- start 2024-02-19T19:04:14.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.894+0000 7fd367e9e700 1 -- start start 2024-02-19T19:04:14.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.894+0000 7fd367e9e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3601016a0 0x7fd360196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:14.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.894+0000 7fd367e9e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd360103fd0 0x7fd360196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:14.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.894+0000 7fd367e9e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd360196e90 con 0x7fd3601016a0 2024-02-19T19:04:14.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.894+0000 7fd367e9e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd360196fd0 con 0x7fd360103fd0 2024-02-19T19:04:14.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.894+0000 7fd365c3a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3601016a0 0x7fd360196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:14.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.894+0000 7fd365c3a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3601016a0 0x7fd360196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60208/0 (socket says 172.21.15.114:60208) 2024-02-19T19:04:14.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.894+0000 7fd365c3a700 1 -- 172.21.15.114:0/322830778 learned_addr learned my addr 172.21.15.114:0/322830778 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:04:14.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.894+0000 7fd365439700 1 --2- 172.21.15.114:0/322830778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd360103fd0 0x7fd360196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:14.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.895+0000 7fd365c3a700 1 -- 172.21.15.114:0/322830778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd360103fd0 msgr2=0x7fd360196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:14.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.895+0000 7fd365c3a700 1 --2- 172.21.15.114:0/322830778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd360103fd0 0x7fd360196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:14.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.895+0000 7fd365c3a700 1 -- 172.21.15.114:0/322830778 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3500095f0 con 0x7fd3601016a0 2024-02-19T19:04:14.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.895+0000 7fd365c3a700 1 --2- 172.21.15.114:0/322830778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3601016a0 0x7fd360196330 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00e920 tx=0x7fd35c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:14.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.895+0000 7fd356ffd700 1 -- 172.21.15.114:0/322830778 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd35c00c9d0 con 0x7fd3601016a0 2024-02-19T19:04:14.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.895+0000 7fd356ffd700 1 -- 172.21.15.114:0/322830778 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd35c010430 con 0x7fd3601016a0 2024-02-19T19:04:14.916 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.895+0000 7fd367e9e700 1 -- 172.21.15.114:0/322830778 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd36019ba80 con 0x7fd3601016a0 2024-02-19T19:04:14.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.895+0000 7fd356ffd700 1 -- 172.21.15.114:0/322830778 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd35c018650 con 0x7fd3601016a0 2024-02-19T19:04:14.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.895+0000 7fd367e9e700 1 -- 172.21.15.114:0/322830778 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd36019bfd0 con 0x7fd3601016a0 2024-02-19T19:04:14.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.896+0000 7fd367e9e700 1 -- 172.21.15.114:0/322830778 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd360190520 con 0x7fd3601016a0 2024-02-19T19:04:14.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.896+0000 7fd356ffd700 1 -- 172.21.15.114:0/322830778 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 36) v1 ==== 91597+0+0 (secure 0 0 0) 0x7fd35c00cb30 con 0x7fd3601016a0 2024-02-19T19:04:14.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.897+0000 7fd356ffd700 1 --2- 172.21.15.114:0/322830778 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fd34c06d9e0 0x7fd34c06fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:14.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.897+0000 7fd356ffd700 1 -- 172.21.15.114:0/322830778 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fd35c014070 con 0x7fd3601016a0 2024-02-19T19:04:14.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.897+0000 7fd365439700 1 --2- 172.21.15.114:0/322830778 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fd34c06d9e0 0x7fd34c06fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:14.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.897+0000 7fd365439700 1 --2- 172.21.15.114:0/322830778 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fd34c06d9e0 0x7fd34c06fea0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd350009fd0 tx=0x7fd350005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:14.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:14.902+0000 7fd356ffd700 1 -- 172.21.15.114:0/322830778 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd35c052d20 con 0x7fd3601016a0 2024-02-19T19:04:15.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:14 smithi114 ceph-mon[92983]: from='client.34156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:15.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:14 smithi114 ceph-mon[92983]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:15.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:14 smithi114 ceph-mon[92983]: pgmap v20: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:15.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:15.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:15.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-19T19:04:15.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:15.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:14 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/558209544' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:04:15.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:14 smithi174 ceph-mon[87279]: from='client.34156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:15.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:14 smithi174 ceph-mon[87279]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:15.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:14 smithi174 ceph-mon[87279]: pgmap v20: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:15.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:15.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:15.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-19T19:04:15.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:15.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:14 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/558209544' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:04:15.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.130+0000 7fd367e9e700 1 -- 172.21.15.114:0/322830778 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3600611d0 con 0x7fd34c06d9e0 2024-02-19T19:04:15.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.131+0000 7fd356ffd700 1 -- 172.21.15.114:0/322830778 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fd3600611d0 con 0x7fd34c06d9e0 2024-02-19T19:04:15.151 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:04:15.151 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:04:15.151 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-02-19T19:04:15.151 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:04:15.151 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [ 2024-02-19T19:04:15.151 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr", 2024-02-19T19:04:15.151 INFO:teuthology.orchestra.run.smithi114.stdout: "mon" 2024-02-19T19:04:15.151 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-02-19T19:04:15.152 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "4/23 daemons upgraded", 2024-02-19T19:04:15.152 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "Currently upgrading mon daemons", 2024-02-19T19:04:15.152 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-02-19T19:04:15.152 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:04:15.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.133+0000 7fd367e9e700 1 -- 172.21.15.114:0/322830778 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fd34c06d9e0 msgr2=0x7fd34c06fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:15.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.133+0000 7fd367e9e700 1 --2- 172.21.15.114:0/322830778 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fd34c06d9e0 0x7fd34c06fea0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd350009fd0 tx=0x7fd350005940 comp rx=0 tx=0).stop 2024-02-19T19:04:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.133+0000 7fd367e9e700 1 -- 172.21.15.114:0/322830778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3601016a0 msgr2=0x7fd360196330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.134+0000 7fd367e9e700 1 --2- 172.21.15.114:0/322830778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3601016a0 0x7fd360196330 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd35c00e920 tx=0x7fd35c00ec30 comp rx=0 tx=0).stop 2024-02-19T19:04:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.134+0000 7fd367e9e700 1 -- 172.21.15.114:0/322830778 shutdown_connections 2024-02-19T19:04:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.134+0000 7fd367e9e700 1 --2- 172.21.15.114:0/322830778 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fd34c06d9e0 0x7fd34c06fea0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.134+0000 7fd367e9e700 1 --2- 172.21.15.114:0/322830778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3601016a0 0x7fd360196330 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.134+0000 7fd367e9e700 1 --2- 172.21.15.114:0/322830778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd360103fd0 0x7fd360196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.134+0000 7fd367e9e700 1 -- 172.21.15.114:0/322830778 >> 172.21.15.114:0/322830778 conn(0x7fd3600fb030 msgr2=0x7fd3600fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.134+0000 7fd367e9e700 1 -- 172.21.15.114:0/322830778 shutdown_connections 2024-02-19T19:04:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.134+0000 7fd367e9e700 1 -- 172.21.15.114:0/322830778 wait complete. 2024-02-19T19:04:15.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.239+0000 7f4fe2c8a700 1 -- 172.21.15.114:0/2670403357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fdc100be0 msgr2=0x7f4fdc101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:15.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.239+0000 7f4fe2c8a700 1 --2- 172.21.15.114:0/2670403357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fdc100be0 0x7f4fdc101000 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0009910 tx=0x7f4fd0009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:15.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.239+0000 7f4fe2c8a700 1 -- 172.21.15.114:0/2670403357 shutdown_connections 2024-02-19T19:04:15.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.239+0000 7f4fe2c8a700 1 --2- 172.21.15.114:0/2670403357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fdc101de0 0x7f4fdc102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:15.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.239+0000 7f4fe2c8a700 1 --2- 172.21.15.114:0/2670403357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fdc100be0 0x7f4fdc101000 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:15.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.239+0000 7f4fe2c8a700 1 -- 172.21.15.114:0/2670403357 >> 172.21.15.114:0/2670403357 conn(0x7f4fdc0fc140 msgr2=0x7f4fdc0fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:15.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.239+0000 7f4fe2c8a700 1 -- 172.21.15.114:0/2670403357 shutdown_connections 2024-02-19T19:04:15.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.239+0000 7f4fe2c8a700 1 -- 172.21.15.114:0/2670403357 wait complete. 2024-02-19T19:04:15.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.240+0000 7f4fe2c8a700 1 Processor -- start 2024-02-19T19:04:15.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.240+0000 7f4fe2c8a700 1 -- start start 2024-02-19T19:04:15.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.240+0000 7f4fe2c8a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fdc100be0 0x7f4fdc196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:15.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.240+0000 7f4fe2c8a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fdc101de0 0x7f4fdc196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:15.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.240+0000 7f4fe2c8a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc196f70 con 0x7f4fdc100be0 2024-02-19T19:04:15.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.240+0000 7f4fe2c8a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc1970b0 con 0x7f4fdc101de0 2024-02-19T19:04:15.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.240+0000 7f4fe0a26700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fdc100be0 0x7f4fdc196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:15.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.240+0000 7f4fe0a26700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fdc100be0 0x7f4fdc196410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60226/0 (socket says 172.21.15.114:60226) 2024-02-19T19:04:15.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.240+0000 7f4fe0a26700 1 -- 172.21.15.114:0/2337469935 learned_addr learned my addr 172.21.15.114:0/2337469935 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:04:15.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.240+0000 7f4fdbfff700 1 --2- 172.21.15.114:0/2337469935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fdc101de0 0x7f4fdc196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:15.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.240+0000 7f4fe0a26700 1 -- 172.21.15.114:0/2337469935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fdc101de0 msgr2=0x7f4fdc196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:15.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.240+0000 7f4fe0a26700 1 --2- 172.21.15.114:0/2337469935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fdc101de0 0x7f4fdc196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:15.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.240+0000 7f4fe0a26700 1 -- 172.21.15.114:0/2337469935 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd00095f0 con 0x7f4fdc100be0 2024-02-19T19:04:15.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.240+0000 7f4fe0a26700 1 --2- 172.21.15.114:0/2337469935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fdc100be0 0x7f4fdc196410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000bda0 tx=0x7f4fd000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:15.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.241+0000 7f4fd9ffb700 1 -- 172.21.15.114:0/2337469935 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fd001d070 con 0x7f4fdc100be0 2024-02-19T19:04:15.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.241+0000 7f4fd9ffb700 1 -- 172.21.15.114:0/2337469935 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4fd0004c50 con 0x7f4fdc100be0 2024-02-19T19:04:15.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.241+0000 7f4fe2c8a700 1 -- 172.21.15.114:0/2337469935 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fdc19bb00 con 0x7f4fdc100be0 2024-02-19T19:04:15.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.241+0000 7f4fd9ffb700 1 -- 172.21.15.114:0/2337469935 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fd000f690 con 0x7f4fdc100be0 2024-02-19T19:04:15.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.241+0000 7f4fe2c8a700 1 -- 172.21.15.114:0/2337469935 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fdc19c070 con 0x7f4fdc100be0 2024-02-19T19:04:15.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.242+0000 7f4fe2c8a700 1 -- 172.21.15.114:0/2337469935 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fdc1097c0 con 0x7f4fdc100be0 2024-02-19T19:04:15.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.242+0000 7f4fd9ffb700 1 -- 172.21.15.114:0/2337469935 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 36) v1 ==== 91597+0+0 (secure 0 0 0) 0x7f4fd000f7f0 con 0x7f4fdc100be0 2024-02-19T19:04:15.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.243+0000 7f4fd9ffb700 1 --2- 172.21.15.114:0/2337469935 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f4fcc06d9d0 0x7f4fcc06fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:15.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.243+0000 7f4fd9ffb700 1 -- 172.21.15.114:0/2337469935 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f4fd008eed0 con 0x7f4fdc100be0 2024-02-19T19:04:15.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.243+0000 7f4fdbfff700 1 --2- 172.21.15.114:0/2337469935 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f4fcc06d9d0 0x7f4fcc06fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:15.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.243+0000 7f4fdbfff700 1 --2- 172.21.15.114:0/2337469935 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f4fcc06d9d0 0x7f4fcc06fe90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8006fd0 tx=0x7f4fc8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:15.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.248+0000 7f4fd9ffb700 1 -- 172.21.15.114:0/2337469935 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4fd005c060 con 0x7f4fdc100be0 2024-02-19T19:04:15.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.531+0000 7f4fe2c8a700 1 -- 172.21.15.114:0/2337469935 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4fdc19c700 con 0x7f4fdc100be0 2024-02-19T19:04:15.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.531+0000 7f4fd9ffb700 1 -- 172.21.15.114:0/2337469935 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f4fd0027090 con 0x7f4fdc100be0 2024-02-19T19:04:15.551 INFO:teuthology.orchestra.run.smithi114.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:04:15.551 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:04:15.551 INFO:teuthology.orchestra.run.smithi114.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:04:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.533+0000 7f4fe2c8a700 1 -- 172.21.15.114:0/2337469935 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f4fcc06d9d0 msgr2=0x7f4fcc06fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.533+0000 7f4fe2c8a700 1 --2- 172.21.15.114:0/2337469935 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f4fcc06d9d0 0x7f4fcc06fe90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8006fd0 tx=0x7f4fc8008040 comp rx=0 tx=0).stop 2024-02-19T19:04:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.533+0000 7f4fe2c8a700 1 -- 172.21.15.114:0/2337469935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fdc100be0 msgr2=0x7f4fdc196410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.533+0000 7f4fe2c8a700 1 --2- 172.21.15.114:0/2337469935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fdc100be0 0x7f4fdc196410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000bda0 tx=0x7f4fd000be80 comp rx=0 tx=0).stop 2024-02-19T19:04:15.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.533+0000 7f4fe2c8a700 1 -- 172.21.15.114:0/2337469935 shutdown_connections 2024-02-19T19:04:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.533+0000 7f4fe2c8a700 1 --2- 172.21.15.114:0/2337469935 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f4fcc06d9d0 0x7f4fcc06fe90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.533+0000 7f4fe2c8a700 1 --2- 172.21.15.114:0/2337469935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fdc100be0 0x7f4fdc196410 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.533+0000 7f4fe2c8a700 1 --2- 172.21.15.114:0/2337469935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fdc101de0 0x7f4fdc196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.533+0000 7f4fe2c8a700 1 -- 172.21.15.114:0/2337469935 >> 172.21.15.114:0/2337469935 conn(0x7f4fdc0fc140 msgr2=0x7f4fdc105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.534+0000 7f4fe2c8a700 1 -- 172.21.15.114:0/2337469935 shutdown_connections 2024-02-19T19:04:15.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:15.534+0000 7f4fe2c8a700 1 -- 172.21.15.114:0/2337469935 wait complete. 2024-02-19T19:04:15.979 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:15 smithi114 ceph-mon[92983]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:15.979 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:15 smithi114 ceph-mon[92983]: Reconfiguring osd.2 (monmap changed)... 2024-02-19T19:04:15.979 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:15 smithi114 ceph-mon[92983]: Reconfiguring daemon osd.2 on smithi114 2024-02-19T19:04:15.979 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:15 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/2226272788' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:04:15.979 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:15 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/2337469935' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:04:16.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:15 smithi174 ceph-mon[87279]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:16.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:15 smithi174 ceph-mon[87279]: Reconfiguring osd.2 (monmap changed)... 2024-02-19T19:04:16.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:15 smithi174 ceph-mon[87279]: Reconfiguring daemon osd.2 on smithi114 2024-02-19T19:04:16.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:15 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/2226272788' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:04:16.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:15 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/2337469935' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:04:16.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:16 smithi114 ceph-mon[92983]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:16.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:16 smithi114 ceph-mon[92983]: pgmap v21: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:16.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:16 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:16.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:16 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:16.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:16 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.vpoeop", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:04:16.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:16 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:17.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:16 smithi174 ceph-mon[87279]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:17.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:16 smithi174 ceph-mon[87279]: pgmap v21: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:17.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:16 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:17.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:16 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:17.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:16 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.vpoeop", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:04:17.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:16 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:18.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:17 smithi114 ceph-mon[92983]: Reconfiguring mds.cephfs.smithi114.vpoeop (monmap changed)... 2024-02-19T19:04:18.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:17 smithi114 ceph-mon[92983]: Reconfiguring daemon mds.cephfs.smithi114.vpoeop on smithi114 2024-02-19T19:04:18.046 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:17 smithi174 ceph-mon[87279]: Reconfiguring mds.cephfs.smithi114.vpoeop (monmap changed)... 2024-02-19T19:04:18.046 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:17 smithi174 ceph-mon[87279]: Reconfiguring daemon mds.cephfs.smithi114.vpoeop on smithi114 2024-02-19T19:04:19.003 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:18 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:19.003 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:18 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:19.003 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:18 smithi114 ceph-mon[92983]: pgmap v22: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:19.003 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:18 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.tcwpod", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:04:19.003 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:18 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:19.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:18 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:19.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:18 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:19.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:18 smithi174 ceph-mon[87279]: pgmap v22: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:19.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:18 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi114.tcwpod", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:04:19.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:18 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:20.005 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:19 smithi174 ceph-mon[87279]: Reconfiguring mds.cephfs.smithi114.tcwpod (monmap changed)... 2024-02-19T19:04:20.005 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:19 smithi174 ceph-mon[87279]: Reconfiguring daemon mds.cephfs.smithi114.tcwpod on smithi114 2024-02-19T19:04:20.005 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:19 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:20.005 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:19 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:20.006 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:19 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi174", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:04:20.006 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:19 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi174", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:04:20.006 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:19 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi174", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-02-19T19:04:20.006 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:19 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi174"}]: dispatch 2024-02-19T19:04:20.006 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:19 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:20.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:19 smithi114 ceph-mon[92983]: Reconfiguring mds.cephfs.smithi114.tcwpod (monmap changed)... 2024-02-19T19:04:20.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:19 smithi114 ceph-mon[92983]: Reconfiguring daemon mds.cephfs.smithi114.tcwpod on smithi114 2024-02-19T19:04:20.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:19 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:20.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:19 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:20.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:19 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi174", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:04:20.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:19 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi174", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-02-19T19:04:20.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:19 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi174", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-02-19T19:04:20.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:19 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi174"}]: dispatch 2024-02-19T19:04:20.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:19 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:21.008 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:20 smithi174 ceph-mon[87279]: Reconfiguring ceph-exporter.smithi174 (monmap changed)... 2024-02-19T19:04:21.008 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:20 smithi174 ceph-mon[87279]: Unable to update caps for client.ceph-exporter.smithi174 2024-02-19T19:04:21.008 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:20 smithi174 ceph-mon[87279]: Reconfiguring daemon ceph-exporter.smithi174 on smithi174 2024-02-19T19:04:21.008 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:20 smithi174 ceph-mon[87279]: pgmap v23: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:21.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:20 smithi114 ceph-mon[92983]: Reconfiguring ceph-exporter.smithi174 (monmap changed)... 2024-02-19T19:04:21.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:20 smithi114 ceph-mon[92983]: Unable to update caps for client.ceph-exporter.smithi174 2024-02-19T19:04:21.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:20 smithi114 ceph-mon[92983]: Reconfiguring daemon ceph-exporter.smithi174 on smithi174 2024-02-19T19:04:21.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:20 smithi114 ceph-mon[92983]: pgmap v23: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:22.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:21 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:22.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:21 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:22.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:21 smithi174 ceph-mon[87279]: Reconfiguring crash.smithi174 (monmap changed)... 2024-02-19T19:04:22.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:21 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi174", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T19:04:22.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:21 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:22.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:21 smithi174 ceph-mon[87279]: Reconfiguring daemon crash.smithi174 on smithi174 2024-02-19T19:04:22.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:21 smithi174 ceph-mon[87279]: pgmap v24: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:22.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:21 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:22.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:21 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:22.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:21 smithi114 ceph-mon[92983]: Reconfiguring crash.smithi174 (monmap changed)... 2024-02-19T19:04:22.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:21 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi174", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T19:04:22.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:21 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:22.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:21 smithi114 ceph-mon[92983]: Reconfiguring daemon crash.smithi174 on smithi174 2024-02-19T19:04:22.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:21 smithi114 ceph-mon[92983]: pgmap v24: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:23.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:23 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:23.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:23 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:23.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:23 smithi174 ceph-mon[87279]: Reconfiguring mgr.smithi174.lydvnj (monmap changed)... 2024-02-19T19:04:23.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:23 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi174.lydvnj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:04:23.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:23 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:04:23.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:23 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:23.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:23 smithi174 ceph-mon[87279]: Reconfiguring daemon mgr.smithi174.lydvnj on smithi174 2024-02-19T19:04:24.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:23 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:24.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:23 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:24.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:23 smithi114 ceph-mon[92983]: Reconfiguring mgr.smithi174.lydvnj (monmap changed)... 2024-02-19T19:04:24.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:23 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi174.lydvnj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:04:24.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:23 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:04:24.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:23 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:24.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:23 smithi114 ceph-mon[92983]: Reconfiguring daemon mgr.smithi174.lydvnj on smithi174 2024-02-19T19:04:24.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:24 smithi174 ceph-mon[87279]: pgmap v25: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:24.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:24 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:24.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:24 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:24.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:24 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T19:04:24.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:24 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T19:04:24.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:24 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:25.013 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:24 smithi114 ceph-mon[92983]: pgmap v25: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:25.013 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:24 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:25.013 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:24 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:25.013 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:24 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T19:04:25.013 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:24 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T19:04:25.013 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:24 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:26.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:25 smithi114 ceph-mon[92983]: Reconfiguring mon.smithi174 (monmap changed)... 2024-02-19T19:04:26.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:25 smithi114 ceph-mon[92983]: Reconfiguring daemon mon.smithi174 on smithi174 2024-02-19T19:04:26.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:04:26.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:25 smithi174 ceph-mon[87279]: Reconfiguring mon.smithi174 (monmap changed)... 2024-02-19T19:04:26.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:25 smithi174 ceph-mon[87279]: Reconfiguring daemon mon.smithi174 on smithi174 2024-02-19T19:04:26.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:04:27.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:26 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:27.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:26 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:27.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:26 smithi114 ceph-mon[92983]: pgmap v26: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:27.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:26 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-19T19:04:27.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:26 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:27.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:26 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:27.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:26 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:27.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:26 smithi174 ceph-mon[87279]: pgmap v26: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:27.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:26 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-19T19:04:27.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:26 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:28.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:27 smithi114 ceph-mon[92983]: Reconfiguring osd.3 (monmap changed)... 2024-02-19T19:04:28.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:27 smithi114 ceph-mon[92983]: Reconfiguring daemon osd.3 on smithi174 2024-02-19T19:04:28.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:27 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:28.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:27 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:28.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:27 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-19T19:04:28.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:27 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:28.110 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:27 smithi174 ceph-mon[87279]: Reconfiguring osd.3 (monmap changed)... 2024-02-19T19:04:28.110 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:27 smithi174 ceph-mon[87279]: Reconfiguring daemon osd.3 on smithi174 2024-02-19T19:04:28.110 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:27 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:28.110 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:27 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:28.110 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:27 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-19T19:04:28.110 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:27 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:29.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:28 smithi114 ceph-mon[92983]: Reconfiguring osd.4 (monmap changed)... 2024-02-19T19:04:29.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:28 smithi114 ceph-mon[92983]: Reconfiguring daemon osd.4 on smithi174 2024-02-19T19:04:29.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:28 smithi114 ceph-mon[92983]: pgmap v27: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:29.074 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:28 smithi174 ceph-mon[87279]: Reconfiguring osd.4 (monmap changed)... 2024-02-19T19:04:29.074 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:28 smithi174 ceph-mon[87279]: Reconfiguring daemon osd.4 on smithi174 2024-02-19T19:04:29.075 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:28 smithi174 ceph-mon[87279]: pgmap v27: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:30.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:30 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:30.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:30 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:30.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:30 smithi174 ceph-mon[87279]: Reconfiguring osd.5 (monmap changed)... 2024-02-19T19:04:30.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:30 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-19T19:04:30.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:30 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:30.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:30 smithi174 ceph-mon[87279]: Reconfiguring daemon osd.5 on smithi174 2024-02-19T19:04:30.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:30 smithi174 ceph-mon[87279]: pgmap v28: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:30.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:30 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:30.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:30 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:30.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:30 smithi114 ceph-mon[92983]: Reconfiguring osd.5 (monmap changed)... 2024-02-19T19:04:30.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:30 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-19T19:04:30.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:30 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:30.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:30 smithi114 ceph-mon[92983]: Reconfiguring daemon osd.5 on smithi174 2024-02-19T19:04:30.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:30 smithi114 ceph-mon[92983]: pgmap v28: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:32.030 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:31 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:32.030 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:31 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:32.031 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:31 smithi174 ceph-mon[87279]: Reconfiguring mds.cephfs.smithi174.znhuvd (monmap changed)... 2024-02-19T19:04:32.031 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:31 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi174.znhuvd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:04:32.031 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:31 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:32.031 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:31 smithi174 ceph-mon[87279]: Reconfiguring daemon mds.cephfs.smithi174.znhuvd on smithi174 2024-02-19T19:04:32.031 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:31 smithi174 ceph-mon[87279]: pgmap v29: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:32.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:31 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:32.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:31 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:32.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:31 smithi114 ceph-mon[92983]: Reconfiguring mds.cephfs.smithi174.znhuvd (monmap changed)... 2024-02-19T19:04:32.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:31 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi174.znhuvd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:04:32.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:31 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:32.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:31 smithi114 ceph-mon[92983]: Reconfiguring daemon mds.cephfs.smithi174.znhuvd on smithi174 2024-02-19T19:04:32.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:31 smithi114 ceph-mon[92983]: pgmap v29: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:33 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:33 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:33 smithi174 ceph-mon[87279]: Reconfiguring mds.cephfs.smithi174.stnfnf (monmap changed)... 2024-02-19T19:04:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:33 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi174.stnfnf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:04:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:33 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:33.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:33 smithi174 ceph-mon[87279]: Reconfiguring daemon mds.cephfs.smithi174.stnfnf on smithi174 2024-02-19T19:04:33.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:33 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:33.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:33 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:33.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:33 smithi114 ceph-mon[92983]: Reconfiguring mds.cephfs.smithi174.stnfnf (monmap changed)... 2024-02-19T19:04:33.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:33 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi174.stnfnf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T19:04:33.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:33 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:33.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:33 smithi114 ceph-mon[92983]: Reconfiguring daemon mds.cephfs.smithi174.stnfnf on smithi174 2024-02-19T19:04:34.758 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:34 smithi114 ceph-mon[92983]: pgmap v30: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:34.758 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:34 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:34.759 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:34 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:34.759 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:34 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:04:34.759 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:34 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:04:34.759 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:34 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:04:34.759 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:34 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:34.759 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:34 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi114"}]: dispatch 2024-02-19T19:04:34.759 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:34 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi114"}]': finished 2024-02-19T19:04:34.759 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:34 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi174"}]: dispatch 2024-02-19T19:04:34.759 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:34 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi174"}]': finished 2024-02-19T19:04:34.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:34 smithi174 ceph-mon[87279]: pgmap v30: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:34.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:34 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:34.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:34 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:34.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:34 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:04:34.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:34 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:04:34.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:34 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:04:34.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:34 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:34.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:34 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi114"}]: dispatch 2024-02-19T19:04:34.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:34 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi114"}]': finished 2024-02-19T19:04:34.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:34 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi174"}]: dispatch 2024-02-19T19:04:34.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:34 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi174"}]': finished 2024-02-19T19:04:35.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:35 smithi114 ceph-mon[92983]: Upgrade: Setting container_image for all mon 2024-02-19T19:04:35.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:35 smithi174 ceph-mon[87279]: Upgrade: Setting container_image for all mon 2024-02-19T19:04:36.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:36 smithi114 ceph-mon[92983]: pgmap v31: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:36.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:36 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:36.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:36 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi114", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T19:04:36.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:36 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:36.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:36 smithi174 ceph-mon[87279]: pgmap v31: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:36.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:36 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:36.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:36 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi114", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T19:04:36.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:36 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:37.741 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:37 smithi114 ceph-mon[92983]: Upgrade: Updating crash.smithi114 (1/2) 2024-02-19T19:04:37.741 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:37 smithi114 ceph-mon[92983]: Deploying daemon crash.smithi114 on smithi114 2024-02-19T19:04:37.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:37 smithi174 ceph-mon[87279]: Upgrade: Updating crash.smithi114 (1/2) 2024-02-19T19:04:37.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:37 smithi174 ceph-mon[87279]: Deploying daemon crash.smithi114 on smithi114 2024-02-19T19:04:38.548 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:38 smithi114 ceph-mon[92983]: pgmap v32: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:38.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:38 smithi174 ceph-mon[87279]: pgmap v32: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:39.943 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:39 smithi114 ceph-mon[92983]: pgmap v33: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:39.943 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:39 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:04:40.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:39 smithi174 ceph-mon[87279]: pgmap v33: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:40.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:39 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:04:41.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:41 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:41.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:41 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:41.320 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:41 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:41.320 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:41 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:42.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:42 smithi174 ceph-mon[87279]: Upgrade: Updating crash.smithi174 (2/2) 2024-02-19T19:04:42.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:42 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:42.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:42 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi174", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T19:04:42.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:42 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:42.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:42 smithi174 ceph-mon[87279]: Deploying daemon crash.smithi174 on smithi174 2024-02-19T19:04:42.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:42 smithi174 ceph-mon[87279]: pgmap v34: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:43.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:42 smithi114 ceph-mon[92983]: Upgrade: Updating crash.smithi174 (2/2) 2024-02-19T19:04:43.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:42 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:43.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:42 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi174", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T19:04:43.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:42 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:43.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:42 smithi114 ceph-mon[92983]: Deploying daemon crash.smithi174 on smithi174 2024-02-19T19:04:43.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:42 smithi114 ceph-mon[92983]: pgmap v34: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:44.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:43 smithi114 ceph-mon[92983]: pgmap v35: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:44.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:43 smithi174 ceph-mon[87279]: pgmap v35: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:45.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.650+0000 7f6c9c02f700 1 -- 172.21.15.114:0/2891922271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c94100bd0 msgr2=0x7f6c94100ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:45.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.650+0000 7f6c9c02f700 1 --2- 172.21.15.114:0/2891922271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c94100bd0 0x7f6c94100ff0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009910 tx=0x7f6c84009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:45.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.650+0000 7f6c9c02f700 1 -- 172.21.15.114:0/2891922271 shutdown_connections 2024-02-19T19:04:45.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.650+0000 7f6c9c02f700 1 --2- 172.21.15.114:0/2891922271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c94101dd0 0x7f6c94102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:45.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.650+0000 7f6c9c02f700 1 --2- 172.21.15.114:0/2891922271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c94100bd0 0x7f6c94100ff0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:45.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.650+0000 7f6c9c02f700 1 -- 172.21.15.114:0/2891922271 >> 172.21.15.114:0/2891922271 conn(0x7f6c940fc170 msgr2=0x7f6c940fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:45.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.650+0000 7f6c9c02f700 1 -- 172.21.15.114:0/2891922271 shutdown_connections 2024-02-19T19:04:45.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.650+0000 7f6c9c02f700 1 -- 172.21.15.114:0/2891922271 wait complete. 2024-02-19T19:04:45.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.651+0000 7f6c9c02f700 1 Processor -- start 2024-02-19T19:04:45.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.651+0000 7f6c9c02f700 1 -- start start 2024-02-19T19:04:45.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.651+0000 7f6c9c02f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c94100bd0 0x7f6c94196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:45.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.651+0000 7f6c9c02f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c94101dd0 0x7f6c941968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:45.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.652+0000 7f6c9c02f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c94196ee0 con 0x7f6c94101dd0 2024-02-19T19:04:45.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.652+0000 7f6c9c02f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c94197020 con 0x7f6c94100bd0 2024-02-19T19:04:45.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.652+0000 7f6c995ca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c94101dd0 0x7f6c941968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:45.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.652+0000 7f6c995ca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c94101dd0 0x7f6c941968c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55248/0 (socket says 172.21.15.114:55248) 2024-02-19T19:04:45.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.652+0000 7f6c995ca700 1 -- 172.21.15.114:0/2308826746 learned_addr learned my addr 172.21.15.114:0/2308826746 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:04:45.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.652+0000 7f6c99dcb700 1 --2- 172.21.15.114:0/2308826746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c94100bd0 0x7f6c94196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:45.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.652+0000 7f6c995ca700 1 -- 172.21.15.114:0/2308826746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c94100bd0 msgr2=0x7f6c94196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:45.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.652+0000 7f6c995ca700 1 --2- 172.21.15.114:0/2308826746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c94100bd0 0x7f6c94196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:45.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.652+0000 7f6c995ca700 1 -- 172.21.15.114:0/2308826746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c840095f0 con 0x7f6c94101dd0 2024-02-19T19:04:45.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.652+0000 7f6c995ca700 1 --2- 172.21.15.114:0/2308826746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c94101dd0 0x7f6c941968c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000e920 tx=0x7f6c9000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:45.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.652+0000 7f6c8affd700 1 -- 172.21.15.114:0/2308826746 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c9000c9d0 con 0x7f6c94101dd0 2024-02-19T19:04:45.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.652+0000 7f6c8affd700 1 -- 172.21.15.114:0/2308826746 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6c90010430 con 0x7f6c94101dd0 2024-02-19T19:04:45.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.652+0000 7f6c8affd700 1 -- 172.21.15.114:0/2308826746 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c90018650 con 0x7f6c94101dd0 2024-02-19T19:04:45.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.652+0000 7f6c9c02f700 1 -- 172.21.15.114:0/2308826746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9419bad0 con 0x7f6c94101dd0 2024-02-19T19:04:45.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.652+0000 7f6c9c02f700 1 -- 172.21.15.114:0/2308826746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9419c020 con 0x7f6c94101dd0 2024-02-19T19:04:45.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.654+0000 7f6c9c02f700 1 -- 172.21.15.114:0/2308826746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c94109730 con 0x7f6c94101dd0 2024-02-19T19:04:45.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.654+0000 7f6c8affd700 1 -- 172.21.15.114:0/2308826746 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 36) v1 ==== 91597+0+0 (secure 0 0 0) 0x7f6c900105a0 con 0x7f6c94101dd0 2024-02-19T19:04:45.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.655+0000 7f6c8affd700 1 --2- 172.21.15.114:0/2308826746 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6c8006d9e0 0x7f6c8006fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:45.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.655+0000 7f6c8affd700 1 -- 172.21.15.114:0/2308826746 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6c90014070 con 0x7f6c94101dd0 2024-02-19T19:04:45.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.655+0000 7f6c99dcb700 1 --2- 172.21.15.114:0/2308826746 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6c8006d9e0 0x7f6c8006fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:45.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.655+0000 7f6c99dcb700 1 --2- 172.21.15.114:0/2308826746 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6c8006d9e0 0x7f6c8006fea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400b6f0 tx=0x7f6c84005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.659+0000 7f6c8affd700 1 -- 172.21.15.114:0/2308826746 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6c90052d20 con 0x7f6c94101dd0 2024-02-19T19:04:45.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.884+0000 7f6c9c02f700 1 -- 172.21.15.114:0/2308826746 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c94106720 con 0x7f6c8006d9e0 2024-02-19T19:04:45.905 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.885+0000 7f6c8affd700 1 -- 172.21.15.114:0/2308826746 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f6c94106720 con 0x7f6c8006d9e0 2024-02-19T19:04:45.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.887+0000 7f6c9c02f700 1 -- 172.21.15.114:0/2308826746 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6c8006d9e0 msgr2=0x7f6c8006fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:45.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.887+0000 7f6c9c02f700 1 --2- 172.21.15.114:0/2308826746 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6c8006d9e0 0x7f6c8006fea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400b6f0 tx=0x7f6c84005940 comp rx=0 tx=0).stop 2024-02-19T19:04:45.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.887+0000 7f6c9c02f700 1 -- 172.21.15.114:0/2308826746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c94101dd0 msgr2=0x7f6c941968c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:45.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.887+0000 7f6c9c02f700 1 --2- 172.21.15.114:0/2308826746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c94101dd0 0x7f6c941968c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000e920 tx=0x7f6c9000ec30 comp rx=0 tx=0).stop 2024-02-19T19:04:45.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.887+0000 7f6c9c02f700 1 -- 172.21.15.114:0/2308826746 shutdown_connections 2024-02-19T19:04:45.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.887+0000 7f6c9c02f700 1 --2- 172.21.15.114:0/2308826746 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6c8006d9e0 0x7f6c8006fea0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:45.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.887+0000 7f6c9c02f700 1 --2- 172.21.15.114:0/2308826746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c94100bd0 0x7f6c94196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:45.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.887+0000 7f6c9c02f700 1 --2- 172.21.15.114:0/2308826746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c94101dd0 0x7f6c941968c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:45.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.887+0000 7f6c9c02f700 1 -- 172.21.15.114:0/2308826746 >> 172.21.15.114:0/2308826746 conn(0x7f6c940fc170 msgr2=0x7f6c94105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:45.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.887+0000 7f6c9c02f700 1 -- 172.21.15.114:0/2308826746 shutdown_connections 2024-02-19T19:04:45.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:45.887+0000 7f6c9c02f700 1 -- 172.21.15.114:0/2308826746 wait complete. 2024-02-19T19:04:45.917 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-02-19T19:04:46.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.000+0000 7f4104e4f700 1 -- 172.21.15.114:0/663817009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41000fee80 msgr2=0x7f41000ff2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:46.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.000+0000 7f4104e4f700 1 --2- 172.21.15.114:0/663817009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41000fee80 0x7f41000ff2a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009910 tx=0x7f40f0009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:46.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.000+0000 7f4104e4f700 1 -- 172.21.15.114:0/663817009 shutdown_connections 2024-02-19T19:04:46.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.000+0000 7f4104e4f700 1 --2- 172.21.15.114:0/663817009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4100101e60 0x7f41001022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:46.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.000+0000 7f4104e4f700 1 --2- 172.21.15.114:0/663817009 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41000fee80 0x7f41000ff2a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:46.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.000+0000 7f4104e4f700 1 -- 172.21.15.114:0/663817009 >> 172.21.15.114:0/663817009 conn(0x7f41000fa7c0 msgr2=0x7f41000fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:46.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.000+0000 7f4104e4f700 1 -- 172.21.15.114:0/663817009 shutdown_connections 2024-02-19T19:04:46.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.000+0000 7f4104e4f700 1 -- 172.21.15.114:0/663817009 wait complete. 2024-02-19T19:04:46.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.001+0000 7f4104e4f700 1 Processor -- start 2024-02-19T19:04:46.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.001+0000 7f4104e4f700 1 -- start start 2024-02-19T19:04:46.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.001+0000 7f4104e4f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41000fee80 0x7f41001963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:46.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.001+0000 7f4104e4f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4100101e60 0x7f41001968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:46.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.001+0000 7f4104e4f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4100196f00 con 0x7f41000fee80 2024-02-19T19:04:46.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.001+0000 7f4104e4f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4100197040 con 0x7f4100101e60 2024-02-19T19:04:46.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.001+0000 7f40fe59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41000fee80 0x7f41001963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:46.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.002+0000 7f40fe59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41000fee80 0x7f41001963a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55268/0 (socket says 172.21.15.114:55268) 2024-02-19T19:04:46.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.002+0000 7f40fe59c700 1 -- 172.21.15.114:0/399349627 learned_addr learned my addr 172.21.15.114:0/399349627 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:04:46.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.002+0000 7f40fdd9b700 1 --2- 172.21.15.114:0/399349627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4100101e60 0x7f41001968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:46.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.002+0000 7f40fe59c700 1 -- 172.21.15.114:0/399349627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4100101e60 msgr2=0x7f41001968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:46.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.002+0000 7f40fe59c700 1 --2- 172.21.15.114:0/399349627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4100101e60 0x7f41001968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:46.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.002+0000 7f40fe59c700 1 -- 172.21.15.114:0/399349627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f00095f0 con 0x7f41000fee80 2024-02-19T19:04:46.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.002+0000 7f40fe59c700 1 --2- 172.21.15.114:0/399349627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41000fee80 0x7f41001963a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f40f000f6c0 tx=0x7f40f000f7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:46.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.002+0000 7f40ef7fe700 1 -- 172.21.15.114:0/399349627 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40f001d070 con 0x7f41000fee80 2024-02-19T19:04:46.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.002+0000 7f40ef7fe700 1 -- 172.21.15.114:0/399349627 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f40f0005dc0 con 0x7f41000fee80 2024-02-19T19:04:46.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.002+0000 7f4104e4f700 1 -- 172.21.15.114:0/399349627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410019ba90 con 0x7f41000fee80 2024-02-19T19:04:46.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.002+0000 7f40ef7fe700 1 -- 172.21.15.114:0/399349627 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40f0017630 con 0x7f41000fee80 2024-02-19T19:04:46.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.002+0000 7f4104e4f700 1 -- 172.21.15.114:0/399349627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f410019bf80 con 0x7f41000fee80 2024-02-19T19:04:46.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.004+0000 7f4104e4f700 1 -- 172.21.15.114:0/399349627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4100109720 con 0x7f41000fee80 2024-02-19T19:04:46.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.005+0000 7f40ef7fe700 1 -- 172.21.15.114:0/399349627 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 36) v1 ==== 91597+0+0 (secure 0 0 0) 0x7f40f0003680 con 0x7f41000fee80 2024-02-19T19:04:46.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.005+0000 7f40ef7fe700 1 --2- 172.21.15.114:0/399349627 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f40e806d9e0 0x7f40e806fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:46.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.005+0000 7f40ef7fe700 1 -- 172.21.15.114:0/399349627 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f40f008eb80 con 0x7f41000fee80 2024-02-19T19:04:46.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.005+0000 7f40fdd9b700 1 --2- 172.21.15.114:0/399349627 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f40e806d9e0 0x7f40e806fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:46.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.006+0000 7f40fdd9b700 1 --2- 172.21.15.114:0/399349627 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f40e806d9e0 0x7f40e806fea0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009bf0 tx=0x7f40f4009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:46.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.010+0000 7f40ef7fe700 1 -- 172.21.15.114:0/399349627 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f40f005bd10 con 0x7f41000fee80 2024-02-19T19:04:46.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.245+0000 7f4104e4f700 1 -- 172.21.15.114:0/399349627 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f410019c6d0 con 0x7f40e806d9e0 2024-02-19T19:04:46.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.246+0000 7f40ef7fe700 1 -- 172.21.15.114:0/399349627 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f410019c6d0 con 0x7f40e806d9e0 2024-02-19T19:04:46.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.248+0000 7f4104e4f700 1 -- 172.21.15.114:0/399349627 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f40e806d9e0 msgr2=0x7f40e806fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:46.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.248+0000 7f4104e4f700 1 --2- 172.21.15.114:0/399349627 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f40e806d9e0 0x7f40e806fea0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009bf0 tx=0x7f40f4009260 comp rx=0 tx=0).stop 2024-02-19T19:04:46.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.248+0000 7f4104e4f700 1 -- 172.21.15.114:0/399349627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41000fee80 msgr2=0x7f41001963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:46.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.248+0000 7f4104e4f700 1 --2- 172.21.15.114:0/399349627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41000fee80 0x7f41001963a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f40f000f6c0 tx=0x7f40f000f7a0 comp rx=0 tx=0).stop 2024-02-19T19:04:46.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.249+0000 7f4104e4f700 1 -- 172.21.15.114:0/399349627 shutdown_connections 2024-02-19T19:04:46.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.249+0000 7f4104e4f700 1 --2- 172.21.15.114:0/399349627 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f40e806d9e0 0x7f40e806fea0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:46.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.249+0000 7f4104e4f700 1 --2- 172.21.15.114:0/399349627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41000fee80 0x7f41001963a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:46.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.249+0000 7f4104e4f700 1 --2- 172.21.15.114:0/399349627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4100101e60 0x7f41001968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:46.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.249+0000 7f4104e4f700 1 -- 172.21.15.114:0/399349627 >> 172.21.15.114:0/399349627 conn(0x7f41000fa7c0 msgr2=0x7f4100105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:46.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.249+0000 7f4104e4f700 1 -- 172.21.15.114:0/399349627 shutdown_connections 2024-02-19T19:04:46.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.249+0000 7f4104e4f700 1 -- 172.21.15.114:0/399349627 wait complete. 2024-02-19T19:04:46.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.356+0000 7f23cb761700 1 -- 172.21.15.114:0/4281221691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c4100ce0 msgr2=0x7f23c4101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:46.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.356+0000 7f23cb761700 1 --2- 172.21.15.114:0/4281221691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c4100ce0 0x7f23c4101140 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f23b4009910 tx=0x7f23b4009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:46.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.356+0000 7f23cb761700 1 -- 172.21.15.114:0/4281221691 shutdown_connections 2024-02-19T19:04:46.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.356+0000 7f23cb761700 1 --2- 172.21.15.114:0/4281221691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c4100ce0 0x7f23c4101140 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:46.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.356+0000 7f23cb761700 1 --2- 172.21.15.114:0/4281221691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23c40ffae0 0x7f23c40fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:46.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.356+0000 7f23cb761700 1 -- 172.21.15.114:0/4281221691 >> 172.21.15.114:0/4281221691 conn(0x7f23c40fb080 msgr2=0x7f23c40fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:46.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.356+0000 7f23cb761700 1 -- 172.21.15.114:0/4281221691 shutdown_connections 2024-02-19T19:04:46.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.356+0000 7f23cb761700 1 -- 172.21.15.114:0/4281221691 wait complete. 2024-02-19T19:04:46.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.357+0000 7f23cb761700 1 Processor -- start 2024-02-19T19:04:46.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.357+0000 7f23cb761700 1 -- start start 2024-02-19T19:04:46.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.357+0000 7f23cb761700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23c40ffae0 0x7f23c40698d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:46.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.357+0000 7f23cb761700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c4100ce0 0x7f23c419b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:46.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.357+0000 7f23cb761700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c419b630 con 0x7f23c4100ce0 2024-02-19T19:04:46.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.357+0000 7f23cb761700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c419b770 con 0x7f23c40ffae0 2024-02-19T19:04:46.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.357+0000 7f23c8cfc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c4100ce0 0x7f23c419b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:46.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.357+0000 7f23c8cfc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c4100ce0 0x7f23c419b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55294/0 (socket says 172.21.15.114:55294) 2024-02-19T19:04:46.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.357+0000 7f23c8cfc700 1 -- 172.21.15.114:0/3304654744 learned_addr learned my addr 172.21.15.114:0/3304654744 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:04:46.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.357+0000 7f23c94fd700 1 --2- 172.21.15.114:0/3304654744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23c40ffae0 0x7f23c40698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:46.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.357+0000 7f23c8cfc700 1 -- 172.21.15.114:0/3304654744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23c40ffae0 msgr2=0x7f23c40698d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:46.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.357+0000 7f23c8cfc700 1 --2- 172.21.15.114:0/3304654744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23c40ffae0 0x7f23c40698d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:46.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.357+0000 7f23c8cfc700 1 -- 172.21.15.114:0/3304654744 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b40095f0 con 0x7f23c4100ce0 2024-02-19T19:04:46.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.357+0000 7f23c8cfc700 1 --2- 172.21.15.114:0/3304654744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c4100ce0 0x7f23c419b0f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f23b400b5c0 tx=0x7f23b400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:46.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.358+0000 7f23ba7fc700 1 -- 172.21.15.114:0/3304654744 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23b401d070 con 0x7f23c4100ce0 2024-02-19T19:04:46.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.358+0000 7f23ba7fc700 1 -- 172.21.15.114:0/3304654744 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f23b4022470 con 0x7f23c4100ce0 2024-02-19T19:04:46.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.358+0000 7f23cb761700 1 -- 172.21.15.114:0/3304654744 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c419b9f0 con 0x7f23c4100ce0 2024-02-19T19:04:46.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.358+0000 7f23ba7fc700 1 -- 172.21.15.114:0/3304654744 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23b400f650 con 0x7f23c4100ce0 2024-02-19T19:04:46.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.358+0000 7f23cb761700 1 -- 172.21.15.114:0/3304654744 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c419bee0 con 0x7f23c4100ce0 2024-02-19T19:04:46.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.359+0000 7f23cb761700 1 -- 172.21.15.114:0/3304654744 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c4108640 con 0x7f23c4100ce0 2024-02-19T19:04:46.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.359+0000 7f23ba7fc700 1 -- 172.21.15.114:0/3304654744 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 36) v1 ==== 91597+0+0 (secure 0 0 0) 0x7f23b40225e0 con 0x7f23c4100ce0 2024-02-19T19:04:46.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.360+0000 7f23ba7fc700 1 --2- 172.21.15.114:0/3304654744 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f23b006d9d0 0x7f23b006fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:46.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.360+0000 7f23ba7fc700 1 -- 172.21.15.114:0/3304654744 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f23b408ef60 con 0x7f23c4100ce0 2024-02-19T19:04:46.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.360+0000 7f23c94fd700 1 --2- 172.21.15.114:0/3304654744 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f23b006d9d0 0x7f23b006fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:46.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.360+0000 7f23c94fd700 1 --2- 172.21.15.114:0/3304654744 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f23b006d9d0 0x7f23b006fe90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f23c4100b40 tx=0x7f23c000a3b0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:46.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.364+0000 7f23ba7fc700 1 -- 172.21.15.114:0/3304654744 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f23b405c010 con 0x7f23c4100ce0 2024-02-19T19:04:46.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.584+0000 7f23cb761700 1 -- 172.21.15.114:0/3304654744 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f23c4105630 con 0x7f23b006d9d0 2024-02-19T19:04:46.613 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:46 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:46.613 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:46 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:46.613 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:46 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:04:46.613 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:46 smithi174 ceph-mon[87279]: pgmap v36: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:46.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.594+0000 7f23ba7fc700 1 -- 172.21.15.114:0/3304654744 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f23c4105630 con 0x7f23b006d9d0 2024-02-19T19:04:46.614 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:04:46.614 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (8m) 60s ago 11m 26.4M - 0.25.0 c8568f914cd2 30444b676ca4 2024-02-19T19:04:46.614 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (11m) 60s ago 11m 7852k - 18.2.1 94eadd990e0a 4918cd87e312 2024-02-19T19:04:46.614 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi174 smithi174 running (6m) 48s ago 9m 7780k - 18.2.1 94eadd990e0a 516c34431b93 2024-02-19T19:04:46.614 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 starting - - - - 2024-02-19T19:04:46.615 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi174 smithi174 starting - - - - 2024-02-19T19:04:46.615 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (8m) 60s ago 10m 85.7M - 9.4.7 954c08fa6188 478bd533e827 2024-02-19T19:04:46.615 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tcwpod smithi114 running (4m) 60s ago 4m 26.7M - 18.2.1 94eadd990e0a 420175777cf4 2024-02-19T19:04:46.615 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.vpoeop smithi114 running (4m) 60s ago 4m 22.1M - 18.2.1 94eadd990e0a f81a79e53ead 2024-02-19T19:04:46.615 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.stnfnf smithi174 running (4m) 48s ago 4m 24.0M - 18.2.1 94eadd990e0a 3df35954a492 2024-02-19T19:04:46.615 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.znhuvd smithi174 running (4m) 48s ago 4m 22.7M - 18.2.1 94eadd990e0a 2d2267494ae1 2024-02-19T19:04:46.615 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.pcvtrk smithi114 *:8443,9283,8765 running (2m) 60s ago 12m 470M - 19.0.0-1450-gf78a58c0 f081f847070a 05b5c06122b1 2024-02-19T19:04:46.615 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi174.lydvnj smithi174 *:8443,9283,8765 running (89s) 48s ago 9m 450M - 19.0.0-1450-gf78a58c0 f081f847070a f962b1af5aed 2024-02-19T19:04:46.615 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (75s) 60s ago 13m 29.8M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e51f650f9be9 2024-02-19T19:04:46.615 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi174 smithi174 running (52s) 48s ago 9m 23.8M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e811581f05fb 2024-02-19T19:04:46.615 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (11m) 60s ago 11m 15.0M - 1.5.0 0da6a335fe13 b6b258074cce 2024-02-19T19:04:46.615 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi174 smithi174 *:9100 running (9m) 48s ago 9m 14.7M - 1.5.0 0da6a335fe13 d0b4fdf67b22 2024-02-19T19:04:46.615 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (8m) 60s ago 8m 48.9M 1216M 18.2.1 94eadd990e0a 990afc982b6c 2024-02-19T19:04:46.615 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (7m) 60s ago 7m 49.3M 1216M 18.2.1 94eadd990e0a 932e3ad37db5 2024-02-19T19:04:46.615 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (7m) 60s ago 7m 51.3M 1216M 18.2.1 94eadd990e0a 12953988af86 2024-02-19T19:04:46.615 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi174 running (7m) 48s ago 7m 48.9M 2240M 18.2.1 94eadd990e0a 979f45cabbce 2024-02-19T19:04:46.616 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi174 running (6m) 48s ago 6m 49.5M 2240M 18.2.1 94eadd990e0a 8b8c3745b8c5 2024-02-19T19:04:46.616 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi174 running (6m) 48s ago 6m 51.9M 2240M 18.2.1 94eadd990e0a 5771a995ed75 2024-02-19T19:04:46.616 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (94s) 60s ago 9m 45.0M - 2.43.0 a07b618ecd1d a36922c7c9a1 2024-02-19T19:04:46.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.596+0000 7f23cb761700 1 -- 172.21.15.114:0/3304654744 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f23b006d9d0 msgr2=0x7f23b006fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:46.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.596+0000 7f23cb761700 1 --2- 172.21.15.114:0/3304654744 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f23b006d9d0 0x7f23b006fe90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f23c4100b40 tx=0x7f23c000a3b0 comp rx=0 tx=0).stop 2024-02-19T19:04:46.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.596+0000 7f23cb761700 1 -- 172.21.15.114:0/3304654744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c4100ce0 msgr2=0x7f23c419b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:46.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.596+0000 7f23cb761700 1 --2- 172.21.15.114:0/3304654744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c4100ce0 0x7f23c419b0f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f23b400b5c0 tx=0x7f23b400bd70 comp rx=0 tx=0).stop 2024-02-19T19:04:46.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.596+0000 7f23cb761700 1 -- 172.21.15.114:0/3304654744 shutdown_connections 2024-02-19T19:04:46.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.596+0000 7f23cb761700 1 --2- 172.21.15.114:0/3304654744 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f23b006d9d0 0x7f23b006fe90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:46.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.596+0000 7f23cb761700 1 --2- 172.21.15.114:0/3304654744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23c40ffae0 0x7f23c40698d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:46.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.596+0000 7f23cb761700 1 --2- 172.21.15.114:0/3304654744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f23c4100ce0 0x7f23c419b0f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:46.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.596+0000 7f23cb761700 1 -- 172.21.15.114:0/3304654744 >> 172.21.15.114:0/3304654744 conn(0x7f23c40fb080 msgr2=0x7f23c4103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:46.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.596+0000 7f23cb761700 1 -- 172.21.15.114:0/3304654744 shutdown_connections 2024-02-19T19:04:46.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.596+0000 7f23cb761700 1 -- 172.21.15.114:0/3304654744 wait complete. 2024-02-19T19:04:46.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.701+0000 7f15e9858700 1 -- 172.21.15.114:0/638497188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15e4101dc0 msgr2=0x7f15e4102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:46.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.701+0000 7f15e9858700 1 --2- 172.21.15.114:0/638497188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15e4101dc0 0x7f15e4102220 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f15d8009910 tx=0x7f15d8009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:46.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.701+0000 7f15e9858700 1 -- 172.21.15.114:0/638497188 shutdown_connections 2024-02-19T19:04:46.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.701+0000 7f15e9858700 1 --2- 172.21.15.114:0/638497188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15e4101dc0 0x7f15e4102220 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:46.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.701+0000 7f15e9858700 1 --2- 172.21.15.114:0/638497188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15e4100bc0 0x7f15e4100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:46.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.701+0000 7f15e9858700 1 -- 172.21.15.114:0/638497188 >> 172.21.15.114:0/638497188 conn(0x7f15e40fc140 msgr2=0x7f15e40fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:46.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.701+0000 7f15e9858700 1 -- 172.21.15.114:0/638497188 shutdown_connections 2024-02-19T19:04:46.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.701+0000 7f15e9858700 1 -- 172.21.15.114:0/638497188 wait complete. 2024-02-19T19:04:46.722 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:46 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:46.722 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:46 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:46.723 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:46 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:04:46.723 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:46 smithi114 ceph-mon[92983]: pgmap v36: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:46.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.702+0000 7f15e9858700 1 Processor -- start 2024-02-19T19:04:46.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.702+0000 7f15e9858700 1 -- start start 2024-02-19T19:04:46.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.702+0000 7f15e9858700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15e4100bc0 0x7f15e4196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:46.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.702+0000 7f15e9858700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15e4101dc0 0x7f15e41968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:46.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.702+0000 7f15e9858700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e4196ef0 con 0x7f15e4100bc0 2024-02-19T19:04:46.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.702+0000 7f15e9858700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e4197030 con 0x7f15e4101dc0 2024-02-19T19:04:46.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.702+0000 7f15e2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15e4100bc0 0x7f15e4196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:46.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.702+0000 7f15e2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15e4100bc0 0x7f15e4196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55316/0 (socket says 172.21.15.114:55316) 2024-02-19T19:04:46.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.702+0000 7f15e2ffd700 1 -- 172.21.15.114:0/2036141051 learned_addr learned my addr 172.21.15.114:0/2036141051 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:04:46.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.702+0000 7f15e2ffd700 1 -- 172.21.15.114:0/2036141051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15e4101dc0 msgr2=0x7f15e41968d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T19:04:46.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.702+0000 7f15e2ffd700 1 --2- 172.21.15.114:0/2036141051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15e4101dc0 0x7f15e41968d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:46.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.702+0000 7f15e2ffd700 1 -- 172.21.15.114:0/2036141051 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d80095f0 con 0x7f15e4100bc0 2024-02-19T19:04:46.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.703+0000 7f15e2ffd700 1 --2- 172.21.15.114:0/2036141051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15e4100bc0 0x7f15e4196390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f15d400e8d0 tx=0x7f15d400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:46.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.703+0000 7f15e8856700 1 -- 172.21.15.114:0/2036141051 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15d4009770 con 0x7f15e4100bc0 2024-02-19T19:04:46.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.703+0000 7f15e8856700 1 -- 172.21.15.114:0/2036141051 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f15d400eed0 con 0x7f15e4100bc0 2024-02-19T19:04:46.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.703+0000 7f15e8856700 1 -- 172.21.15.114:0/2036141051 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15d4003e40 con 0x7f15e4100bc0 2024-02-19T19:04:46.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.703+0000 7f15e9858700 1 -- 172.21.15.114:0/2036141051 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e419bae0 con 0x7f15e4100bc0 2024-02-19T19:04:46.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.703+0000 7f15e9858700 1 -- 172.21.15.114:0/2036141051 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e419bfe0 con 0x7f15e4100bc0 2024-02-19T19:04:46.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.704+0000 7f15e9858700 1 -- 172.21.15.114:0/2036141051 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e4109740 con 0x7f15e4100bc0 2024-02-19T19:04:46.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.704+0000 7f15e8856700 1 -- 172.21.15.114:0/2036141051 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 36) v1 ==== 91597+0+0 (secure 0 0 0) 0x7f15d400cba0 con 0x7f15e4100bc0 2024-02-19T19:04:46.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.705+0000 7f15e8856700 1 --2- 172.21.15.114:0/2036141051 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f15cc06d970 0x7f15cc06fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:46.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.705+0000 7f15e8856700 1 -- 172.21.15.114:0/2036141051 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f15d4014070 con 0x7f15e4100bc0 2024-02-19T19:04:46.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.705+0000 7f15e27fc700 1 --2- 172.21.15.114:0/2036141051 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f15cc06d970 0x7f15cc06fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:46.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.706+0000 7f15e27fc700 1 --2- 172.21.15.114:0/2036141051 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f15cc06d970 0x7f15cc06fe30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f15d8009fd0 tx=0x7f15d8005bc0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:46.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:46.710+0000 7f15e8856700 1 -- 172.21.15.114:0/2036141051 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f15d405ac70 con 0x7f15e4100bc0 2024-02-19T19:04:47.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.015+0000 7f15e9858700 1 -- 172.21.15.114:0/2036141051 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f15e419c6e0 con 0x7f15e4100bc0 2024-02-19T19:04:47.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.016+0000 7f15e8856700 1 -- 172.21.15.114:0/2036141051 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+688 (secure 0 0 0) 0x7f15d405a800 con 0x7f15e4100bc0 2024-02-19T19:04:47.036 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:04:47.036 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-02-19T19:04:47.036 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:04:47.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:04:47.036 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-02-19T19:04:47.037 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:04:47.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:04:47.037 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-02-19T19:04:47.037 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-02-19T19:04:47.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:04:47.037 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-02-19T19:04:47.037 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:04:47.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:04:47.037 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-02-19T19:04:47.037 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 10, 2024-02-19T19:04:47.037 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 4 2024-02-19T19:04:47.037 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-02-19T19:04:47.037 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:04:47.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.018+0000 7f15e9858700 1 -- 172.21.15.114:0/2036141051 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f15cc06d970 msgr2=0x7f15cc06fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:47.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.018+0000 7f15e9858700 1 --2- 172.21.15.114:0/2036141051 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f15cc06d970 0x7f15cc06fe30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f15d8009fd0 tx=0x7f15d8005bc0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.018+0000 7f15e9858700 1 -- 172.21.15.114:0/2036141051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15e4100bc0 msgr2=0x7f15e4196390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:47.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.018+0000 7f15e9858700 1 --2- 172.21.15.114:0/2036141051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15e4100bc0 0x7f15e4196390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f15d400e8d0 tx=0x7f15d400ec90 comp rx=0 tx=0).stop 2024-02-19T19:04:47.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.018+0000 7f15e9858700 1 -- 172.21.15.114:0/2036141051 shutdown_connections 2024-02-19T19:04:47.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.018+0000 7f15e9858700 1 --2- 172.21.15.114:0/2036141051 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f15cc06d970 0x7f15cc06fe30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.018+0000 7f15e9858700 1 --2- 172.21.15.114:0/2036141051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15e4100bc0 0x7f15e4196390 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.018+0000 7f15e9858700 1 --2- 172.21.15.114:0/2036141051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15e4101dc0 0x7f15e41968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.018+0000 7f15e9858700 1 -- 172.21.15.114:0/2036141051 >> 172.21.15.114:0/2036141051 conn(0x7f15e40fc140 msgr2=0x7f15e4104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:47.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.019+0000 7f15e9858700 1 -- 172.21.15.114:0/2036141051 shutdown_connections 2024-02-19T19:04:47.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.019+0000 7f15e9858700 1 -- 172.21.15.114:0/2036141051 wait complete. 2024-02-19T19:04:47.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.123+0000 7f06b45a6700 1 -- 172.21.15.114:0/1554610398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06ac104010 msgr2=0x7f06ac106400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:47.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.123+0000 7f06b45a6700 1 --2- 172.21.15.114:0/1554610398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06ac104010 0x7f06ac106400 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f069c009910 tx=0x7f069c009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:47.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.123+0000 7f06b45a6700 1 -- 172.21.15.114:0/1554610398 shutdown_connections 2024-02-19T19:04:47.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.123+0000 7f06b45a6700 1 --2- 172.21.15.114:0/1554610398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06ac104010 0x7f06ac106400 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.123+0000 7f06b45a6700 1 --2- 172.21.15.114:0/1554610398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06ac1016e0 0x7f06ac103ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.123+0000 7f06b45a6700 1 -- 172.21.15.114:0/1554610398 >> 172.21.15.114:0/1554610398 conn(0x7f06ac0fb0b0 msgr2=0x7f06ac0fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:47.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.123+0000 7f06b45a6700 1 -- 172.21.15.114:0/1554610398 shutdown_connections 2024-02-19T19:04:47.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.123+0000 7f06b45a6700 1 -- 172.21.15.114:0/1554610398 wait complete. 2024-02-19T19:04:47.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.123+0000 7f06b45a6700 1 Processor -- start 2024-02-19T19:04:47.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.123+0000 7f06b45a6700 1 -- start start 2024-02-19T19:04:47.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.124+0000 7f06b45a6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06ac1016e0 0x7f06ac196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:47.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.124+0000 7f06b45a6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06ac104010 0x7f06ac196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:47.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.124+0000 7f06b45a6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac196e90 con 0x7f06ac1016e0 2024-02-19T19:04:47.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.124+0000 7f06b45a6700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac196fd0 con 0x7f06ac104010 2024-02-19T19:04:47.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.124+0000 7f06b2342700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06ac1016e0 0x7f06ac196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:47.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.124+0000 7f06b2342700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06ac1016e0 0x7f06ac196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55336/0 (socket says 172.21.15.114:55336) 2024-02-19T19:04:47.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.124+0000 7f06b2342700 1 -- 172.21.15.114:0/2485343386 learned_addr learned my addr 172.21.15.114:0/2485343386 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:04:47.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.124+0000 7f06b1b41700 1 --2- 172.21.15.114:0/2485343386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06ac104010 0x7f06ac196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:47.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.124+0000 7f06b2342700 1 -- 172.21.15.114:0/2485343386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06ac104010 msgr2=0x7f06ac196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:47.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.124+0000 7f06b2342700 1 --2- 172.21.15.114:0/2485343386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06ac104010 0x7f06ac196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.124+0000 7f06b2342700 1 -- 172.21.15.114:0/2485343386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c0095f0 con 0x7f06ac1016e0 2024-02-19T19:04:47.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.124+0000 7f06b2342700 1 --2- 172.21.15.114:0/2485343386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06ac1016e0 0x7f06ac196330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f06a800e920 tx=0x7f06a800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:47.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.124+0000 7f06a37fe700 1 -- 172.21.15.114:0/2485343386 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06a8009790 con 0x7f06ac1016e0 2024-02-19T19:04:47.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.124+0000 7f06a37fe700 1 -- 172.21.15.114:0/2485343386 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f06a800cba0 con 0x7f06ac1016e0 2024-02-19T19:04:47.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.124+0000 7f06b45a6700 1 -- 172.21.15.114:0/2485343386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ac19ba80 con 0x7f06ac1016e0 2024-02-19T19:04:47.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.124+0000 7f06a37fe700 1 -- 172.21.15.114:0/2485343386 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06a8018630 con 0x7f06ac1016e0 2024-02-19T19:04:47.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.124+0000 7f06b45a6700 1 -- 172.21.15.114:0/2485343386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ac19bfd0 con 0x7f06ac1016e0 2024-02-19T19:04:47.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.125+0000 7f06b45a6700 1 -- 172.21.15.114:0/2485343386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ac190560 con 0x7f06ac1016e0 2024-02-19T19:04:47.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.126+0000 7f06a37fe700 1 -- 172.21.15.114:0/2485343386 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 36) v1 ==== 91597+0+0 (secure 0 0 0) 0x7f06a800cd10 con 0x7f06ac1016e0 2024-02-19T19:04:47.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.126+0000 7f06a37fe700 1 --2- 172.21.15.114:0/2485343386 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f069806d9e0 0x7f069806fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:47.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.126+0000 7f06a37fe700 1 -- 172.21.15.114:0/2485343386 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f06a8014070 con 0x7f06ac1016e0 2024-02-19T19:04:47.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.126+0000 7f06b1b41700 1 --2- 172.21.15.114:0/2485343386 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f069806d9e0 0x7f069806fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:47.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.127+0000 7f06b1b41700 1 --2- 172.21.15.114:0/2485343386 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f069806d9e0 0x7f069806fea0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f069c0059d0 tx=0x7f069c005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:47.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.131+0000 7f06a37fe700 1 -- 172.21.15.114:0/2485343386 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f06a8052d20 con 0x7f06ac1016e0 2024-02-19T19:04:47.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.380+0000 7f06b45a6700 1 -- 172.21.15.114:0/2485343386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f06ac02cec0 con 0x7f06ac1016e0 2024-02-19T19:04:47.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.380+0000 7f06a37fe700 1 -- 172.21.15.114:0/2485343386 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1871 (secure 0 0 0) 0x7f06a805a910 con 0x7f06ac1016e0 2024-02-19T19:04:47.400 INFO:teuthology.orchestra.run.smithi114.stdout:e18 2024-02-19T19:04:47.400 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:04:47.400 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:04:47.401 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-02-19T19:04:47.401 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:04:47.401 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:04:47.401 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-02-19T19:04:47.401 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 18 2024-02-19T19:04:47.401 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:04:47.401 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:04:47.401 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-02-19T19:00:21.203922+0000 2024-02-19T19:04:47.401 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-02-19T19:04:47.401 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-02-19T19:04:47.401 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-02-19T19:04:47.401 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-02-19T19:04:47.401 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-02-19T19:04:47.401 INFO:teuthology.orchestra.run.smithi114.stdout:max_xattr_size 65536 2024-02-19T19:04:47.402 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-02-19T19:04:47.402 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-02-19T19:04:47.402 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 47 2024-02-19T19:04:47.402 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:04:47.402 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 2 2024-02-19T19:04:47.402 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1 2024-02-19T19:04:47.402 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=14530,1=24269} 2024-02-19T19:04:47.402 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-02-19T19:04:47.402 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-02-19T19:04:47.402 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-02-19T19:04:47.402 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-02-19T19:04:47.402 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-02-19T19:04:47.402 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data enabled 2024-02-19T19:04:47.402 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-02-19T19:04:47.402 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-02-19T19:04:47.402 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 1 2024-02-19T19:04:47.403 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:04:47.403 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.vpoeop{0:14540} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:04:47.403 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.znhuvd{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:04:47.403 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.stnfnf{1:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:04:47.403 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:04:47.403 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:04:47.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.382+0000 7f06b45a6700 1 -- 172.21.15.114:0/2485343386 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f069806d9e0 msgr2=0x7f069806fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.382+0000 7f06b45a6700 1 --2- 172.21.15.114:0/2485343386 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f069806d9e0 0x7f069806fea0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f069c0059d0 tx=0x7f069c005940 comp rx=0 tx=0).stop 2024-02-19T19:04:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.382+0000 7f06b45a6700 1 -- 172.21.15.114:0/2485343386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06ac1016e0 msgr2=0x7f06ac196330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.382+0000 7f06b45a6700 1 --2- 172.21.15.114:0/2485343386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06ac1016e0 0x7f06ac196330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f06a800e920 tx=0x7f06a800ece0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.382+0000 7f06b45a6700 1 -- 172.21.15.114:0/2485343386 shutdown_connections 2024-02-19T19:04:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.382+0000 7f06b45a6700 1 --2- 172.21.15.114:0/2485343386 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f069806d9e0 0x7f069806fea0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.382+0000 7f06b45a6700 1 --2- 172.21.15.114:0/2485343386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06ac1016e0 0x7f06ac196330 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.382+0000 7f06b45a6700 1 --2- 172.21.15.114:0/2485343386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06ac104010 0x7f06ac196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.382+0000 7f06b45a6700 1 -- 172.21.15.114:0/2485343386 >> 172.21.15.114:0/2485343386 conn(0x7f06ac0fb0b0 msgr2=0x7f06ac0fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.382+0000 7f06b45a6700 1 -- 172.21.15.114:0/2485343386 shutdown_connections 2024-02-19T19:04:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.382+0000 7f06b45a6700 1 -- 172.21.15.114:0/2485343386 wait complete. 2024-02-19T19:04:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 18 2024-02-19T19:04:47.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.488+0000 7efe764c7700 1 -- 172.21.15.114:0/2770559756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe700ffab0 msgr2=0x7efe700ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:47.508 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.488+0000 7efe764c7700 1 --2- 172.21.15.114:0/2770559756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe700ffab0 0x7efe700ffed0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efe60009910 tx=0x7efe60009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:47.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.488+0000 7efe764c7700 1 -- 172.21.15.114:0/2770559756 shutdown_connections 2024-02-19T19:04:47.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.488+0000 7efe764c7700 1 --2- 172.21.15.114:0/2770559756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe70100cb0 0x7efe70101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.488+0000 7efe764c7700 1 --2- 172.21.15.114:0/2770559756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe700ffab0 0x7efe700ffed0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.488+0000 7efe764c7700 1 -- 172.21.15.114:0/2770559756 >> 172.21.15.114:0/2770559756 conn(0x7efe700fb050 msgr2=0x7efe700fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:47.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.488+0000 7efe764c7700 1 -- 172.21.15.114:0/2770559756 shutdown_connections 2024-02-19T19:04:47.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.488+0000 7efe764c7700 1 -- 172.21.15.114:0/2770559756 wait complete. 2024-02-19T19:04:47.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.489+0000 7efe764c7700 1 Processor -- start 2024-02-19T19:04:47.509 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.489+0000 7efe764c7700 1 -- start start 2024-02-19T19:04:47.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.489+0000 7efe764c7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe700ffab0 0x7efe70196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:47.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.489+0000 7efe764c7700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe70100cb0 0x7efe701968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:47.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.489+0000 7efe764c7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe70196ed0 con 0x7efe700ffab0 2024-02-19T19:04:47.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.489+0000 7efe764c7700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe70197010 con 0x7efe70100cb0 2024-02-19T19:04:47.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.489+0000 7efe6ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe700ffab0 0x7efe70196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:47.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.489+0000 7efe6ffff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe700ffab0 0x7efe70196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55354/0 (socket says 172.21.15.114:55354) 2024-02-19T19:04:47.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.489+0000 7efe6ffff700 1 -- 172.21.15.114:0/921300588 learned_addr learned my addr 172.21.15.114:0/921300588 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:04:47.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.489+0000 7efe6f7fe700 1 --2- 172.21.15.114:0/921300588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe70100cb0 0x7efe701968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:47.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.490+0000 7efe6ffff700 1 -- 172.21.15.114:0/921300588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe70100cb0 msgr2=0x7efe701968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:47.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.490+0000 7efe6ffff700 1 --2- 172.21.15.114:0/921300588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe70100cb0 0x7efe701968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.490+0000 7efe6ffff700 1 -- 172.21.15.114:0/921300588 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe600095f0 con 0x7efe700ffab0 2024-02-19T19:04:47.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.490+0000 7efe6ffff700 1 --2- 172.21.15.114:0/921300588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe700ffab0 0x7efe70196370 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efe60000c00 tx=0x7efe6000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:47.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.491+0000 7efe6d7fa700 1 -- 172.21.15.114:0/921300588 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe6001d070 con 0x7efe700ffab0 2024-02-19T19:04:47.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.491+0000 7efe6d7fa700 1 -- 172.21.15.114:0/921300588 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efe60005dc0 con 0x7efe700ffab0 2024-02-19T19:04:47.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.491+0000 7efe764c7700 1 -- 172.21.15.114:0/921300588 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7019ba60 con 0x7efe700ffab0 2024-02-19T19:04:47.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.492+0000 7efe6d7fa700 1 -- 172.21.15.114:0/921300588 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe60022470 con 0x7efe700ffab0 2024-02-19T19:04:47.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.492+0000 7efe764c7700 1 -- 172.21.15.114:0/921300588 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7019bf50 con 0x7efe700ffab0 2024-02-19T19:04:47.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.493+0000 7efe764c7700 1 -- 172.21.15.114:0/921300588 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe70108610 con 0x7efe700ffab0 2024-02-19T19:04:47.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.493+0000 7efe6d7fa700 1 -- 172.21.15.114:0/921300588 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 36) v1 ==== 91597+0+0 (secure 0 0 0) 0x7efe6000fa70 con 0x7efe700ffab0 2024-02-19T19:04:47.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.493+0000 7efe6d7fa700 1 --2- 172.21.15.114:0/921300588 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7efe5806d9e0 0x7efe5806fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:47.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.493+0000 7efe6d7fa700 1 -- 172.21.15.114:0/921300588 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7efe6008ebd0 con 0x7efe700ffab0 2024-02-19T19:04:47.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.493+0000 7efe6f7fe700 1 --2- 172.21.15.114:0/921300588 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7efe5806d9e0 0x7efe5806fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:47.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.494+0000 7efe6f7fe700 1 --2- 172.21.15.114:0/921300588 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7efe5806d9e0 0x7efe5806fea0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efe64009c60 tx=0x7efe64009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:47.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.498+0000 7efe6d7fa700 1 -- 172.21.15.114:0/921300588 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efe6005bd60 con 0x7efe700ffab0 2024-02-19T19:04:47.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.720+0000 7efe764c7700 1 -- 172.21.15.114:0/921300588 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efe70069390 con 0x7efe5806d9e0 2024-02-19T19:04:47.740 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:47 smithi114 ceph-mon[92983]: from='client.34184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:47.740 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:47 smithi114 ceph-mon[92983]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:47.740 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:47 smithi114 ceph-mon[92983]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:47.740 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:47 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/2036141051' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:04:47.741 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:47 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/2485343386' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:04:47.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.721+0000 7efe6d7fa700 1 -- 172.21.15.114:0/921300588 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7efe70069390 con 0x7efe5806d9e0 2024-02-19T19:04:47.741 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:04:47.741 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:04:47.741 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-02-19T19:04:47.741 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:04:47.742 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [ 2024-02-19T19:04:47.742 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr", 2024-02-19T19:04:47.742 INFO:teuthology.orchestra.run.smithi114.stdout: "mon" 2024-02-19T19:04:47.742 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-02-19T19:04:47.742 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "4/23 daemons upgraded", 2024-02-19T19:04:47.742 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "Currently upgrading crash daemons", 2024-02-19T19:04:47.742 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-02-19T19:04:47.742 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:04:47.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.724+0000 7efe764c7700 1 -- 172.21.15.114:0/921300588 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7efe5806d9e0 msgr2=0x7efe5806fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:47.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.724+0000 7efe764c7700 1 --2- 172.21.15.114:0/921300588 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7efe5806d9e0 0x7efe5806fea0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efe64009c60 tx=0x7efe64009260 comp rx=0 tx=0).stop 2024-02-19T19:04:47.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.724+0000 7efe764c7700 1 -- 172.21.15.114:0/921300588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe700ffab0 msgr2=0x7efe70196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:47.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.724+0000 7efe764c7700 1 --2- 172.21.15.114:0/921300588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe700ffab0 0x7efe70196370 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efe60000c00 tx=0x7efe6000f690 comp rx=0 tx=0).stop 2024-02-19T19:04:47.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.725+0000 7efe764c7700 1 -- 172.21.15.114:0/921300588 shutdown_connections 2024-02-19T19:04:47.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.725+0000 7efe764c7700 1 --2- 172.21.15.114:0/921300588 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7efe5806d9e0 0x7efe5806fea0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.725+0000 7efe764c7700 1 --2- 172.21.15.114:0/921300588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe700ffab0 0x7efe70196370 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.725+0000 7efe764c7700 1 --2- 172.21.15.114:0/921300588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe70100cb0 0x7efe701968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.725+0000 7efe764c7700 1 -- 172.21.15.114:0/921300588 >> 172.21.15.114:0/921300588 conn(0x7efe700fb050 msgr2=0x7efe70068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:47.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.725+0000 7efe764c7700 1 -- 172.21.15.114:0/921300588 shutdown_connections 2024-02-19T19:04:47.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.725+0000 7efe764c7700 1 -- 172.21.15.114:0/921300588 wait complete. 2024-02-19T19:04:47.857 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:47 smithi174 ceph-mon[87279]: from='client.34184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:47.857 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:47 smithi174 ceph-mon[87279]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:47.857 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:47 smithi174 ceph-mon[87279]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:47.857 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:47 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/2036141051' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:04:47.857 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:47 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/2485343386' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:04:47.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.840+0000 7f01d82f7700 1 -- 172.21.15.114:0/4234197175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d0101e00 msgr2=0x7f01d0102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:47.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.840+0000 7f01d82f7700 1 --2- 172.21.15.114:0/4234197175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d0101e00 0x7f01d0102260 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f01c4009910 tx=0x7f01c4009c20 comp rx=0 tx=0).stop 2024-02-19T19:04:47.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.840+0000 7f01d82f7700 1 -- 172.21.15.114:0/4234197175 shutdown_connections 2024-02-19T19:04:47.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.840+0000 7f01d82f7700 1 --2- 172.21.15.114:0/4234197175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d0101e00 0x7f01d0102260 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.840+0000 7f01d82f7700 1 --2- 172.21.15.114:0/4234197175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d0100c00 0x7f01d0101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.840+0000 7f01d82f7700 1 -- 172.21.15.114:0/4234197175 >> 172.21.15.114:0/4234197175 conn(0x7f01d00fc180 msgr2=0x7f01d00fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:47.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.840+0000 7f01d82f7700 1 -- 172.21.15.114:0/4234197175 shutdown_connections 2024-02-19T19:04:47.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.840+0000 7f01d82f7700 1 -- 172.21.15.114:0/4234197175 wait complete. 2024-02-19T19:04:47.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.840+0000 7f01d82f7700 1 Processor -- start 2024-02-19T19:04:47.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.840+0000 7f01d82f7700 1 -- start start 2024-02-19T19:04:47.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.841+0000 7f01d82f7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d0100c00 0x7f01d0196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:47.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.841+0000 7f01d82f7700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d0101e00 0x7f01d01968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:47.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.841+0000 7f01d82f7700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0196ee0 con 0x7f01d0100c00 2024-02-19T19:04:47.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.841+0000 7f01d82f7700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0197020 con 0x7f01d0101e00 2024-02-19T19:04:47.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.841+0000 7f01d6093700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d0100c00 0x7f01d0196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:47.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.841+0000 7f01d6093700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d0100c00 0x7f01d0196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55360/0 (socket says 172.21.15.114:55360) 2024-02-19T19:04:47.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.841+0000 7f01d6093700 1 -- 172.21.15.114:0/1354532194 learned_addr learned my addr 172.21.15.114:0/1354532194 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:04:47.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.841+0000 7f01d5892700 1 --2- 172.21.15.114:0/1354532194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d0101e00 0x7f01d01968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:47.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.841+0000 7f01d6093700 1 -- 172.21.15.114:0/1354532194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d0101e00 msgr2=0x7f01d01968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:47.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.841+0000 7f01d6093700 1 --2- 172.21.15.114:0/1354532194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d0101e00 0x7f01d01968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:47.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.841+0000 7f01d6093700 1 -- 172.21.15.114:0/1354532194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c40095f0 con 0x7f01d0100c00 2024-02-19T19:04:47.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.841+0000 7f01d6093700 1 --2- 172.21.15.114:0/1354532194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d0100c00 0x7f01d0196380 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f01c8007e90 tx=0x7f01c800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:47.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.841+0000 7f01cf7fe700 1 -- 172.21.15.114:0/1354532194 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01c8017070 con 0x7f01d0100c00 2024-02-19T19:04:47.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.841+0000 7f01d82f7700 1 -- 172.21.15.114:0/1354532194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d019bad0 con 0x7f01d0100c00 2024-02-19T19:04:47.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.841+0000 7f01cf7fe700 1 -- 172.21.15.114:0/1354532194 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f01c800f040 con 0x7f01d0100c00 2024-02-19T19:04:47.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.841+0000 7f01cf7fe700 1 -- 172.21.15.114:0/1354532194 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01c8004820 con 0x7f01d0100c00 2024-02-19T19:04:47.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.841+0000 7f01d82f7700 1 -- 172.21.15.114:0/1354532194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d019c020 con 0x7f01d0100c00 2024-02-19T19:04:47.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.842+0000 7f01d82f7700 1 -- 172.21.15.114:0/1354532194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d0109760 con 0x7f01d0100c00 2024-02-19T19:04:47.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.843+0000 7f01cf7fe700 1 -- 172.21.15.114:0/1354532194 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 36) v1 ==== 91597+0+0 (secure 0 0 0) 0x7f01c8003a00 con 0x7f01d0100c00 2024-02-19T19:04:47.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.843+0000 7f01cf7fe700 1 --2- 172.21.15.114:0/1354532194 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f01b806d9e0 0x7f01b806fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:04:47.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.843+0000 7f01d5892700 1 --2- 172.21.15.114:0/1354532194 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f01b806d9e0 0x7f01b806fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:04:47.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.843+0000 7f01cf7fe700 1 -- 172.21.15.114:0/1354532194 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f01c808d0a0 con 0x7f01d0100c00 2024-02-19T19:04:47.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.844+0000 7f01d5892700 1 --2- 172.21.15.114:0/1354532194 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f01b806d9e0 0x7f01b806fea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f01c4006010 tx=0x7f01c400b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:04:47.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:47.848+0000 7f01cf7fe700 1 -- 172.21.15.114:0/1354532194 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f01c805a1b0 con 0x7f01d0100c00 2024-02-19T19:04:48.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:48.135+0000 7f01d82f7700 1 -- 172.21.15.114:0/1354532194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f01d0109950 con 0x7f01d0100c00 2024-02-19T19:04:48.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:48.135+0000 7f01cf7fe700 1 -- 172.21.15.114:0/1354532194 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f01c8059d40 con 0x7f01d0100c00 2024-02-19T19:04:48.155 INFO:teuthology.orchestra.run.smithi114.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:04:48.155 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:04:48.155 INFO:teuthology.orchestra.run.smithi114.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:04:48.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:48.137+0000 7f01d82f7700 1 -- 172.21.15.114:0/1354532194 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f01b806d9e0 msgr2=0x7f01b806fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:48.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:48.137+0000 7f01d82f7700 1 --2- 172.21.15.114:0/1354532194 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f01b806d9e0 0x7f01b806fea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f01c4006010 tx=0x7f01c400b540 comp rx=0 tx=0).stop 2024-02-19T19:04:48.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:48.137+0000 7f01d82f7700 1 -- 172.21.15.114:0/1354532194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d0100c00 msgr2=0x7f01d0196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:04:48.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:48.137+0000 7f01d82f7700 1 --2- 172.21.15.114:0/1354532194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d0100c00 0x7f01d0196380 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f01c8007e90 tx=0x7f01c800d3b0 comp rx=0 tx=0).stop 2024-02-19T19:04:48.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:48.137+0000 7f01d82f7700 1 -- 172.21.15.114:0/1354532194 shutdown_connections 2024-02-19T19:04:48.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:48.137+0000 7f01d82f7700 1 --2- 172.21.15.114:0/1354532194 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f01b806d9e0 0x7f01b806fea0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:48.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:48.137+0000 7f01d82f7700 1 --2- 172.21.15.114:0/1354532194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01d0100c00 0x7f01d0196380 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:48.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:48.137+0000 7f01d82f7700 1 --2- 172.21.15.114:0/1354532194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d0101e00 0x7f01d01968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:04:48.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:48.137+0000 7f01d82f7700 1 -- 172.21.15.114:0/1354532194 >> 172.21.15.114:0/1354532194 conn(0x7f01d00fc180 msgr2=0x7f01d0105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:04:48.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:48.137+0000 7f01d82f7700 1 -- 172.21.15.114:0/1354532194 shutdown_connections 2024-02-19T19:04:48.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:04:48.137+0000 7f01d82f7700 1 -- 172.21.15.114:0/1354532194 wait complete. 2024-02-19T19:04:48.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:48 smithi114 ceph-mon[92983]: from='client.34204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:48.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:48 smithi114 ceph-mon[92983]: pgmap v37: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:48.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:48 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/1354532194' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:04:48.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:48 smithi174 ceph-mon[87279]: from='client.34204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:04:48.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:48 smithi174 ceph-mon[87279]: pgmap v37: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:48.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:48 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/1354532194' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:04:50.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:49 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:50.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:49 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:50.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:49 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:50.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:49 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:50.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:50 smithi174 ceph-mon[87279]: pgmap v38: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:50.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:50 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:50.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:50 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:51.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:50 smithi114 ceph-mon[92983]: pgmap v38: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:51.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:50 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:51.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:50 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:52.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:51 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:52.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:51 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:52.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:51 smithi114 ceph-mon[92983]: pgmap v39: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:52.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:51 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:52.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:51 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:52.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:51 smithi174 ceph-mon[87279]: pgmap v39: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:53.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:52 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:53.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:52 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:53.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:52 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:53.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:52 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:54.432 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:54 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:54.432 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:54 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:54.432 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:54 smithi114 ceph-mon[92983]: pgmap v40: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:54.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:54 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:54.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:54 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:54.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:54 smithi174 ceph-mon[87279]: pgmap v40: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:55.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:55 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:04:55.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:55 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:04:57.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:57.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:56 smithi114 ceph-mon[92983]: pgmap v41: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:57.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:57.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:57.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:04:57.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:57.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:04:57.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:04:57.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:04:57.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:04:57.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:57.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi114"}]: dispatch 2024-02-19T19:04:57.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi114"}]': finished 2024-02-19T19:04:57.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi174"}]: dispatch 2024-02-19T19:04:57.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi174"}]': finished 2024-02-19T19:04:57.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-19T19:04:57.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:57.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:56 smithi174 ceph-mon[87279]: pgmap v41: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:04:57.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:57.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:57.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:04:57.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:57.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:04:57.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:04:57.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:04:57.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:04:57.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:57.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi114"}]: dispatch 2024-02-19T19:04:57.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi114"}]': finished 2024-02-19T19:04:57.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi174"}]: dispatch 2024-02-19T19:04:57.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi174"}]': finished 2024-02-19T19:04:57.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-19T19:04:58.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:57 smithi114 ceph-mon[92983]: Upgrade: Setting container_image for all crash 2024-02-19T19:04:58.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:57 smithi114 ceph-mon[92983]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-19T19:04:58.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:57 smithi114 ceph-mon[92983]: Upgrade: osd.0 is safe to restart 2024-02-19T19:04:58.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:57 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:58.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:57 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-19T19:04:58.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:57 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:58.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:57 smithi174 ceph-mon[87279]: Upgrade: Setting container_image for all crash 2024-02-19T19:04:58.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:57 smithi174 ceph-mon[87279]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-19T19:04:58.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:57 smithi174 ceph-mon[87279]: Upgrade: osd.0 is safe to restart 2024-02-19T19:04:58.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:57 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:04:58.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:57 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-19T19:04:58.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:57 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:04:58.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:58 smithi114 ceph-mon[92983]: Upgrade: Updating osd.0 2024-02-19T19:04:58.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:58 smithi114 ceph-mon[92983]: Deploying daemon osd.0 on smithi114 2024-02-19T19:04:58.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:04:58 smithi114 ceph-mon[92983]: pgmap v42: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:04:59.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:58 smithi174 ceph-mon[87279]: Upgrade: Updating osd.0 2024-02-19T19:04:59.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:58 smithi174 ceph-mon[87279]: Deploying daemon osd.0 on smithi114 2024-02-19T19:04:59.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:04:58 smithi174 ceph-mon[87279]: pgmap v42: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:05:01.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:00 smithi114 ceph-mon[92983]: pgmap v43: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:05:01.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:00 smithi174 ceph-mon[87279]: pgmap v43: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:05:01.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:01 smithi114 systemd[1]: Stopping Ceph osd.0 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T19:05:02.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:01 smithi114 ceph-mon[92983]: osd.0 marked itself down and dead 2024-02-19T19:05:02.039 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[48069]: 2024-02-19T19:05:01.718+0000 7fef6cb20700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T19:05:02.039 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[48069]: 2024-02-19T19:05:01.718+0000 7fef6cb20700 -1 osd.0 50 *** Got signal Terminated *** 2024-02-19T19:05:02.039 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[48069]: 2024-02-19T19:05:01.718+0000 7fef6cb20700 -1 osd.0 50 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T19:05:02.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:01 smithi174 ceph-mon[87279]: osd.0 marked itself down and dead 2024-02-19T19:05:02.431 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:02 smithi114 podman[101407]: 2024-02-19 19:05:02.11650991 +0000 UTC m=+0.514816654 container died 990afc982b6c39e8fbcf3de49d43b561316e39f7e36c3511c051a82eb2bc0dae (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-02-19T19:05:02.773 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:02 smithi114 podman[101407]: 2024-02-19 19:05:02.585897459 +0000 UTC m=+0.984204267 container cleanup 990afc982b6c39e8fbcf3de49d43b561316e39f7e36c3511c051a82eb2bc0dae (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, RELEASE=HEAD) 2024-02-19T19:05:02.773 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:02 smithi114 bash[101407]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0 2024-02-19T19:05:03.038 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:02 smithi114 podman[101423]: 2024-02-19 19:05:02.887618439 +0000 UTC m=+0.766620155 container remove 990afc982b6c39e8fbcf3de49d43b561316e39f7e36c3511c051a82eb2bc0dae (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1) 2024-02-19T19:05:03.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:02 smithi114 ceph-mon[92983]: pgmap v44: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:05:03.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:02 smithi114 ceph-mon[92983]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:05:03.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:02 smithi114 ceph-mon[92983]: osdmap e51: 6 total, 5 up, 6 in 2024-02-19T19:05:03.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:02 smithi174 ceph-mon[87279]: pgmap v44: 65 pgs: 65 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:05:03.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:02 smithi174 ceph-mon[87279]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:05:03.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:02 smithi174 ceph-mon[87279]: osdmap e51: 6 total, 5 up, 6 in 2024-02-19T19:05:03.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:03 smithi114 podman[101519]: 2024-02-19 19:05:03.141804251 +0000 UTC m=+0.018514614 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:05:03.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:03 smithi114 podman[101519]: 2024-02-19 19:05:03.263944453 +0000 UTC m=+0.140654854 container create 6735e8d5322f1b0c387804c586f0bfdf963f47a7daa52f31d4b4c7d73f4bd890 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:05:03.857 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:03 smithi114 ceph-mon[92983]: osdmap e52: 6 total, 5 up, 6 in 2024-02-19T19:05:03.857 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:03 smithi114 podman[101519]: 2024-02-19 19:05:03.536955869 +0000 UTC m=+0.413666233 container init 6735e8d5322f1b0c387804c586f0bfdf963f47a7daa52f31d4b4c7d73f4bd890 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, ceph=True) 2024-02-19T19:05:03.857 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:03 smithi114 podman[101519]: 2024-02-19 19:05:03.540604783 +0000 UTC m=+0.417315151 container start 6735e8d5322f1b0c387804c586f0bfdf963f47a7daa52f31d4b4c7d73f4bd890 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0) 2024-02-19T19:05:03.857 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:03 smithi114 podman[101519]: 2024-02-19 19:05:03.641781466 +0000 UTC m=+0.518492004 container attach 6735e8d5322f1b0c387804c586f0bfdf963f47a7daa52f31d4b4c7d73f4bd890 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:05:04.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:03 smithi174 ceph-mon[87279]: osdmap e52: 6 total, 5 up, 6 in 2024-02-19T19:05:04.287 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:03 smithi114 podman[101519]: 2024-02-19 19:05:03.838177258 +0000 UTC m=+0.714887620 container died 6735e8d5322f1b0c387804c586f0bfdf963f47a7daa52f31d4b4c7d73f4bd890 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T19:05:04.945 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:04 smithi114 ceph-mon[92983]: pgmap v47: 65 pgs: 9 stale+active+clean, 56 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:05:04.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:04 smithi114 podman[101545]: 2024-02-19 19:05:04.521048377 +0000 UTC m=+0.677502162 container remove 6735e8d5322f1b0c387804c586f0bfdf963f47a7daa52f31d4b4c7d73f4bd890 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) 2024-02-19T19:05:04.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:04 smithi114 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-19T19:05:04.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:04 smithi114 systemd[1]: Stopped Ceph osd.0 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T19:05:04.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:04 smithi114 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.0.service: Consumed 4.696s CPU time. 2024-02-19T19:05:04.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:04 smithi114 systemd[1]: Starting Ceph osd.0 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T19:05:05.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:04 smithi174 ceph-mon[87279]: pgmap v47: 65 pgs: 9 stale+active+clean, 56 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-02-19T19:05:05.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:04 smithi114 podman[101647]: 2024-02-19 19:05:04.885292577 +0000 UTC m=+0.017682440 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:05:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 podman[101647]: 2024-02-19 19:05:05.006481468 +0000 UTC m=+0.138871326 container create 48598249b44099f4d22cf35c97e2627b1c8d8c75783e957be271a2efb57b73ac (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-activate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.vendor=CentOS) 2024-02-19T19:05:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 podman[101647]: 2024-02-19 19:05:05.250712568 +0000 UTC m=+0.383102431 container init 48598249b44099f4d22cf35c97e2627b1c8d8c75783e957be271a2efb57b73ac (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-activate, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, maintainer=Guillaume Abrioux ) 2024-02-19T19:05:05.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 podman[101647]: 2024-02-19 19:05:05.253938912 +0000 UTC m=+0.386328778 container start 48598249b44099f4d22cf35c97e2627b1c8d8c75783e957be271a2efb57b73ac (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-activate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0) 2024-02-19T19:05:05.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 podman[101647]: 2024-02-19 19:05:05.334243426 +0000 UTC m=+0.466633296 container attach 48598249b44099f4d22cf35c97e2627b1c8d8c75783e957be271a2efb57b73ac (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-activate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T19:05:06.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:05 smithi114 ceph-mon[92983]: pgmap v48: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 2 op/s; 25/129 objects degraded (19.380%) 2024-02-19T19:05:06.039 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-activate[101662]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-19T19:05:06.039 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-activate[101662]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-19T19:05:06.039 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 bash[101647]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-19T19:05:06.039 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 bash[101647]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-19T19:05:06.039 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 bash[101647]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-19T19:05:06.039 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 bash[101647]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-19T19:05:06.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 bash[101647]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-19T19:05:06.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 bash[101647]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-19T19:05:06.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 bash[101647]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-19T19:05:06.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-activate[101662]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-19T19:05:06.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-activate[101662]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-19T19:05:06.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-activate[101662]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-19T19:05:06.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-activate[101662]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-19T19:05:06.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-activate[101662]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-19T19:05:06.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:05 smithi114 podman[101647]: 2024-02-19 19:05:05.740568091 +0000 UTC m=+0.872957950 container died 48598249b44099f4d22cf35c97e2627b1c8d8c75783e957be271a2efb57b73ac (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-02-19T19:05:06.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:05 smithi174 ceph-mon[87279]: pgmap v48: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 2 op/s; 25/129 objects degraded (19.380%) 2024-02-19T19:05:06.789 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:06 smithi114 podman[101786]: 2024-02-19 19:05:06.489569097 +0000 UTC m=+0.738575075 container remove 48598249b44099f4d22cf35c97e2627b1c8d8c75783e957be271a2efb57b73ac (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-activate, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-02-19T19:05:06.789 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:06 smithi114 podman[101849]: 2024-02-19 19:05:06.607485716 +0000 UTC m=+0.017921822 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:05:06.789 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:06 smithi114 podman[101849]: 2024-02-19 19:05:06.715468931 +0000 UTC m=+0.125905031 container create 764769439e0ec1441476971554e8740debb987b61e0be6f8185311fc190106b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-19T19:05:07.079 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:06 smithi114 ceph-mon[92983]: Health check failed: Degraded data redundancy: 25/129 objects degraded (19.380%), 13 pgs degraded (PG_DEGRADED) 2024-02-19T19:05:07.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:06 smithi114 podman[101849]: 2024-02-19 19:05:06.959720186 +0000 UTC m=+0.370156287 container init 764769439e0ec1441476971554e8740debb987b61e0be6f8185311fc190106b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:05:07.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:06 smithi114 podman[101849]: 2024-02-19 19:05:06.962826005 +0000 UTC m=+0.373262113 container start 764769439e0ec1441476971554e8740debb987b61e0be6f8185311fc190106b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-02-19T19:05:07.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:06 smithi114 ceph-osd[101868]: -- 172.21.15.114:0/2753534260 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56054f63b1e0 con 0x56054f60cc00 2024-02-19T19:05:07.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:06 smithi174 ceph-mon[87279]: Health check failed: Degraded data redundancy: 25/129 objects degraded (19.380%), 13 pgs degraded (PG_DEGRADED) 2024-02-19T19:05:07.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:07 smithi114 bash[101849]: 764769439e0ec1441476971554e8740debb987b61e0be6f8185311fc190106b9 2024-02-19T19:05:07.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:07 smithi114 systemd[1]: Started Ceph osd.0 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T19:05:07.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:05:07.506+0000 7f766126c7c0 -1 Falling back to public interface 2024-02-19T19:05:08.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:08 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:08.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:08 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:08.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:08 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:08.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:08 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:08.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:08 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:08.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:08 smithi114 ceph-mon[92983]: pgmap v49: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 2 op/s; 25/129 objects degraded (19.380%) 2024-02-19T19:05:08.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:08 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:08.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:08 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:08.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:08 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:08.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:08 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:08.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:08 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:08.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:08 smithi174 ceph-mon[87279]: pgmap v49: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 2 op/s; 25/129 objects degraded (19.380%) 2024-02-19T19:05:10.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:09 smithi174 ceph-mon[87279]: pgmap v50: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 25/129 objects degraded (19.380%) 2024-02-19T19:05:10.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:09 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:10.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:09 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:05:10.277 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:09 smithi114 ceph-mon[92983]: pgmap v50: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 593 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 25/129 objects degraded (19.380%) 2024-02-19T19:05:10.277 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:09 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:10.277 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:09 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:05:11.288 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:05:10.963+0000 7f766126c7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-02-19T19:05:11.288 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:05:10.991+0000 7f766126c7c0 -1 osd.0 50 log_to_monitors true 2024-02-19T19:05:11.969 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:11 smithi114 ceph-mon[92983]: mgrmap e37: smithi114.pcvtrk(active, since 91s), standbys: smithi174.lydvnj 2024-02-19T19:05:11.969 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:11 smithi114 ceph-mon[92983]: from='osd.0 [v2:172.21.15.114:6802/410495267,v1:172.21.15.114:6803/410495267]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-19T19:05:11.969 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:11 smithi114 ceph-mon[92983]: pgmap v51: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 594 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 307 B/s wr, 2 op/s; 25/129 objects degraded (19.380%) 2024-02-19T19:05:12.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:11 smithi174 ceph-mon[87279]: mgrmap e37: smithi114.pcvtrk(active, since 91s), standbys: smithi174.lydvnj 2024-02-19T19:05:12.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:11 smithi174 ceph-mon[87279]: from='osd.0 [v2:172.21.15.114:6802/410495267,v1:172.21.15.114:6803/410495267]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-19T19:05:12.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:11 smithi174 ceph-mon[87279]: pgmap v51: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 594 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 409 B/s rd, 307 B/s wr, 2 op/s; 25/129 objects degraded (19.380%) 2024-02-19T19:05:13.039 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:05:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:05:12.685+0000 7f764e6bc700 -1 osd.0 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T19:05:13.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:12 smithi114 ceph-mon[92983]: from='osd.0 [v2:172.21.15.114:6802/410495267,v1:172.21.15.114:6803/410495267]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-19T19:05:13.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:12 smithi114 ceph-mon[92983]: osdmap e53: 6 total, 5 up, 6 in 2024-02-19T19:05:13.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:12 smithi114 ceph-mon[92983]: from='osd.0 [v2:172.21.15.114:6802/410495267,v1:172.21.15.114:6803/410495267]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-02-19T19:05:13.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:12 smithi114 ceph-mon[92983]: from='osd.0 [v2:172.21.15.114:6802/410495267,v1:172.21.15.114:6803/410495267]' entity='osd.0' 2024-02-19T19:05:13.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:13.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:13.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:12 smithi174 ceph-mon[87279]: from='osd.0 [v2:172.21.15.114:6802/410495267,v1:172.21.15.114:6803/410495267]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-19T19:05:13.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:12 smithi174 ceph-mon[87279]: osdmap e53: 6 total, 5 up, 6 in 2024-02-19T19:05:13.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:12 smithi174 ceph-mon[87279]: from='osd.0 [v2:172.21.15.114:6802/410495267,v1:172.21.15.114:6803/410495267]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-02-19T19:05:13.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:12 smithi174 ceph-mon[87279]: from='osd.0 [v2:172.21.15.114:6802/410495267,v1:172.21.15.114:6803/410495267]' entity='osd.0' 2024-02-19T19:05:13.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:13.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:14.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:13 smithi114 ceph-mon[92983]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:05:14.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:13 smithi114 ceph-mon[92983]: osd.0 [v2:172.21.15.114:6802/410495267,v1:172.21.15.114:6803/410495267] boot 2024-02-19T19:05:14.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:13 smithi114 ceph-mon[92983]: osdmap e54: 6 total, 6 up, 6 in 2024-02-19T19:05:14.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:13 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:05:14.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:13 smithi114 ceph-mon[92983]: pgmap v54: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 594 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 383 B/s wr, 2 op/s; 25/129 objects degraded (19.380%) 2024-02-19T19:05:14.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:13 smithi174 ceph-mon[87279]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:05:14.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:13 smithi174 ceph-mon[87279]: osd.0 [v2:172.21.15.114:6802/410495267,v1:172.21.15.114:6803/410495267] boot 2024-02-19T19:05:14.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:13 smithi174 ceph-mon[87279]: osdmap e54: 6 total, 6 up, 6 in 2024-02-19T19:05:14.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:13 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:05:14.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:13 smithi174 ceph-mon[87279]: pgmap v54: 65 pgs: 13 active+undersized+degraded, 21 active+undersized, 31 active+clean; 594 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 383 B/s wr, 2 op/s; 25/129 objects degraded (19.380%) 2024-02-19T19:05:15.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:15.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:15.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:14 smithi114 ceph-mon[92983]: osdmap e55: 6 total, 6 up, 6 in 2024-02-19T19:05:15.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:15.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:15.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:14 smithi174 ceph-mon[87279]: osdmap e55: 6 total, 6 up, 6 in 2024-02-19T19:05:16.125 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:15 smithi114 ceph-mon[92983]: pgmap v56: 65 pgs: 2 active+undersized+degraded, 7 active+undersized, 56 active+clean; 594 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s; 5/129 objects degraded (3.876%) 2024-02-19T19:05:16.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:15 smithi174 ceph-mon[87279]: pgmap v56: 65 pgs: 2 active+undersized+degraded, 7 active+undersized, 56 active+clean; 594 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s; 5/129 objects degraded (3.876%) 2024-02-19T19:05:17.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:16 smithi114 ceph-mon[92983]: Health check update: Degraded data redundancy: 5/129 objects degraded (3.876%), 2 pgs degraded (PG_DEGRADED) 2024-02-19T19:05:17.394 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:16 smithi174 ceph-mon[87279]: Health check update: Degraded data redundancy: 5/129 objects degraded (3.876%), 2 pgs degraded (PG_DEGRADED) 2024-02-19T19:05:18.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.244+0000 7f448874b700 1 -- 172.21.15.114:0/1417504872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4480101f70 msgr2=0x7f44801023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:18.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.244+0000 7f448874b700 1 --2- 172.21.15.114:0/1417504872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4480101f70 0x7f44801023d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4470009910 tx=0x7f4470009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:18.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.244+0000 7f448874b700 1 -- 172.21.15.114:0/1417504872 shutdown_connections 2024-02-19T19:05:18.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.244+0000 7f448874b700 1 --2- 172.21.15.114:0/1417504872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4480101f70 0x7f44801023d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:18.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.244+0000 7f448874b700 1 --2- 172.21.15.114:0/1417504872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44800feef0 0x7f44800ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:18.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.244+0000 7f448874b700 1 -- 172.21.15.114:0/1417504872 >> 172.21.15.114:0/1417504872 conn(0x7f44800fa810 msgr2=0x7f44800fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:18.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.244+0000 7f448874b700 1 -- 172.21.15.114:0/1417504872 shutdown_connections 2024-02-19T19:05:18.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.244+0000 7f448874b700 1 -- 172.21.15.114:0/1417504872 wait complete. 2024-02-19T19:05:18.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.245+0000 7f448874b700 1 Processor -- start 2024-02-19T19:05:18.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.245+0000 7f448874b700 1 -- start start 2024-02-19T19:05:18.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.245+0000 7f448874b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44800feef0 0x7f4480196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:18.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.245+0000 7f448874b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4480101f70 0x7f4480196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:18.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.245+0000 7f448874b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4480196fa0 con 0x7f4480101f70 2024-02-19T19:05:18.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.245+0000 7f448874b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44801970e0 con 0x7f44800feef0 2024-02-19T19:05:18.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.245+0000 7f4485ce6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4480101f70 0x7f4480196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:18.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.245+0000 7f4485ce6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4480101f70 0x7f4480196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34142/0 (socket says 172.21.15.114:34142) 2024-02-19T19:05:18.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.245+0000 7f4485ce6700 1 -- 172.21.15.114:0/20140583 learned_addr learned my addr 172.21.15.114:0/20140583 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:05:18.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.246+0000 7f4485ce6700 1 -- 172.21.15.114:0/20140583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44800feef0 msgr2=0x7f4480196440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:18.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.246+0000 7f4485ce6700 1 --2- 172.21.15.114:0/20140583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44800feef0 0x7f4480196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:18.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.246+0000 7f4485ce6700 1 -- 172.21.15.114:0/20140583 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44700095f0 con 0x7f4480101f70 2024-02-19T19:05:18.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.246+0000 7f4485ce6700 1 --2- 172.21.15.114:0/20140583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4480101f70 0x7f4480196980 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4470006010 tx=0x7f447000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:18.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.246+0000 7f44777fe700 1 -- 172.21.15.114:0/20140583 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f447001d070 con 0x7f4480101f70 2024-02-19T19:05:18.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.246+0000 7f44777fe700 1 -- 172.21.15.114:0/20140583 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f447000f460 con 0x7f4480101f70 2024-02-19T19:05:18.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.246+0000 7f448874b700 1 -- 172.21.15.114:0/20140583 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f448019bb30 con 0x7f4480101f70 2024-02-19T19:05:18.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.246+0000 7f44777fe700 1 -- 172.21.15.114:0/20140583 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4470021640 con 0x7f4480101f70 2024-02-19T19:05:18.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.246+0000 7f448874b700 1 -- 172.21.15.114:0/20140583 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f448019c0a0 con 0x7f4480101f70 2024-02-19T19:05:18.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.247+0000 7f448874b700 1 -- 172.21.15.114:0/20140583 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4480109810 con 0x7f4480101f70 2024-02-19T19:05:18.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.248+0000 7f44777fe700 1 -- 172.21.15.114:0/20140583 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f447000faf0 con 0x7f4480101f70 2024-02-19T19:05:18.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.248+0000 7f44777fe700 1 --2- 172.21.15.114:0/20140583 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f446c06d8a0 0x7f446c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:18.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.248+0000 7f44777fe700 1 -- 172.21.15.114:0/20140583 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f447008e8f0 con 0x7f4480101f70 2024-02-19T19:05:18.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.248+0000 7f44864e7700 1 --2- 172.21.15.114:0/20140583 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f446c06d8a0 0x7f446c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:18.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.249+0000 7f44864e7700 1 --2- 172.21.15.114:0/20140583 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f446c06d8a0 0x7f446c06fd60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4480068af0 tx=0x7f447c006d20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:18.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.253+0000 7f44777fe700 1 -- 172.21.15.114:0/20140583 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f447005bc20 con 0x7f4480101f70 2024-02-19T19:05:18.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.472+0000 7f448874b700 1 -- 172.21.15.114:0/20140583 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f44801068c0 con 0x7f446c06d8a0 2024-02-19T19:05:18.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.474+0000 7f44777fe700 1 -- 172.21.15.114:0/20140583 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f44801068c0 con 0x7f446c06d8a0 2024-02-19T19:05:18.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.476+0000 7f448874b700 1 -- 172.21.15.114:0/20140583 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f446c06d8a0 msgr2=0x7f446c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:18.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.476+0000 7f448874b700 1 --2- 172.21.15.114:0/20140583 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f446c06d8a0 0x7f446c06fd60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4480068af0 tx=0x7f447c006d20 comp rx=0 tx=0).stop 2024-02-19T19:05:18.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.476+0000 7f448874b700 1 -- 172.21.15.114:0/20140583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4480101f70 msgr2=0x7f4480196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:18.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.476+0000 7f448874b700 1 --2- 172.21.15.114:0/20140583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4480101f70 0x7f4480196980 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4470006010 tx=0x7f447000be20 comp rx=0 tx=0).stop 2024-02-19T19:05:18.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.476+0000 7f448874b700 1 -- 172.21.15.114:0/20140583 shutdown_connections 2024-02-19T19:05:18.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.476+0000 7f448874b700 1 --2- 172.21.15.114:0/20140583 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f446c06d8a0 0x7f446c06fd60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:18.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.476+0000 7f448874b700 1 --2- 172.21.15.114:0/20140583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44800feef0 0x7f4480196440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:18.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.476+0000 7f448874b700 1 --2- 172.21.15.114:0/20140583 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4480101f70 0x7f4480196980 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:18.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.476+0000 7f448874b700 1 -- 172.21.15.114:0/20140583 >> 172.21.15.114:0/20140583 conn(0x7f44800fa810 msgr2=0x7f44801051a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:18.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.476+0000 7f448874b700 1 -- 172.21.15.114:0/20140583 shutdown_connections 2024-02-19T19:05:18.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.476+0000 7f448874b700 1 -- 172.21.15.114:0/20140583 wait complete. 2024-02-19T19:05:18.506 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-02-19T19:05:18.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.581+0000 7fa3ef266700 1 -- 172.21.15.114:0/2119837619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3e80ff4d0 msgr2=0x7fa3e80ff8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:18.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.581+0000 7fa3ef266700 1 --2- 172.21.15.114:0/2119837619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3e80ff4d0 0x7fa3e80ff8f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8009910 tx=0x7fa3d8009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:18.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.581+0000 7fa3ef266700 1 -- 172.21.15.114:0/2119837619 shutdown_connections 2024-02-19T19:05:18.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.581+0000 7fa3ef266700 1 --2- 172.21.15.114:0/2119837619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3e80ffe30 0x7fa3e81002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:18.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.581+0000 7fa3ef266700 1 --2- 172.21.15.114:0/2119837619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3e80ff4d0 0x7fa3e80ff8f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:18.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.581+0000 7fa3ef266700 1 -- 172.21.15.114:0/2119837619 >> 172.21.15.114:0/2119837619 conn(0x7fa3e80fb050 msgr2=0x7fa3e80fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:18.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.581+0000 7fa3ef266700 1 -- 172.21.15.114:0/2119837619 shutdown_connections 2024-02-19T19:05:18.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.581+0000 7fa3ef266700 1 -- 172.21.15.114:0/2119837619 wait complete. 2024-02-19T19:05:18.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.582+0000 7fa3ef266700 1 Processor -- start 2024-02-19T19:05:18.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.582+0000 7fa3ef266700 1 -- start start 2024-02-19T19:05:18.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.582+0000 7fa3ef266700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3e80ff4d0 0x7fa3e8198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:18.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.582+0000 7fa3ef266700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3e80ffe30 0x7fa3e8198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:18.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.582+0000 7fa3ef266700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e8199160 con 0x7fa3e80ff4d0 2024-02-19T19:05:18.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.582+0000 7fa3ef266700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e81992a0 con 0x7fa3e80ffe30 2024-02-19T19:05:18.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.582+0000 7fa3ed002700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3e80ff4d0 0x7fa3e8198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:18.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.582+0000 7fa3ed002700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3e80ff4d0 0x7fa3e8198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34160/0 (socket says 172.21.15.114:34160) 2024-02-19T19:05:18.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.582+0000 7fa3ed002700 1 -- 172.21.15.114:0/1801376127 learned_addr learned my addr 172.21.15.114:0/1801376127 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:05:18.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.582+0000 7fa3ec801700 1 --2- 172.21.15.114:0/1801376127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3e80ffe30 0x7fa3e8198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:18.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.583+0000 7fa3ed002700 1 -- 172.21.15.114:0/1801376127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3e80ffe30 msgr2=0x7fa3e8198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:18.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.583+0000 7fa3ed002700 1 --2- 172.21.15.114:0/1801376127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3e80ffe30 0x7fa3e8198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:18.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.583+0000 7fa3ed002700 1 -- 172.21.15.114:0/1801376127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d80095f0 con 0x7fa3e80ff4d0 2024-02-19T19:05:18.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.583+0000 7fa3ed002700 1 --2- 172.21.15.114:0/1801376127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3e80ff4d0 0x7fa3e8198600 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8006010 tx=0x7fa3d800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:18.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.585+0000 7fa3e67fc700 1 -- 172.21.15.114:0/1801376127 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3d801d070 con 0x7fa3e80ff4d0 2024-02-19T19:05:18.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.585+0000 7fa3e67fc700 1 -- 172.21.15.114:0/1801376127 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa3d800f460 con 0x7fa3e80ff4d0 2024-02-19T19:05:18.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.585+0000 7fa3ef266700 1 -- 172.21.15.114:0/1801376127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e819dcf0 con 0x7fa3e80ff4d0 2024-02-19T19:05:18.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.585+0000 7fa3e67fc700 1 -- 172.21.15.114:0/1801376127 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3d8021640 con 0x7fa3e80ff4d0 2024-02-19T19:05:18.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.585+0000 7fa3ef266700 1 -- 172.21.15.114:0/1801376127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e819e210 con 0x7fa3e80ff4d0 2024-02-19T19:05:18.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.586+0000 7fa3ef266700 1 -- 172.21.15.114:0/1801376127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e810b9b0 con 0x7fa3e80ff4d0 2024-02-19T19:05:18.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.586+0000 7fa3e67fc700 1 -- 172.21.15.114:0/1801376127 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7fa3d802b440 con 0x7fa3e80ff4d0 2024-02-19T19:05:18.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.586+0000 7fa3e67fc700 1 --2- 172.21.15.114:0/1801376127 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa3d006d8a0 0x7fa3d006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:18.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.587+0000 7fa3e67fc700 1 -- 172.21.15.114:0/1801376127 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fa3d808e820 con 0x7fa3e80ff4d0 2024-02-19T19:05:18.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.587+0000 7fa3ec801700 1 --2- 172.21.15.114:0/1801376127 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa3d006d8a0 0x7fa3d006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:18.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.587+0000 7fa3ec801700 1 --2- 172.21.15.114:0/1801376127 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa3d006d8a0 0x7fa3d006fd60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc009a10 tx=0x7fa3dc009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:18.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.591+0000 7fa3e67fc700 1 -- 172.21.15.114:0/1801376127 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa3d805bbd0 con 0x7fa3e80ff4d0 2024-02-19T19:05:18.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.806+0000 7fa3ef266700 1 -- 172.21.15.114:0/1801376127 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3e8109220 con 0x7fa3d006d8a0 2024-02-19T19:05:18.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.807+0000 7fa3e67fc700 1 -- 172.21.15.114:0/1801376127 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fa3e8109220 con 0x7fa3d006d8a0 2024-02-19T19:05:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.809+0000 7fa3ef266700 1 -- 172.21.15.114:0/1801376127 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa3d006d8a0 msgr2=0x7fa3d006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.809+0000 7fa3ef266700 1 --2- 172.21.15.114:0/1801376127 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa3d006d8a0 0x7fa3d006fd60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc009a10 tx=0x7fa3dc009190 comp rx=0 tx=0).stop 2024-02-19T19:05:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.809+0000 7fa3ef266700 1 -- 172.21.15.114:0/1801376127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3e80ff4d0 msgr2=0x7fa3e8198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.809+0000 7fa3ef266700 1 --2- 172.21.15.114:0/1801376127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3e80ff4d0 0x7fa3e8198600 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8006010 tx=0x7fa3d800bd70 comp rx=0 tx=0).stop 2024-02-19T19:05:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.809+0000 7fa3ef266700 1 -- 172.21.15.114:0/1801376127 shutdown_connections 2024-02-19T19:05:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.809+0000 7fa3ef266700 1 --2- 172.21.15.114:0/1801376127 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa3d006d8a0 0x7fa3d006fd60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:18.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.809+0000 7fa3ef266700 1 --2- 172.21.15.114:0/1801376127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa3e80ff4d0 0x7fa3e8198600 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:18.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.809+0000 7fa3ef266700 1 --2- 172.21.15.114:0/1801376127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3e80ffe30 0x7fa3e8198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:18.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.809+0000 7fa3ef266700 1 -- 172.21.15.114:0/1801376127 >> 172.21.15.114:0/1801376127 conn(0x7fa3e80fb050 msgr2=0x7fa3e8107b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:18.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.809+0000 7fa3ef266700 1 -- 172.21.15.114:0/1801376127 shutdown_connections 2024-02-19T19:05:18.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.809+0000 7fa3ef266700 1 -- 172.21.15.114:0/1801376127 wait complete. 2024-02-19T19:05:18.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:18 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:18.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:18 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:18.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:18 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:18.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:18 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:05:18.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:18 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:18.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:18 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:18.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:18 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:18.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:18 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:18.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:18 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:18.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:18 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-19T19:05:18.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:18 smithi174 ceph-mon[87279]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-19T19:05:18.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:18 smithi174 ceph-mon[87279]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-02-19T19:05:18.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:18 smithi174 ceph-mon[87279]: pgmap v57: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-02-19T19:05:18.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.912+0000 7f3389b4c700 1 -- 172.21.15.114:0/82501812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3384102200 msgr2=0x7f3384102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:18.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.912+0000 7f3389b4c700 1 --2- 172.21.15.114:0/82501812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3384102200 0x7f3384102620 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3374009910 tx=0x7f3374009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:18.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.912+0000 7f3389b4c700 1 -- 172.21.15.114:0/82501812 shutdown_connections 2024-02-19T19:05:18.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.912+0000 7f3389b4c700 1 --2- 172.21.15.114:0/82501812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33840fec80 0x7f33840ff100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:18.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.912+0000 7f3389b4c700 1 --2- 172.21.15.114:0/82501812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3384102200 0x7f3384102620 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:18.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.912+0000 7f3389b4c700 1 -- 172.21.15.114:0/82501812 >> 172.21.15.114:0/82501812 conn(0x7f33840fa800 msgr2=0x7f33840fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:18.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.912+0000 7f3389b4c700 1 -- 172.21.15.114:0/82501812 shutdown_connections 2024-02-19T19:05:18.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.912+0000 7f3389b4c700 1 -- 172.21.15.114:0/82501812 wait complete. 2024-02-19T19:05:18.933 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:18 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:18.934 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:18 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:18.934 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:18 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:18.934 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:18 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:05:18.934 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:18 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:18.934 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:18 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:18.934 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:18 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:18.934 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:18 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:18.934 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:18 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:18.934 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:18 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-19T19:05:18.934 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:18 smithi114 ceph-mon[92983]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-19T19:05:18.934 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:18 smithi114 ceph-mon[92983]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-02-19T19:05:18.934 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:18 smithi114 ceph-mon[92983]: pgmap v57: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-02-19T19:05:18.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.913+0000 7f3389b4c700 1 Processor -- start 2024-02-19T19:05:18.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.913+0000 7f3389b4c700 1 -- start start 2024-02-19T19:05:18.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.913+0000 7f3389b4c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33840fec80 0x7f3384196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:18.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.913+0000 7f3389b4c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3384102200 0x7f33841968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:18.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.913+0000 7f3389b4c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3384196ee0 con 0x7f33840fec80 2024-02-19T19:05:18.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.913+0000 7f3389b4c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3384197020 con 0x7f3384102200 2024-02-19T19:05:18.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.913+0000 7f33837fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33840fec80 0x7f3384196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:18.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.913+0000 7f33837fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33840fec80 0x7f3384196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34186/0 (socket says 172.21.15.114:34186) 2024-02-19T19:05:18.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.913+0000 7f33837fe700 1 -- 172.21.15.114:0/4288157 learned_addr learned my addr 172.21.15.114:0/4288157 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:05:18.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.913+0000 7f337affd700 1 --2- 172.21.15.114:0/4288157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3384102200 0x7f33841968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:18.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.914+0000 7f33837fe700 1 -- 172.21.15.114:0/4288157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3384102200 msgr2=0x7f33841968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:18.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.914+0000 7f33837fe700 1 --2- 172.21.15.114:0/4288157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3384102200 0x7f33841968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:18.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.914+0000 7f33837fe700 1 -- 172.21.15.114:0/4288157 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33740095f0 con 0x7f33840fec80 2024-02-19T19:05:18.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.914+0000 7f33837fe700 1 --2- 172.21.15.114:0/4288157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33840fec80 0x7f3384196380 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f337400bda0 tx=0x7f337400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:18.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.914+0000 7f33817fa700 1 -- 172.21.15.114:0/4288157 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f337401d070 con 0x7f33840fec80 2024-02-19T19:05:18.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.914+0000 7f3389b4c700 1 -- 172.21.15.114:0/4288157 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338419ba70 con 0x7f33840fec80 2024-02-19T19:05:18.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.914+0000 7f3389b4c700 1 -- 172.21.15.114:0/4288157 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338419bf60 con 0x7f33840fec80 2024-02-19T19:05:18.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.914+0000 7f33817fa700 1 -- 172.21.15.114:0/4288157 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f337400f460 con 0x7f33840fec80 2024-02-19T19:05:18.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.914+0000 7f33817fa700 1 -- 172.21.15.114:0/4288157 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3374021620 con 0x7f33840fec80 2024-02-19T19:05:18.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.915+0000 7f3389b4c700 1 -- 172.21.15.114:0/4288157 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3384109780 con 0x7f33840fec80 2024-02-19T19:05:18.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.916+0000 7f33817fa700 1 -- 172.21.15.114:0/4288157 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f337400f5d0 con 0x7f33840fec80 2024-02-19T19:05:18.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.916+0000 7f33817fa700 1 --2- 172.21.15.114:0/4288157 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f337006d8a0 0x7f337006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:18.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.916+0000 7f33817fa700 1 -- 172.21.15.114:0/4288157 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f337408e700 con 0x7f33840fec80 2024-02-19T19:05:18.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.916+0000 7f337affd700 1 --2- 172.21.15.114:0/4288157 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f337006d8a0 0x7f337006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:18.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.917+0000 7f337affd700 1 --2- 172.21.15.114:0/4288157 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f337006d8a0 0x7f337006fd60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3384068d40 tx=0x7f336c00b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:18.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:18.921+0000 7f33817fa700 1 -- 172.21.15.114:0/4288157 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f337405bab0 con 0x7f33840fec80 2024-02-19T19:05:19.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.125+0000 7f3389b4c700 1 -- 172.21.15.114:0/4288157 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f33841068f0 con 0x7f337006d8a0 2024-02-19T19:05:19.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.136+0000 7f33817fa700 1 -- 172.21.15.114:0/4288157 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f33841068f0 con 0x7f337006d8a0 2024-02-19T19:05:19.156 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:05:19.156 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (9m) 6s ago 11m 26.4M - 0.25.0 c8568f914cd2 30444b676ca4 2024-02-19T19:05:19.156 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (12m) 6s ago 12m 7964k - 18.2.1 94eadd990e0a 4918cd87e312 2024-02-19T19:05:19.156 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi174 smithi174 running (6m) 30s ago 10m 7847k - 18.2.1 94eadd990e0a 516c34431b93 2024-02-19T19:05:19.156 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (39s) 6s ago 12m 6660k - 19.0.0-1450-gf78a58c0 f081f847070a f10b46e9d636 2024-02-19T19:05:19.157 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi174 smithi174 running (34s) 30s ago 9m 6660k - 19.0.0-1450-gf78a58c0 f081f847070a fb3b17c81bd8 2024-02-19T19:05:19.157 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (9m) 6s ago 10m 85.7M - 9.4.7 954c08fa6188 478bd533e827 2024-02-19T19:05:19.157 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tcwpod smithi114 running (5m) 6s ago 5m 27.5M - 18.2.1 94eadd990e0a 420175777cf4 2024-02-19T19:05:19.157 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.vpoeop smithi114 running (5m) 6s ago 5m 24.7M - 18.2.1 94eadd990e0a f81a79e53ead 2024-02-19T19:05:19.157 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.stnfnf smithi174 running (5m) 30s ago 5m 26.0M - 18.2.1 94eadd990e0a 3df35954a492 2024-02-19T19:05:19.157 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.znhuvd smithi174 running (5m) 30s ago 5m 24.2M - 18.2.1 94eadd990e0a 2d2267494ae1 2024-02-19T19:05:19.157 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.pcvtrk smithi114 *:8443,9283,8765 running (2m) 6s ago 13m 497M - 19.0.0-1450-gf78a58c0 f081f847070a 05b5c06122b1 2024-02-19T19:05:19.157 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi174.lydvnj smithi174 *:8443,9283,8765 running (2m) 30s ago 9m 450M - 19.0.0-1450-gf78a58c0 f081f847070a f962b1af5aed 2024-02-19T19:05:19.157 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (108s) 6s ago 13m 45.1M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e51f650f9be9 2024-02-19T19:05:19.157 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi174 smithi174 running (84s) 30s ago 9m 33.9M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e811581f05fb 2024-02-19T19:05:19.157 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (12m) 6s ago 12m 14.6M - 1.5.0 0da6a335fe13 b6b258074cce 2024-02-19T19:05:19.157 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi174 smithi174 *:9100 running (9m) 30s ago 9m 14.9M - 1.5.0 0da6a335fe13 d0b4fdf67b22 2024-02-19T19:05:19.157 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (12s) 6s ago 8m 12.5M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 764769439e0e 2024-02-19T19:05:19.158 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (8m) 6s ago 8m 50.5M 1216M 18.2.1 94eadd990e0a 932e3ad37db5 2024-02-19T19:05:19.158 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (7m) 6s ago 7m 51.4M 1216M 18.2.1 94eadd990e0a 12953988af86 2024-02-19T19:05:19.158 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi174 running (7m) 30s ago 7m 49.2M 2240M 18.2.1 94eadd990e0a 979f45cabbce 2024-02-19T19:05:19.158 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi174 running (7m) 30s ago 7m 49.7M 2240M 18.2.1 94eadd990e0a 8b8c3745b8c5 2024-02-19T19:05:19.158 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi174 running (6m) 30s ago 6m 52.2M 2240M 18.2.1 94eadd990e0a 5771a995ed75 2024-02-19T19:05:19.158 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (2m) 6s ago 10m 58.3M - 2.43.0 a07b618ecd1d a36922c7c9a1 2024-02-19T19:05:19.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.138+0000 7f3389b4c700 1 -- 172.21.15.114:0/4288157 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f337006d8a0 msgr2=0x7f337006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:19.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.138+0000 7f3389b4c700 1 --2- 172.21.15.114:0/4288157 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f337006d8a0 0x7f337006fd60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3384068d40 tx=0x7f336c00b410 comp rx=0 tx=0).stop 2024-02-19T19:05:19.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.138+0000 7f3389b4c700 1 -- 172.21.15.114:0/4288157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33840fec80 msgr2=0x7f3384196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:19.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.138+0000 7f3389b4c700 1 --2- 172.21.15.114:0/4288157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33840fec80 0x7f3384196380 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f337400bda0 tx=0x7f337400bdd0 comp rx=0 tx=0).stop 2024-02-19T19:05:19.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.138+0000 7f3389b4c700 1 -- 172.21.15.114:0/4288157 shutdown_connections 2024-02-19T19:05:19.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.138+0000 7f3389b4c700 1 --2- 172.21.15.114:0/4288157 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f337006d8a0 0x7f337006fd60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:19.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.138+0000 7f3389b4c700 1 --2- 172.21.15.114:0/4288157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33840fec80 0x7f3384196380 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:19.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.138+0000 7f3389b4c700 1 --2- 172.21.15.114:0/4288157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3384102200 0x7f33841968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:19.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.138+0000 7f3389b4c700 1 -- 172.21.15.114:0/4288157 >> 172.21.15.114:0/4288157 conn(0x7f33840fa800 msgr2=0x7f33841051d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:19.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.138+0000 7f3389b4c700 1 -- 172.21.15.114:0/4288157 shutdown_connections 2024-02-19T19:05:19.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.138+0000 7f3389b4c700 1 -- 172.21.15.114:0/4288157 wait complete. 2024-02-19T19:05:19.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.241+0000 7f8d992a6700 1 -- 172.21.15.114:0/3744591306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d94103f90 msgr2=0x7f8d94106380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:19.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.241+0000 7f8d992a6700 1 --2- 172.21.15.114:0/3744591306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d94103f90 0x7f8d94106380 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c009910 tx=0x7f8d7c009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:19.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.241+0000 7f8d992a6700 1 -- 172.21.15.114:0/3744591306 shutdown_connections 2024-02-19T19:05:19.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.241+0000 7f8d992a6700 1 --2- 172.21.15.114:0/3744591306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d94103f90 0x7f8d94106380 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:19.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.241+0000 7f8d992a6700 1 --2- 172.21.15.114:0/3744591306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d94101660 0x7f8d94103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:19.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.241+0000 7f8d992a6700 1 -- 172.21.15.114:0/3744591306 >> 172.21.15.114:0/3744591306 conn(0x7f8d940fb050 msgr2=0x7f8d940fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:19.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.241+0000 7f8d992a6700 1 -- 172.21.15.114:0/3744591306 shutdown_connections 2024-02-19T19:05:19.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.241+0000 7f8d992a6700 1 -- 172.21.15.114:0/3744591306 wait complete. 2024-02-19T19:05:19.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.242+0000 7f8d992a6700 1 Processor -- start 2024-02-19T19:05:19.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.242+0000 7f8d992a6700 1 -- start start 2024-02-19T19:05:19.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.242+0000 7f8d992a6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d94101660 0x7f8d94196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:19.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.242+0000 7f8d992a6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d94103f90 0x7f8d941968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.242+0000 7f8d992a6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d94196ed0 con 0x7f8d94103f90 2024-02-19T19:05:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.242+0000 7f8d992a6700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d94197010 con 0x7f8d94101660 2024-02-19T19:05:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.242+0000 7f8d8a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d94103f90 0x7f8d941968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.242+0000 7f8d8a7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d94103f90 0x7f8d941968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34208/0 (socket says 172.21.15.114:34208) 2024-02-19T19:05:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.242+0000 7f8d8a7fc700 1 -- 172.21.15.114:0/507727361 learned_addr learned my addr 172.21.15.114:0/507727361 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:05:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.242+0000 7f8d92ffd700 1 --2- 172.21.15.114:0/507727361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d94101660 0x7f8d94196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.242+0000 7f8d8a7fc700 1 -- 172.21.15.114:0/507727361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d94101660 msgr2=0x7f8d94196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.242+0000 7f8d8a7fc700 1 --2- 172.21.15.114:0/507727361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d94101660 0x7f8d94196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.242+0000 7f8d8a7fc700 1 -- 172.21.15.114:0/507727361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c0095f0 con 0x7f8d94103f90 2024-02-19T19:05:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.243+0000 7f8d8a7fc700 1 --2- 172.21.15.114:0/507727361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d94103f90 0x7f8d941968b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c006010 tx=0x7f8d7c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.243+0000 7f8d90ff9700 1 -- 172.21.15.114:0/507727361 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d7c01d070 con 0x7f8d94103f90 2024-02-19T19:05:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.243+0000 7f8d992a6700 1 -- 172.21.15.114:0/507727361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9419ba60 con 0x7f8d94103f90 2024-02-19T19:05:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.243+0000 7f8d90ff9700 1 -- 172.21.15.114:0/507727361 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8d7c00f460 con 0x7f8d94103f90 2024-02-19T19:05:19.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.243+0000 7f8d90ff9700 1 -- 172.21.15.114:0/507727361 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d7c021640 con 0x7f8d94103f90 2024-02-19T19:05:19.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.243+0000 7f8d992a6700 1 -- 172.21.15.114:0/507727361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9419bf50 con 0x7f8d94103f90 2024-02-19T19:05:19.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.244+0000 7f8d992a6700 1 -- 172.21.15.114:0/507727361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d94190540 con 0x7f8d94103f90 2024-02-19T19:05:19.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.245+0000 7f8d90ff9700 1 -- 172.21.15.114:0/507727361 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f8d7c00fab0 con 0x7f8d94103f90 2024-02-19T19:05:19.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.245+0000 7f8d90ff9700 1 --2- 172.21.15.114:0/507727361 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f8d8006d8a0 0x7f8d8006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:19.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.245+0000 7f8d90ff9700 1 -- 172.21.15.114:0/507727361 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8d7c08e650 con 0x7f8d94103f90 2024-02-19T19:05:19.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.245+0000 7f8d92ffd700 1 --2- 172.21.15.114:0/507727361 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f8d8006d8a0 0x7f8d8006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:19.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.246+0000 7f8d92ffd700 1 --2- 172.21.15.114:0/507727361 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f8d8006d8a0 0x7f8d8006fd60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8d84005950 tx=0x7f8d8400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:19.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.250+0000 7f8d90ff9700 1 -- 172.21.15.114:0/507727361 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8d7c05ba10 con 0x7f8d94103f90 2024-02-19T19:05:19.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.529+0000 7f8d992a6700 1 -- 172.21.15.114:0/507727361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8d9402cc70 con 0x7f8d94103f90 2024-02-19T19:05:19.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.530+0000 7f8d90ff9700 1 -- 172.21.15.114:0/507727361 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f8d7c05b5a0 con 0x7f8d94103f90 2024-02-19T19:05:19.550 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:05:19.550 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-02-19T19:05:19.550 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:05:19.550 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:05:19.550 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-02-19T19:05:19.550 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:05:19.550 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:05:19.551 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-02-19T19:05:19.551 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 5, 2024-02-19T19:05:19.551 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 1 2024-02-19T19:05:19.551 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:05:19.551 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-02-19T19:05:19.551 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:05:19.551 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:05:19.551 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-02-19T19:05:19.551 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 9, 2024-02-19T19:05:19.551 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 5 2024-02-19T19:05:19.551 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-02-19T19:05:19.551 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:05:19.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.532+0000 7f8d992a6700 1 -- 172.21.15.114:0/507727361 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f8d8006d8a0 msgr2=0x7f8d8006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:19.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.532+0000 7f8d992a6700 1 --2- 172.21.15.114:0/507727361 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f8d8006d8a0 0x7f8d8006fd60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8d84005950 tx=0x7f8d8400b410 comp rx=0 tx=0).stop 2024-02-19T19:05:19.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.532+0000 7f8d992a6700 1 -- 172.21.15.114:0/507727361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d94103f90 msgr2=0x7f8d941968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:19.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.532+0000 7f8d992a6700 1 --2- 172.21.15.114:0/507727361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d94103f90 0x7f8d941968b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c006010 tx=0x7f8d7c00bd70 comp rx=0 tx=0).stop 2024-02-19T19:05:19.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.532+0000 7f8d992a6700 1 -- 172.21.15.114:0/507727361 shutdown_connections 2024-02-19T19:05:19.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.532+0000 7f8d992a6700 1 --2- 172.21.15.114:0/507727361 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f8d8006d8a0 0x7f8d8006fd60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:19.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.532+0000 7f8d992a6700 1 --2- 172.21.15.114:0/507727361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d94101660 0x7f8d94196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:19.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.532+0000 7f8d992a6700 1 --2- 172.21.15.114:0/507727361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d94103f90 0x7f8d941968b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:19.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.532+0000 7f8d992a6700 1 -- 172.21.15.114:0/507727361 >> 172.21.15.114:0/507727361 conn(0x7f8d940fb050 msgr2=0x7f8d940fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:19.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.532+0000 7f8d992a6700 1 -- 172.21.15.114:0/507727361 shutdown_connections 2024-02-19T19:05:19.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.532+0000 7f8d992a6700 1 -- 172.21.15.114:0/507727361 wait complete. 2024-02-19T19:05:19.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.637+0000 7feab8bce700 1 -- 172.21.15.114:0/1023897164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab4100cb0 msgr2=0x7feab4101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:19.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.637+0000 7feab8bce700 1 --2- 172.21.15.114:0/1023897164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab4100cb0 0x7feab4101110 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009910 tx=0x7feaa8009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:19.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.637+0000 7feab8bce700 1 -- 172.21.15.114:0/1023897164 shutdown_connections 2024-02-19T19:05:19.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.637+0000 7feab8bce700 1 --2- 172.21.15.114:0/1023897164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab4100cb0 0x7feab4101110 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:19.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.637+0000 7feab8bce700 1 --2- 172.21.15.114:0/1023897164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feab40ffab0 0x7feab40ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:19.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.637+0000 7feab8bce700 1 -- 172.21.15.114:0/1023897164 >> 172.21.15.114:0/1023897164 conn(0x7feab40fb030 msgr2=0x7feab40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:19.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.637+0000 7feab8bce700 1 -- 172.21.15.114:0/1023897164 shutdown_connections 2024-02-19T19:05:19.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.637+0000 7feab8bce700 1 -- 172.21.15.114:0/1023897164 wait complete. 2024-02-19T19:05:19.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.638+0000 7feab8bce700 1 Processor -- start 2024-02-19T19:05:19.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.638+0000 7feab8bce700 1 -- start start 2024-02-19T19:05:19.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.638+0000 7feab8bce700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab40ffab0 0x7feab4195260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:19.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.638+0000 7feab8bce700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feab4100cb0 0x7feab41957a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:19.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.638+0000 7feab8bce700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab4195dc0 con 0x7feab40ffab0 2024-02-19T19:05:19.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.638+0000 7feab8bce700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab4195f00 con 0x7feab4100cb0 2024-02-19T19:05:19.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.638+0000 7feab259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab40ffab0 0x7feab4195260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:19.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.638+0000 7feab259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab40ffab0 0x7feab4195260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34234/0 (socket says 172.21.15.114:34234) 2024-02-19T19:05:19.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.638+0000 7feab259c700 1 -- 172.21.15.114:0/2991230597 learned_addr learned my addr 172.21.15.114:0/2991230597 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:05:19.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.638+0000 7feab1d9b700 1 --2- 172.21.15.114:0/2991230597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feab4100cb0 0x7feab41957a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:19.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.639+0000 7feab259c700 1 -- 172.21.15.114:0/2991230597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feab4100cb0 msgr2=0x7feab41957a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:19.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.639+0000 7feab259c700 1 --2- 172.21.15.114:0/2991230597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feab4100cb0 0x7feab41957a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:19.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.639+0000 7feab259c700 1 -- 172.21.15.114:0/2991230597 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa80095f0 con 0x7feab40ffab0 2024-02-19T19:05:19.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.639+0000 7feab259c700 1 --2- 172.21.15.114:0/2991230597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab40ffab0 0x7feab4195260 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7feaac00efc0 tx=0x7feaac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:19.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.639+0000 7feaa37fe700 1 -- 172.21.15.114:0/2991230597 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feaac018070 con 0x7feab40ffab0 2024-02-19T19:05:19.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.639+0000 7feab8bce700 1 -- 172.21.15.114:0/2991230597 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab419a950 con 0x7feab40ffab0 2024-02-19T19:05:19.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.639+0000 7feaa37fe700 1 -- 172.21.15.114:0/2991230597 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7feaac00f040 con 0x7feab40ffab0 2024-02-19T19:05:19.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.639+0000 7feaa37fe700 1 -- 172.21.15.114:0/2991230597 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feaac004490 con 0x7feab40ffab0 2024-02-19T19:05:19.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.639+0000 7feab8bce700 1 -- 172.21.15.114:0/2991230597 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab419aea0 con 0x7feab40ffab0 2024-02-19T19:05:19.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.640+0000 7feab8bce700 1 -- 172.21.15.114:0/2991230597 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab4108610 con 0x7feab40ffab0 2024-02-19T19:05:19.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.640+0000 7feaa37fe700 1 -- 172.21.15.114:0/2991230597 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7feaac004710 con 0x7feab40ffab0 2024-02-19T19:05:19.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.641+0000 7feaa37fe700 1 --2- 172.21.15.114:0/2991230597 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fea9c06d8a0 0x7fea9c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:19.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.641+0000 7feaa37fe700 1 -- 172.21.15.114:0/2991230597 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7feaac08d710 con 0x7feab40ffab0 2024-02-19T19:05:19.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.641+0000 7feab1d9b700 1 --2- 172.21.15.114:0/2991230597 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fea9c06d8a0 0x7fea9c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:19.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.641+0000 7feab1d9b700 1 --2- 172.21.15.114:0/2991230597 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fea9c06d8a0 0x7fea9c06fd60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009fd0 tx=0x7feaa8005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:19.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.645+0000 7feaa37fe700 1 -- 172.21.15.114:0/2991230597 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7feaac05aa40 con 0x7feab40ffab0 2024-02-19T19:05:19.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:19 smithi174 ceph-mon[87279]: from='client.34216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:19.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:19 smithi174 ceph-mon[87279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/129 objects degraded (3.876%), 2 pgs degraded) 2024-02-19T19:05:19.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:19 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/507727361' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:19.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.886+0000 7feab8bce700 1 -- 172.21.15.114:0/2991230597 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7feab419b5a0 con 0x7feab40ffab0 2024-02-19T19:05:19.906 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:19 smithi114 ceph-mon[92983]: from='client.34216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:19.906 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:19 smithi114 ceph-mon[92983]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/129 objects degraded (3.876%), 2 pgs degraded) 2024-02-19T19:05:19.906 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:19 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/507727361' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:19.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.886+0000 7feaa37fe700 1 -- 172.21.15.114:0/2991230597 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1871 (secure 0 0 0) 0x7feaac05a5d0 con 0x7feab40ffab0 2024-02-19T19:05:19.907 INFO:teuthology.orchestra.run.smithi114.stdout:e18 2024-02-19T19:05:19.907 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:05:19.907 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:05:19.907 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-02-19T19:05:19.907 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:05:19.907 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:05:19.907 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-02-19T19:05:19.907 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 18 2024-02-19T19:05:19.907 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:05:19.907 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:05:19.908 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-02-19T19:00:21.203922+0000 2024-02-19T19:05:19.908 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-02-19T19:05:19.908 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-02-19T19:05:19.908 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-02-19T19:05:19.908 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-02-19T19:05:19.908 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-02-19T19:05:19.908 INFO:teuthology.orchestra.run.smithi114.stdout:max_xattr_size 65536 2024-02-19T19:05:19.908 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-02-19T19:05:19.908 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-02-19T19:05:19.908 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 47 2024-02-19T19:05:19.908 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:05:19.908 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 2 2024-02-19T19:05:19.908 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1 2024-02-19T19:05:19.908 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=14530,1=24269} 2024-02-19T19:05:19.909 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-02-19T19:05:19.909 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-02-19T19:05:19.909 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-02-19T19:05:19.909 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-02-19T19:05:19.909 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-02-19T19:05:19.909 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data enabled 2024-02-19T19:05:19.909 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-02-19T19:05:19.909 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-02-19T19:05:19.909 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 1 2024-02-19T19:05:19.909 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:05:19.909 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.vpoeop{0:14540} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:05:19.909 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.znhuvd{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:05:19.909 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.stnfnf{1:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:05:19.909 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:05:19.910 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:05:19.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.888+0000 7feab8bce700 1 -- 172.21.15.114:0/2991230597 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fea9c06d8a0 msgr2=0x7fea9c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:19.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.888+0000 7feab8bce700 1 --2- 172.21.15.114:0/2991230597 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fea9c06d8a0 0x7fea9c06fd60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009fd0 tx=0x7feaa8005940 comp rx=0 tx=0).stop 2024-02-19T19:05:19.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.888+0000 7feab8bce700 1 -- 172.21.15.114:0/2991230597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab40ffab0 msgr2=0x7feab4195260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:19.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.888+0000 7feab8bce700 1 --2- 172.21.15.114:0/2991230597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab40ffab0 0x7feab4195260 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7feaac00efc0 tx=0x7feaac00c3a0 comp rx=0 tx=0).stop 2024-02-19T19:05:19.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.888+0000 7feab8bce700 1 -- 172.21.15.114:0/2991230597 shutdown_connections 2024-02-19T19:05:19.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.888+0000 7feab8bce700 1 --2- 172.21.15.114:0/2991230597 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fea9c06d8a0 0x7fea9c06fd60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:19.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.888+0000 7feab8bce700 1 --2- 172.21.15.114:0/2991230597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feab40ffab0 0x7feab4195260 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:19.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.888+0000 7feab8bce700 1 --2- 172.21.15.114:0/2991230597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feab4100cb0 0x7feab41957a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:19.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.888+0000 7feab8bce700 1 -- 172.21.15.114:0/2991230597 >> 172.21.15.114:0/2991230597 conn(0x7feab40fb030 msgr2=0x7feab4103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:19.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.888+0000 7feab8bce700 1 -- 172.21.15.114:0/2991230597 shutdown_connections 2024-02-19T19:05:19.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.889+0000 7feab8bce700 1 -- 172.21.15.114:0/2991230597 wait complete. 2024-02-19T19:05:19.911 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 18 2024-02-19T19:05:20.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.994+0000 7f14732d5700 1 -- 172.21.15.114:0/1563065059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146c103ff0 msgr2=0x7f146c1063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:20.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.994+0000 7f14732d5700 1 --2- 172.21.15.114:0/1563065059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146c103ff0 0x7f146c1063e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f145c009910 tx=0x7f145c009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:20.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.994+0000 7f14732d5700 1 -- 172.21.15.114:0/1563065059 shutdown_connections 2024-02-19T19:05:20.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.994+0000 7f14732d5700 1 --2- 172.21.15.114:0/1563065059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146c103ff0 0x7f146c1063e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:20.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.994+0000 7f14732d5700 1 --2- 172.21.15.114:0/1563065059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f146c1016c0 0x7f146c103ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:20.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.994+0000 7f14732d5700 1 -- 172.21.15.114:0/1563065059 >> 172.21.15.114:0/1563065059 conn(0x7f146c0fb030 msgr2=0x7f146c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:20.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.994+0000 7f14732d5700 1 -- 172.21.15.114:0/1563065059 shutdown_connections 2024-02-19T19:05:20.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.994+0000 7f14732d5700 1 -- 172.21.15.114:0/1563065059 wait complete. 2024-02-19T19:05:20.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.994+0000 7f14732d5700 1 Processor -- start 2024-02-19T19:05:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.994+0000 7f14732d5700 1 -- start start 2024-02-19T19:05:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.995+0000 7f14732d5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f146c1016c0 0x7f146c196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.995+0000 7f14732d5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146c103ff0 0x7f146c196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.995+0000 7f14732d5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146c196e90 con 0x7f146c103ff0 2024-02-19T19:05:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.995+0000 7f14732d5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146c196fd0 con 0x7f146c1016c0 2024-02-19T19:05:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.995+0000 7f1470870700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146c103ff0 0x7f146c196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.995+0000 7f1470870700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146c103ff0 0x7f146c196870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34252/0 (socket says 172.21.15.114:34252) 2024-02-19T19:05:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.995+0000 7f1470870700 1 -- 172.21.15.114:0/2064677549 learned_addr learned my addr 172.21.15.114:0/2064677549 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:05:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.995+0000 7f1471071700 1 --2- 172.21.15.114:0/2064677549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f146c1016c0 0x7f146c196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.995+0000 7f1470870700 1 -- 172.21.15.114:0/2064677549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f146c1016c0 msgr2=0x7f146c196330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.995+0000 7f1470870700 1 --2- 172.21.15.114:0/2064677549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f146c1016c0 0x7f146c196330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:20.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.995+0000 7f1470870700 1 -- 172.21.15.114:0/2064677549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f145c0095f0 con 0x7f146c103ff0 2024-02-19T19:05:20.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.995+0000 7f1470870700 1 --2- 172.21.15.114:0/2064677549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146c103ff0 0x7f146c196870 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f145c006010 tx=0x7f145c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:20.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.995+0000 7f14627fc700 1 -- 172.21.15.114:0/2064677549 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f145c01d070 con 0x7f146c103ff0 2024-02-19T19:05:20.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.995+0000 7f14732d5700 1 -- 172.21.15.114:0/2064677549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f146c19ba20 con 0x7f146c103ff0 2024-02-19T19:05:20.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.995+0000 7f14627fc700 1 -- 172.21.15.114:0/2064677549 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f145c00f460 con 0x7f146c103ff0 2024-02-19T19:05:20.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.995+0000 7f14627fc700 1 -- 172.21.15.114:0/2064677549 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f145c021640 con 0x7f146c103ff0 2024-02-19T19:05:20.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.996+0000 7f14732d5700 1 -- 172.21.15.114:0/2064677549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f146c19bf10 con 0x7f146c103ff0 2024-02-19T19:05:20.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.996+0000 7f14732d5700 1 -- 172.21.15.114:0/2064677549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f146c190520 con 0x7f146c103ff0 2024-02-19T19:05:20.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.997+0000 7f14627fc700 1 -- 172.21.15.114:0/2064677549 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f145c00fab0 con 0x7f146c103ff0 2024-02-19T19:05:20.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.997+0000 7f14627fc700 1 --2- 172.21.15.114:0/2064677549 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f145806d8a0 0x7f145806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:20.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.998+0000 7f14627fc700 1 -- 172.21.15.114:0/2064677549 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f145c08e6f0 con 0x7f146c103ff0 2024-02-19T19:05:20.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.998+0000 7f1471071700 1 --2- 172.21.15.114:0/2064677549 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f145806d8a0 0x7f145806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:20.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:19.998+0000 7f1471071700 1 --2- 172.21.15.114:0/2064677549 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f145806d8a0 0x7f145806fd60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1468009550 tx=0x7f14680091b0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:20.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.002+0000 7f14627fc700 1 -- 172.21.15.114:0/2064677549 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f145c05baa0 con 0x7f146c103ff0 2024-02-19T19:05:20.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.220+0000 7f14732d5700 1 -- 172.21.15.114:0/2064677549 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f146c0611d0 con 0x7f145806d8a0 2024-02-19T19:05:20.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.221+0000 7f14627fc700 1 -- 172.21.15.114:0/2064677549 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f146c0611d0 con 0x7f145806d8a0 2024-02-19T19:05:20.241 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:05:20.241 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:05:20.242 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-02-19T19:05:20.242 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:05:20.242 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [ 2024-02-19T19:05:20.242 INFO:teuthology.orchestra.run.smithi114.stdout: "crash", 2024-02-19T19:05:20.242 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr", 2024-02-19T19:05:20.242 INFO:teuthology.orchestra.run.smithi114.stdout: "mon" 2024-02-19T19:05:20.242 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-02-19T19:05:20.242 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "7/23 daemons upgraded", 2024-02-19T19:05:20.242 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "Currently upgrading osd daemons", 2024-02-19T19:05:20.242 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-02-19T19:05:20.242 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:05:20.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.224+0000 7f14732d5700 1 -- 172.21.15.114:0/2064677549 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f145806d8a0 msgr2=0x7f145806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:20.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.224+0000 7f14732d5700 1 --2- 172.21.15.114:0/2064677549 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f145806d8a0 0x7f145806fd60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1468009550 tx=0x7f14680091b0 comp rx=0 tx=0).stop 2024-02-19T19:05:20.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.224+0000 7f14732d5700 1 -- 172.21.15.114:0/2064677549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146c103ff0 msgr2=0x7f146c196870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.224+0000 7f14732d5700 1 --2- 172.21.15.114:0/2064677549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146c103ff0 0x7f146c196870 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f145c006010 tx=0x7f145c00bd70 comp rx=0 tx=0).stop 2024-02-19T19:05:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.224+0000 7f14732d5700 1 -- 172.21.15.114:0/2064677549 shutdown_connections 2024-02-19T19:05:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.224+0000 7f14732d5700 1 --2- 172.21.15.114:0/2064677549 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f145806d8a0 0x7f145806fd60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.224+0000 7f14732d5700 1 --2- 172.21.15.114:0/2064677549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f146c1016c0 0x7f146c196330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.224+0000 7f14732d5700 1 --2- 172.21.15.114:0/2064677549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f146c103ff0 0x7f146c196870 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.224+0000 7f14732d5700 1 -- 172.21.15.114:0/2064677549 >> 172.21.15.114:0/2064677549 conn(0x7f146c0fb030 msgr2=0x7f146c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.224+0000 7f14732d5700 1 -- 172.21.15.114:0/2064677549 shutdown_connections 2024-02-19T19:05:20.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.224+0000 7f14732d5700 1 -- 172.21.15.114:0/2064677549 wait complete. 2024-02-19T19:05:20.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.329+0000 7f6a411f3700 1 -- 172.21.15.114:0/1439970158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a3c0ffae0 msgr2=0x7f6a3c0fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:20.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.329+0000 7f6a411f3700 1 --2- 172.21.15.114:0/1439970158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a3c0ffae0 0x7f6a3c0fff00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009910 tx=0x7f6a30009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:20.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.329+0000 7f6a411f3700 1 -- 172.21.15.114:0/1439970158 shutdown_connections 2024-02-19T19:05:20.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.329+0000 7f6a411f3700 1 --2- 172.21.15.114:0/1439970158 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a3c100ce0 0x7f6a3c101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:20.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.329+0000 7f6a411f3700 1 --2- 172.21.15.114:0/1439970158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a3c0ffae0 0x7f6a3c0fff00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:20.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.329+0000 7f6a411f3700 1 -- 172.21.15.114:0/1439970158 >> 172.21.15.114:0/1439970158 conn(0x7f6a3c0fb080 msgr2=0x7f6a3c0fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:20.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.329+0000 7f6a411f3700 1 -- 172.21.15.114:0/1439970158 shutdown_connections 2024-02-19T19:05:20.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.329+0000 7f6a411f3700 1 -- 172.21.15.114:0/1439970158 wait complete. 2024-02-19T19:05:20.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.330+0000 7f6a411f3700 1 Processor -- start 2024-02-19T19:05:20.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.330+0000 7f6a411f3700 1 -- start start 2024-02-19T19:05:20.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.330+0000 7f6a411f3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a3c0ffae0 0x7f6a3c196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:20.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.330+0000 7f6a411f3700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a3c100ce0 0x7f6a3c196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:20.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.330+0000 7f6a411f3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3c196f80 con 0x7f6a3c0ffae0 2024-02-19T19:05:20.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.330+0000 7f6a411f3700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3c1970c0 con 0x7f6a3c100ce0 2024-02-19T19:05:20.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.330+0000 7f6a3ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a3c0ffae0 0x7f6a3c196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:20.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.330+0000 7f6a3ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a3c0ffae0 0x7f6a3c196420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34262/0 (socket says 172.21.15.114:34262) 2024-02-19T19:05:20.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.330+0000 7f6a3ad9d700 1 -- 172.21.15.114:0/3550200587 learned_addr learned my addr 172.21.15.114:0/3550200587 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:05:20.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.330+0000 7f6a3a59c700 1 --2- 172.21.15.114:0/3550200587 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a3c100ce0 0x7f6a3c196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:20.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.330+0000 7f6a3ad9d700 1 -- 172.21.15.114:0/3550200587 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a3c100ce0 msgr2=0x7f6a3c196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:20.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.330+0000 7f6a3ad9d700 1 --2- 172.21.15.114:0/3550200587 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a3c100ce0 0x7f6a3c196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:20.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.330+0000 7f6a3ad9d700 1 -- 172.21.15.114:0/3550200587 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a300095f0 con 0x7f6a3c0ffae0 2024-02-19T19:05:20.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.331+0000 7f6a3ad9d700 1 --2- 172.21.15.114:0/3550200587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a3c0ffae0 0x7f6a3c196420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000bda0 tx=0x7f6a3000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:20.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.331+0000 7f6a23fff700 1 -- 172.21.15.114:0/3550200587 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a3001d070 con 0x7f6a3c0ffae0 2024-02-19T19:05:20.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.331+0000 7f6a23fff700 1 -- 172.21.15.114:0/3550200587 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6a30004c50 con 0x7f6a3c0ffae0 2024-02-19T19:05:20.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.331+0000 7f6a23fff700 1 -- 172.21.15.114:0/3550200587 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a3000f690 con 0x7f6a3c0ffae0 2024-02-19T19:05:20.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.331+0000 7f6a411f3700 1 -- 172.21.15.114:0/3550200587 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a3c19bb10 con 0x7f6a3c0ffae0 2024-02-19T19:05:20.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.331+0000 7f6a411f3700 1 -- 172.21.15.114:0/3550200587 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a3c19c000 con 0x7f6a3c0ffae0 2024-02-19T19:05:20.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.332+0000 7f6a411f3700 1 -- 172.21.15.114:0/3550200587 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a3c1086c0 con 0x7f6a3c0ffae0 2024-02-19T19:05:20.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.332+0000 7f6a23fff700 1 -- 172.21.15.114:0/3550200587 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f6a30022a20 con 0x7f6a3c0ffae0 2024-02-19T19:05:20.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.333+0000 7f6a23fff700 1 --2- 172.21.15.114:0/3550200587 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6a2406d8a0 0x7f6a2406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:20.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.333+0000 7f6a23fff700 1 -- 172.21.15.114:0/3550200587 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f6a3008ec40 con 0x7f6a3c0ffae0 2024-02-19T19:05:20.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.333+0000 7f6a3a59c700 1 --2- 172.21.15.114:0/3550200587 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6a2406d8a0 0x7f6a2406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:20.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.333+0000 7f6a3a59c700 1 --2- 172.21.15.114:0/3550200587 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6a2406d8a0 0x7f6a2406fd60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c006fd0 tx=0x7f6a2c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:20.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.337+0000 7f6a23fff700 1 -- 172.21.15.114:0/3550200587 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6a3005c020 con 0x7f6a3c0ffae0 2024-02-19T19:05:20.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.621+0000 7f6a411f3700 1 -- 172.21.15.114:0/3550200587 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6a3c1088b0 con 0x7f6a3c0ffae0 2024-02-19T19:05:20.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.622+0000 7f6a23fff700 1 -- 172.21.15.114:0/3550200587 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f6a300270e0 con 0x7f6a3c0ffae0 2024-02-19T19:05:20.642 INFO:teuthology.orchestra.run.smithi114.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:05:20.642 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:05:20.642 INFO:teuthology.orchestra.run.smithi114.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:05:20.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.624+0000 7f6a411f3700 1 -- 172.21.15.114:0/3550200587 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6a2406d8a0 msgr2=0x7f6a2406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:20.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.624+0000 7f6a411f3700 1 --2- 172.21.15.114:0/3550200587 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6a2406d8a0 0x7f6a2406fd60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c006fd0 tx=0x7f6a2c008040 comp rx=0 tx=0).stop 2024-02-19T19:05:20.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.624+0000 7f6a411f3700 1 -- 172.21.15.114:0/3550200587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a3c0ffae0 msgr2=0x7f6a3c196420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:20.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.624+0000 7f6a411f3700 1 --2- 172.21.15.114:0/3550200587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a3c0ffae0 0x7f6a3c196420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000bda0 tx=0x7f6a3000be80 comp rx=0 tx=0).stop 2024-02-19T19:05:20.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.624+0000 7f6a411f3700 1 -- 172.21.15.114:0/3550200587 shutdown_connections 2024-02-19T19:05:20.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.624+0000 7f6a411f3700 1 --2- 172.21.15.114:0/3550200587 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f6a2406d8a0 0x7f6a2406fd60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:20.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.624+0000 7f6a411f3700 1 --2- 172.21.15.114:0/3550200587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a3c0ffae0 0x7f6a3c196420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:20.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.624+0000 7f6a411f3700 1 --2- 172.21.15.114:0/3550200587 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a3c100ce0 0x7f6a3c196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:20.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.624+0000 7f6a411f3700 1 -- 172.21.15.114:0/3550200587 >> 172.21.15.114:0/3550200587 conn(0x7f6a3c0fb080 msgr2=0x7f6a3c068dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:20.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.624+0000 7f6a411f3700 1 -- 172.21.15.114:0/3550200587 shutdown_connections 2024-02-19T19:05:20.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:20.624+0000 7f6a411f3700 1 -- 172.21.15.114:0/3550200587 wait complete. 2024-02-19T19:05:20.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:20 smithi174 ceph-mon[87279]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:20.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:20 smithi174 ceph-mon[87279]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:20.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:20 smithi174 ceph-mon[87279]: pgmap v58: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 2 op/s 2024-02-19T19:05:20.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:20 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/2991230597' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:05:21.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:20 smithi114 ceph-mon[92983]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:21.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:20 smithi114 ceph-mon[92983]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:21.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:20 smithi114 ceph-mon[92983]: pgmap v58: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 2 op/s 2024-02-19T19:05:21.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:20 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/2991230597' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:05:21.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:21 smithi174 ceph-mon[87279]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:21.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:21 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/3550200587' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:05:22.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:21 smithi114 ceph-mon[92983]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:22.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:21 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/3550200587' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:05:22.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:22 smithi174 ceph-mon[87279]: pgmap v59: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 4 op/s 2024-02-19T19:05:23.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:22 smithi114 ceph-mon[92983]: pgmap v59: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 4 op/s 2024-02-19T19:05:24.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:23 smithi114 ceph-mon[92983]: pgmap v60: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-19T19:05:24.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:23 smithi174 ceph-mon[87279]: pgmap v60: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-19T19:05:26.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:26.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:05:26.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:26.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:25 smithi114 ceph-mon[92983]: pgmap v61: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-02-19T19:05:26.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:26.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:05:26.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:26.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:25 smithi174 ceph-mon[87279]: pgmap v61: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-02-19T19:05:28.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:27 smithi174 ceph-mon[87279]: pgmap v62: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:05:28.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:27 smithi114 ceph-mon[92983]: pgmap v62: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:05:29.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:29 smithi174 ceph-mon[87279]: pgmap v63: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:05:30.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:29 smithi114 ceph-mon[92983]: pgmap v63: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:05:32.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:31 smithi174 ceph-mon[87279]: pgmap v64: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:05:32.288 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:31 smithi114 ceph-mon[92983]: pgmap v64: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:05:33.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:32 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-19T19:05:33.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:32 smithi174 ceph-mon[87279]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-19T19:05:33.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:32 smithi174 ceph-mon[87279]: Upgrade: osd.1 is safe to restart 2024-02-19T19:05:33.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:32 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-19T19:05:33.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:32 smithi114 ceph-mon[92983]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-19T19:05:33.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:32 smithi114 ceph-mon[92983]: Upgrade: osd.1 is safe to restart 2024-02-19T19:05:34.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:33 smithi114 ceph-mon[92983]: pgmap v65: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:05:34.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:33 smithi174 ceph-mon[87279]: pgmap v65: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:05:35.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:35 smithi114 ceph-mon[92983]: Upgrade: Updating osd.1 2024-02-19T19:05:35.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:35 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:35.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:35 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-19T19:05:35.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:35 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:35.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:35 smithi114 ceph-mon[92983]: Deploying daemon osd.1 on smithi114 2024-02-19T19:05:35.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:35 smithi174 ceph-mon[87279]: Upgrade: Updating osd.1 2024-02-19T19:05:35.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:35 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:35.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:35 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-19T19:05:35.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:35 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:35.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:35 smithi174 ceph-mon[87279]: Deploying daemon osd.1 on smithi114 2024-02-19T19:05:36.667 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:36 smithi114 ceph-mon[92983]: pgmap v66: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:05:36.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:36 smithi174 ceph-mon[87279]: pgmap v66: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:05:38.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:37 smithi174 ceph-mon[87279]: pgmap v67: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:05:38.186 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:37 smithi114 ceph-mon[92983]: pgmap v67: 65 pgs: 65 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:05:38.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:37 smithi114 systemd[1]: Stopping Ceph osd.1 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T19:05:38.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[55538]: 2024-02-19T19:05:38.165+0000 7fcca3077700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T19:05:38.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[55538]: 2024-02-19T19:05:38.165+0000 7fcca3077700 -1 osd.1 55 *** Got signal Terminated *** 2024-02-19T19:05:38.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[55538]: 2024-02-19T19:05:38.165+0000 7fcca3077700 -1 osd.1 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T19:05:39.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:38 smithi174 ceph-mon[87279]: osd.1 marked itself down and dead 2024-02-19T19:05:39.206 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:38 smithi114 ceph-mon[92983]: osd.1 marked itself down and dead 2024-02-19T19:05:39.487 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:39 smithi114 podman[106814]: 2024-02-19 19:05:39.186707487 +0000 UTC m=+1.154083645 container died 932e3ad37db535cf2ca3f3cd4a33e63c0321025358a3b0553dc2e68004b5f0a8 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240214, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-19T19:05:39.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:39 smithi114 podman[106814]: 2024-02-19 19:05:39.566537016 +0000 UTC m=+1.533913173 container cleanup 932e3ad37db535cf2ca3f3cd4a33e63c0321025358a3b0553dc2e68004b5f0a8 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, RELEASE=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:05:39.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:39 smithi114 bash[106814]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1 2024-02-19T19:05:40.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:39 smithi174 ceph-mon[87279]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:05:40.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:39 smithi174 ceph-mon[87279]: osdmap e56: 6 total, 5 up, 6 in 2024-02-19T19:05:40.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:39 smithi174 ceph-mon[87279]: pgmap v69: 65 pgs: 12 stale+active+clean, 53 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-02-19T19:05:40.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:39 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [3, 1]}]: dispatch 2024-02-19T19:05:40.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:39 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [3, 2]}]: dispatch 2024-02-19T19:05:40.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:39 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [0, 1]}]: dispatch 2024-02-19T19:05:40.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:39 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.15", "id": [0, 5]}]: dispatch 2024-02-19T19:05:40.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:39 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [3, 1]}]: dispatch 2024-02-19T19:05:40.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:39 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [0, 1]}]: dispatch 2024-02-19T19:05:40.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:39 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 0]}]: dispatch 2024-02-19T19:05:40.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:39 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [3, 2]}]: dispatch 2024-02-19T19:05:40.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:39 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [3, 5]}]: dispatch 2024-02-19T19:05:40.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:39 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 0]}]: dispatch 2024-02-19T19:05:40.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:39 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:40.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:39 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:05:40.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:39 smithi114 podman[106831]: 2024-02-19 19:05:39.83488782 +0000 UTC m=+0.642910968 container remove 932e3ad37db535cf2ca3f3cd4a33e63c0321025358a3b0553dc2e68004b5f0a8 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD) 2024-02-19T19:05:40.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:39 smithi114 ceph-mon[92983]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:05:40.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:39 smithi114 ceph-mon[92983]: osdmap e56: 6 total, 5 up, 6 in 2024-02-19T19:05:40.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:39 smithi114 ceph-mon[92983]: pgmap v69: 65 pgs: 12 stale+active+clean, 53 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-02-19T19:05:40.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:39 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [3, 1]}]: dispatch 2024-02-19T19:05:40.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:39 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [3, 2]}]: dispatch 2024-02-19T19:05:40.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:39 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [0, 1]}]: dispatch 2024-02-19T19:05:40.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:39 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.15", "id": [0, 5]}]: dispatch 2024-02-19T19:05:40.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:39 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [3, 1]}]: dispatch 2024-02-19T19:05:40.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:39 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [0, 1]}]: dispatch 2024-02-19T19:05:40.194 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:39 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 0]}]: dispatch 2024-02-19T19:05:40.194 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:39 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [3, 2]}]: dispatch 2024-02-19T19:05:40.194 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:39 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [3, 5]}]: dispatch 2024-02-19T19:05:40.194 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:39 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 0]}]: dispatch 2024-02-19T19:05:40.194 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:39 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:40.194 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:39 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:05:40.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:40 smithi114 podman[106924]: 2024-02-19 19:05:40.076132359 +0000 UTC m=+0.018274004 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:05:40.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:40 smithi114 podman[106924]: 2024-02-19 19:05:40.211592743 +0000 UTC m=+0.153734383 container create 459b5c195a6032d786b1ebaa02dbc95ee4412e2e77f5e747cb6f874d9f28ced0 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-19T19:05:40.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:40 smithi114 podman[106924]: 2024-02-19 19:05:40.406221232 +0000 UTC m=+0.348362887 container init 459b5c195a6032d786b1ebaa02dbc95ee4412e2e77f5e747cb6f874d9f28ced0 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:05:40.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:40 smithi114 podman[106924]: 2024-02-19 19:05:40.409662721 +0000 UTC m=+0.351804359 container start 459b5c195a6032d786b1ebaa02dbc95ee4412e2e77f5e747cb6f874d9f28ced0 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-19T19:05:40.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:40 smithi114 podman[106924]: 2024-02-19 19:05:40.505970379 +0000 UTC m=+0.448112017 container attach 459b5c195a6032d786b1ebaa02dbc95ee4412e2e77f5e747cb6f874d9f28ced0 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214) 2024-02-19T19:05:40.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:40 smithi114 podman[106924]: 2024-02-19 19:05:40.698128 +0000 UTC m=+0.640269635 container died 459b5c195a6032d786b1ebaa02dbc95ee4412e2e77f5e747cb6f874d9f28ced0 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T19:05:41.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:40 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [3, 1]}]': finished 2024-02-19T19:05:41.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:40 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [3, 2]}]': finished 2024-02-19T19:05:41.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:40 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [0, 1]}]': finished 2024-02-19T19:05:41.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:40 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.15", "id": [0, 5]}]': finished 2024-02-19T19:05:41.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:40 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [3, 1]}]': finished 2024-02-19T19:05:41.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:40 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [0, 1]}]': finished 2024-02-19T19:05:41.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:40 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 0]}]': finished 2024-02-19T19:05:41.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:40 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [3, 2]}]': finished 2024-02-19T19:05:41.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:40 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [3, 5]}]': finished 2024-02-19T19:05:41.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:40 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 0]}]': finished 2024-02-19T19:05:41.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:40 smithi174 ceph-mon[87279]: osdmap e57: 6 total, 5 up, 6 in 2024-02-19T19:05:41.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [3, 1]}]': finished 2024-02-19T19:05:41.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [3, 2]}]': finished 2024-02-19T19:05:41.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [0, 1]}]': finished 2024-02-19T19:05:41.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.15", "id": [0, 5]}]': finished 2024-02-19T19:05:41.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [3, 1]}]': finished 2024-02-19T19:05:41.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [0, 1]}]': finished 2024-02-19T19:05:41.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 0]}]': finished 2024-02-19T19:05:41.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [3, 2]}]': finished 2024-02-19T19:05:41.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [3, 5]}]': finished 2024-02-19T19:05:41.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 0]}]': finished 2024-02-19T19:05:41.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:40 smithi114 ceph-mon[92983]: osdmap e57: 6 total, 5 up, 6 in 2024-02-19T19:05:41.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:41 smithi114 podman[106950]: 2024-02-19 19:05:41.436250496 +0000 UTC m=+0.727885735 container remove 459b5c195a6032d786b1ebaa02dbc95ee4412e2e77f5e747cb6f874d9f28ced0 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-19T19:05:41.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:41 smithi114 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-19T19:05:41.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:41 smithi114 systemd[1]: Stopped Ceph osd.1 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T19:05:41.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:41 smithi114 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.1.service: Consumed 5.498s CPU time. 2024-02-19T19:05:41.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:41 smithi114 systemd[1]: Starting Ceph osd.1 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T19:05:42.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:41 smithi174 ceph-mon[87279]: osdmap e58: 6 total, 5 up, 6 in 2024-02-19T19:05:42.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:41 smithi174 ceph-mon[87279]: pgmap v72: 65 pgs: 4 unknown, 4 peering, 8 active+undersized+degraded, 3 stale+active+clean, 16 active+undersized, 30 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 16/126 objects degraded (12.698%) 2024-02-19T19:05:42.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:41 smithi114 ceph-mon[92983]: osdmap e58: 6 total, 5 up, 6 in 2024-02-19T19:05:42.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:41 smithi114 ceph-mon[92983]: pgmap v72: 65 pgs: 4 unknown, 4 peering, 8 active+undersized+degraded, 3 stale+active+clean, 16 active+undersized, 30 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 16/126 objects degraded (12.698%) 2024-02-19T19:05:42.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:41 smithi114 podman[107053]: 2024-02-19 19:05:41.798113335 +0000 UTC m=+0.018670083 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:05:42.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:41 smithi114 podman[107053]: 2024-02-19 19:05:41.946420202 +0000 UTC m=+0.166976942 container create e7496b104a451525adb58184c452923f4cd1364c1c611c09f5c328f4c747ace5 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-activate, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214) 2024-02-19T19:05:42.194 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 podman[107053]: 2024-02-19 19:05:42.174151951 +0000 UTC m=+0.394708691 container init e7496b104a451525adb58184c452923f4cd1364c1c611c09f5c328f4c747ace5 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-activate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:05:42.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 podman[107053]: 2024-02-19 19:05:42.177388765 +0000 UTC m=+0.397945506 container start e7496b104a451525adb58184c452923f4cd1364c1c611c09f5c328f4c747ace5 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-activate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T19:05:42.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 podman[107053]: 2024-02-19 19:05:42.290817098 +0000 UTC m=+0.511373852 container attach e7496b104a451525adb58184c452923f4cd1364c1c611c09f5c328f4c747ace5 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-19T19:05:43.006 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:42 smithi114 ceph-mon[92983]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-02-19T19:05:43.006 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:42 smithi114 ceph-mon[92983]: Health check failed: Degraded data redundancy: 16/126 objects degraded (12.698%), 8 pgs degraded (PG_DEGRADED) 2024-02-19T19:05:43.006 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:42 smithi114 ceph-mon[92983]: osdmap e59: 6 total, 5 up, 6 in 2024-02-19T19:05:43.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-activate[107068]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-19T19:05:43.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-activate[107068]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-19T19:05:43.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 bash[107053]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-19T19:05:43.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 bash[107053]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-19T19:05:43.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 bash[107053]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-19T19:05:43.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 bash[107053]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-19T19:05:43.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 bash[107053]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-19T19:05:43.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 bash[107053]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-19T19:05:43.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 bash[107053]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-19T19:05:43.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-activate[107068]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-19T19:05:43.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-activate[107068]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-19T19:05:43.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-activate[107068]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-19T19:05:43.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-activate[107068]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-19T19:05:43.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-activate[107068]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-19T19:05:43.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:42 smithi114 podman[107053]: 2024-02-19 19:05:42.642088296 +0000 UTC m=+0.862645040 container died e7496b104a451525adb58184c452923f4cd1364c1c611c09f5c328f4c747ace5 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-activate, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T19:05:43.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:42 smithi174 ceph-mon[87279]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-02-19T19:05:43.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:42 smithi174 ceph-mon[87279]: Health check failed: Degraded data redundancy: 16/126 objects degraded (12.698%), 8 pgs degraded (PG_DEGRADED) 2024-02-19T19:05:43.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:42 smithi174 ceph-mon[87279]: osdmap e59: 6 total, 5 up, 6 in 2024-02-19T19:05:43.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:43 smithi114 podman[107198]: 2024-02-19 19:05:43.696185011 +0000 UTC m=+1.048548220 container remove e7496b104a451525adb58184c452923f4cd1364c1c611c09f5c328f4c747ace5 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-activate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-02-19T19:05:44.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:43 smithi174 ceph-mon[87279]: pgmap v74: 65 pgs: 4 unknown, 4 peering, 9 active+undersized+degraded, 20 active+undersized, 28 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 18/126 objects degraded (14.286%) 2024-02-19T19:05:44.152 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:43 smithi114 ceph-mon[92983]: pgmap v74: 65 pgs: 4 unknown, 4 peering, 9 active+undersized+degraded, 20 active+undersized, 28 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 18/126 objects degraded (14.286%) 2024-02-19T19:05:44.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:43 smithi114 podman[107263]: 2024-02-19 19:05:43.810420329 +0000 UTC m=+0.018749813 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:05:44.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:43 smithi114 podman[107263]: 2024-02-19 19:05:43.922556668 +0000 UTC m=+0.130886144 container create c1c32dbdf57335eabf09d22bbc653009e14ba4000e3d1fa3daa4510894ae380f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:05:44.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:44 smithi114 podman[107263]: 2024-02-19 19:05:44.133426578 +0000 UTC m=+0.341756055 container init c1c32dbdf57335eabf09d22bbc653009e14ba4000e3d1fa3daa4510894ae380f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-02-19T19:05:44.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:44 smithi114 podman[107263]: 2024-02-19 19:05:44.136662878 +0000 UTC m=+0.344992363 container start c1c32dbdf57335eabf09d22bbc653009e14ba4000e3d1fa3daa4510894ae380f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-02-19T19:05:44.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:44 smithi114 ceph-osd[107282]: -- 172.21.15.114:0/718060419 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b9e95911e0 con 0x55b9e9562c00 2024-02-19T19:05:44.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:44 smithi114 bash[107263]: c1c32dbdf57335eabf09d22bbc653009e14ba4000e3d1fa3daa4510894ae380f 2024-02-19T19:05:44.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:44 smithi114 systemd[1]: Started Ceph osd.1 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T19:05:45.039 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:05:44.683+0000 7f4f044447c0 -1 Falling back to public interface 2024-02-19T19:05:45.365 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:45 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:45.365 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:45 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:45.365 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:45 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:45.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:45 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:45.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:45 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:45.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:45 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:46.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:46 smithi114 ceph-mon[92983]: pgmap v75: 65 pgs: 9 active+clean+remapped, 9 active+undersized+degraded, 20 active+undersized, 27 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 18/129 objects degraded (13.953%); 11/129 objects misplaced (8.527%); 111 B/s, 0 objects/s recovering 2024-02-19T19:05:46.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:46 smithi174 ceph-mon[87279]: pgmap v75: 65 pgs: 9 active+clean+remapped, 9 active+undersized+degraded, 20 active+undersized, 27 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 18/129 objects degraded (13.953%); 11/129 objects misplaced (8.527%); 111 B/s, 0 objects/s recovering 2024-02-19T19:05:47.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:47 smithi114 ceph-mon[92983]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-02-19T19:05:47.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:47 smithi174 ceph-mon[87279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-02-19T19:05:48.289 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:05:48.139+0000 7f4f044447c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-02-19T19:05:48.289 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:05:48.173+0000 7f4f044447c0 -1 osd.1 55 log_to_monitors true 2024-02-19T19:05:48.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:48 smithi174 ceph-mon[87279]: pgmap v76: 65 pgs: 9 active+clean+remapped, 9 active+undersized+degraded, 20 active+undersized, 27 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 18/129 objects degraded (13.953%); 11/129 objects misplaced (8.527%); 84 B/s, 0 objects/s recovering 2024-02-19T19:05:48.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:48 smithi174 ceph-mon[87279]: from='osd.1 [v2:172.21.15.114:6810/3646303405,v1:172.21.15.114:6811/3646303405]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-19T19:05:48.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:48 smithi114 ceph-mon[92983]: pgmap v76: 65 pgs: 9 active+clean+remapped, 9 active+undersized+degraded, 20 active+undersized, 27 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 18/129 objects degraded (13.953%); 11/129 objects misplaced (8.527%); 84 B/s, 0 objects/s recovering 2024-02-19T19:05:48.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:48 smithi114 ceph-mon[92983]: from='osd.1 [v2:172.21.15.114:6810/3646303405,v1:172.21.15.114:6811/3646303405]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-19T19:05:49.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:49 smithi114 ceph-mon[92983]: from='osd.1 [v2:172.21.15.114:6810/3646303405,v1:172.21.15.114:6811/3646303405]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-19T19:05:49.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:49 smithi114 ceph-mon[92983]: osdmap e60: 6 total, 5 up, 6 in 2024-02-19T19:05:49.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:49 smithi114 ceph-mon[92983]: from='osd.1 [v2:172.21.15.114:6810/3646303405,v1:172.21.15.114:6811/3646303405]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-02-19T19:05:49.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:49 smithi114 ceph-mon[92983]: from='osd.1 [v2:172.21.15.114:6810/3646303405,v1:172.21.15.114:6811/3646303405]' entity='osd.1' 2024-02-19T19:05:49.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:05:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:05:49.153+0000 7f4ef1894700 -1 osd.1 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T19:05:49.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:49 smithi174 ceph-mon[87279]: from='osd.1 [v2:172.21.15.114:6810/3646303405,v1:172.21.15.114:6811/3646303405]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-19T19:05:49.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:49 smithi174 ceph-mon[87279]: osdmap e60: 6 total, 5 up, 6 in 2024-02-19T19:05:49.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:49 smithi174 ceph-mon[87279]: from='osd.1 [v2:172.21.15.114:6810/3646303405,v1:172.21.15.114:6811/3646303405]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-02-19T19:05:49.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:49 smithi174 ceph-mon[87279]: from='osd.1 [v2:172.21.15.114:6810/3646303405,v1:172.21.15.114:6811/3646303405]' entity='osd.1' 2024-02-19T19:05:50.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.735+0000 7f0ff2040700 1 -- 172.21.15.114:0/2782979737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fec102170 msgr2=0x7f0fec102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:50.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.735+0000 7f0ff2040700 1 --2- 172.21.15.114:0/2782979737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fec102170 0x7f0fec102590 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc009910 tx=0x7f0fdc009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:50.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.735+0000 7f0ff2040700 1 -- 172.21.15.114:0/2782979737 shutdown_connections 2024-02-19T19:05:50.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.735+0000 7f0ff2040700 1 --2- 172.21.15.114:0/2782979737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0fec0febf0 0x7f0fec0ff050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:50.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.735+0000 7f0ff2040700 1 --2- 172.21.15.114:0/2782979737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fec102170 0x7f0fec102590 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:50.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.735+0000 7f0ff2040700 1 -- 172.21.15.114:0/2782979737 >> 172.21.15.114:0/2782979737 conn(0x7f0fec0fa7b0 msgr2=0x7f0fec0fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:50.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.735+0000 7f0ff2040700 1 -- 172.21.15.114:0/2782979737 shutdown_connections 2024-02-19T19:05:50.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.735+0000 7f0ff2040700 1 -- 172.21.15.114:0/2782979737 wait complete. 2024-02-19T19:05:50.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.736+0000 7f0ff2040700 1 Processor -- start 2024-02-19T19:05:50.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.736+0000 7f0ff2040700 1 -- start start 2024-02-19T19:05:50.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.736+0000 7f0ff2040700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0fec0febf0 0x7f0fec101870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:50.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.736+0000 7f0ff2040700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fec102170 0x7f0fec0ffec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:50.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.737+0000 7f0ff2040700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec100400 con 0x7f0fec102170 2024-02-19T19:05:50.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.737+0000 7f0ff2040700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec100540 con 0x7f0fec0febf0 2024-02-19T19:05:50.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.737+0000 7f0feaffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fec102170 0x7f0fec0ffec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:50.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.737+0000 7f0feb7fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0fec0febf0 0x7f0fec101870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:50.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.737+0000 7f0feaffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fec102170 0x7f0fec0ffec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42662/0 (socket says 172.21.15.114:42662) 2024-02-19T19:05:50.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.737+0000 7f0feaffd700 1 -- 172.21.15.114:0/557614155 learned_addr learned my addr 172.21.15.114:0/557614155 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:05:50.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.737+0000 7f0feaffd700 1 -- 172.21.15.114:0/557614155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0fec0febf0 msgr2=0x7f0fec101870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:50.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.737+0000 7f0feaffd700 1 --2- 172.21.15.114:0/557614155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0fec0febf0 0x7f0fec101870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:50.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.737+0000 7f0feaffd700 1 -- 172.21.15.114:0/557614155 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fdc0095f0 con 0x7f0fec102170 2024-02-19T19:05:50.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.737+0000 7f0feaffd700 1 --2- 172.21.15.114:0/557614155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fec102170 0x7f0fec0ffec0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0fe000b4b0 tx=0x7f0fe000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:50.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.737+0000 7f0fe8ff9700 1 -- 172.21.15.114:0/557614155 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fe0011800 con 0x7f0fec102170 2024-02-19T19:05:50.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.738+0000 7f0ff2040700 1 -- 172.21.15.114:0/557614155 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fec100820 con 0x7f0fec102170 2024-02-19T19:05:50.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.738+0000 7f0ff2040700 1 -- 172.21.15.114:0/557614155 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fec19e4d0 con 0x7f0fec102170 2024-02-19T19:05:50.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.738+0000 7f0fe8ff9700 1 -- 172.21.15.114:0/557614155 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0fe0011e10 con 0x7f0fec102170 2024-02-19T19:05:50.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.738+0000 7f0fe8ff9700 1 -- 172.21.15.114:0/557614155 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fe000f2e0 con 0x7f0fec102170 2024-02-19T19:05:50.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.739+0000 7f0ff2040700 1 -- 172.21.15.114:0/557614155 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd8005320 con 0x7f0fec102170 2024-02-19T19:05:50.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.739+0000 7f0fe8ff9700 1 -- 172.21.15.114:0/557614155 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f0fe0011960 con 0x7f0fec102170 2024-02-19T19:05:50.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.739+0000 7f0fe8ff9700 1 --2- 172.21.15.114:0/557614155 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f0fd406d8a0 0x7f0fd406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:50.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.739+0000 7f0fe8ff9700 1 -- 172.21.15.114:0/557614155 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7087+0+0 (secure 0 0 0) 0x7f0fe008d150 con 0x7f0fec102170 2024-02-19T19:05:50.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.739+0000 7f0feb7fe700 1 --2- 172.21.15.114:0/557614155 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f0fd406d8a0 0x7f0fd406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:50.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.740+0000 7f0feb7fe700 1 --2- 172.21.15.114:0/557614155 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f0fd406d8a0 0x7f0fd406fd60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc006010 tx=0x7f0fdc00b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:50.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.745+0000 7f0fe8ff9700 1 -- 172.21.15.114:0/557614155 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0fe005a2c0 con 0x7f0fec102170 2024-02-19T19:05:50.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:50 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:50.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:50 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:50.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:50 smithi114 ceph-mon[92983]: pgmap v78: 65 pgs: 9 active+clean+remapped, 9 active+undersized+degraded, 20 active+undersized, 27 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s; 18/129 objects degraded (13.953%); 11/129 objects misplaced (8.527%); 83 B/s, 0 objects/s recovering 2024-02-19T19:05:50.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:50 smithi114 ceph-mon[92983]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:05:50.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:50 smithi114 ceph-mon[92983]: osd.1 [v2:172.21.15.114:6810/3646303405,v1:172.21.15.114:6811/3646303405] boot 2024-02-19T19:05:50.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:50 smithi114 ceph-mon[92983]: osdmap e61: 6 total, 6 up, 6 in 2024-02-19T19:05:50.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:50 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:05:50.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:50 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:50.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:50 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:50.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:50 smithi174 ceph-mon[87279]: pgmap v78: 65 pgs: 9 active+clean+remapped, 9 active+undersized+degraded, 20 active+undersized, 27 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 0 B/s wr, 0 op/s; 18/129 objects degraded (13.953%); 11/129 objects misplaced (8.527%); 83 B/s, 0 objects/s recovering 2024-02-19T19:05:50.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:50 smithi174 ceph-mon[87279]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:05:50.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:50 smithi174 ceph-mon[87279]: osd.1 [v2:172.21.15.114:6810/3646303405,v1:172.21.15.114:6811/3646303405] boot 2024-02-19T19:05:50.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:50 smithi174 ceph-mon[87279]: osdmap e61: 6 total, 6 up, 6 in 2024-02-19T19:05:50.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:50 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:05:51.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.982+0000 7f0ff2040700 1 -- 172.21.15.114:0/557614155 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fd8000bf0 con 0x7f0fd406d8a0 2024-02-19T19:05:51.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.983+0000 7f0fe8ff9700 1 -- 172.21.15.114:0/557614155 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f0fd8000bf0 con 0x7f0fd406d8a0 2024-02-19T19:05:51.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.985+0000 7f0ff2040700 1 -- 172.21.15.114:0/557614155 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f0fd406d8a0 msgr2=0x7f0fd406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:51.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.985+0000 7f0ff2040700 1 --2- 172.21.15.114:0/557614155 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f0fd406d8a0 0x7f0fd406fd60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc006010 tx=0x7f0fdc00b540 comp rx=0 tx=0).stop 2024-02-19T19:05:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.985+0000 7f0ff2040700 1 -- 172.21.15.114:0/557614155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fec102170 msgr2=0x7f0fec0ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.985+0000 7f0ff2040700 1 --2- 172.21.15.114:0/557614155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fec102170 0x7f0fec0ffec0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0fe000b4b0 tx=0x7f0fe000b7c0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.985+0000 7f0ff2040700 1 -- 172.21.15.114:0/557614155 shutdown_connections 2024-02-19T19:05:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.985+0000 7f0ff2040700 1 --2- 172.21.15.114:0/557614155 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f0fd406d8a0 0x7f0fd406fd60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.985+0000 7f0ff2040700 1 --2- 172.21.15.114:0/557614155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0fec0febf0 0x7f0fec101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.985+0000 7f0ff2040700 1 --2- 172.21.15.114:0/557614155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fec102170 0x7f0fec0ffec0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.985+0000 7f0ff2040700 1 -- 172.21.15.114:0/557614155 >> 172.21.15.114:0/557614155 conn(0x7f0fec0fa7b0 msgr2=0x7f0fec105140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.985+0000 7f0ff2040700 1 -- 172.21.15.114:0/557614155 shutdown_connections 2024-02-19T19:05:51.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:50.985+0000 7f0ff2040700 1 -- 172.21.15.114:0/557614155 wait complete. 2024-02-19T19:05:51.018 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-02-19T19:05:51.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.095+0000 7ff113c57700 1 -- 172.21.15.114:0/2740078826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff10c100c00 msgr2=0x7ff10c101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.095+0000 7ff113c57700 1 --2- 172.21.15.114:0/2740078826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff10c100c00 0x7ff10c101020 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff108009910 tx=0x7ff108009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.095+0000 7ff113c57700 1 -- 172.21.15.114:0/2740078826 shutdown_connections 2024-02-19T19:05:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.095+0000 7ff113c57700 1 --2- 172.21.15.114:0/2740078826 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff10c101e00 0x7ff10c102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.095+0000 7ff113c57700 1 --2- 172.21.15.114:0/2740078826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff10c100c00 0x7ff10c101020 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.095+0000 7ff113c57700 1 -- 172.21.15.114:0/2740078826 >> 172.21.15.114:0/2740078826 conn(0x7ff10c0fc180 msgr2=0x7ff10c0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.095+0000 7ff113c57700 1 -- 172.21.15.114:0/2740078826 shutdown_connections 2024-02-19T19:05:51.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.095+0000 7ff113c57700 1 -- 172.21.15.114:0/2740078826 wait complete. 2024-02-19T19:05:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.096+0000 7ff113c57700 1 Processor -- start 2024-02-19T19:05:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.096+0000 7ff113c57700 1 -- start start 2024-02-19T19:05:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.096+0000 7ff113c57700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff10c100c00 0x7ff10c074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.096+0000 7ff113c57700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff10c101e00 0x7ff10c073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.096+0000 7ff113c57700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c0736c0 con 0x7ff10c101e00 2024-02-19T19:05:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.096+0000 7ff113c57700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c073800 con 0x7ff10c100c00 2024-02-19T19:05:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.096+0000 7ff1111f2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff10c101e00 0x7ff10c073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.096+0000 7ff1111f2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff10c101e00 0x7ff10c073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42680/0 (socket says 172.21.15.114:42680) 2024-02-19T19:05:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.096+0000 7ff1111f2700 1 -- 172.21.15.114:0/1763724475 learned_addr learned my addr 172.21.15.114:0/1763724475 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:05:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.096+0000 7ff1119f3700 1 --2- 172.21.15.114:0/1763724475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff10c100c00 0x7ff10c074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:51.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.096+0000 7ff1111f2700 1 -- 172.21.15.114:0/1763724475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff10c100c00 msgr2=0x7ff10c074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:51.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.096+0000 7ff1111f2700 1 --2- 172.21.15.114:0/1763724475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff10c100c00 0x7ff10c074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.096+0000 7ff1111f2700 1 -- 172.21.15.114:0/1763724475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1080095f0 con 0x7ff10c101e00 2024-02-19T19:05:51.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.097+0000 7ff1111f2700 1 --2- 172.21.15.114:0/1763724475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff10c101e00 0x7ff10c073180 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00d8d0 tx=0x7ff0fc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:51.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.097+0000 7ff102ffd700 1 -- 172.21.15.114:0/1763724475 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0fc009940 con 0x7ff10c101e00 2024-02-19T19:05:51.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.097+0000 7ff113c57700 1 -- 172.21.15.114:0/1763724475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff10c073ae0 con 0x7ff10c101e00 2024-02-19T19:05:51.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.097+0000 7ff102ffd700 1 -- 172.21.15.114:0/1763724475 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff0fc010460 con 0x7ff10c101e00 2024-02-19T19:05:51.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.097+0000 7ff102ffd700 1 -- 172.21.15.114:0/1763724475 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0fc00f5f0 con 0x7ff10c101e00 2024-02-19T19:05:51.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.097+0000 7ff113c57700 1 -- 172.21.15.114:0/1763724475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff10c19e3f0 con 0x7ff10c101e00 2024-02-19T19:05:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.098+0000 7ff113c57700 1 -- 172.21.15.114:0/1763724475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff10c109760 con 0x7ff10c101e00 2024-02-19T19:05:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.099+0000 7ff102ffd700 1 -- 172.21.15.114:0/1763724475 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7ff0fc00f830 con 0x7ff10c101e00 2024-02-19T19:05:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.099+0000 7ff102ffd700 1 --2- 172.21.15.114:0/1763724475 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff0f806d8a0 0x7ff0f806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.099+0000 7ff102ffd700 1 -- 172.21.15.114:0/1763724475 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7087+0+0 (secure 0 0 0) 0x7ff0fc08d740 con 0x7ff10c101e00 2024-02-19T19:05:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.099+0000 7ff1119f3700 1 --2- 172.21.15.114:0/1763724475 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff0f806d8a0 0x7ff0f806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:51.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.099+0000 7ff1119f3700 1 --2- 172.21.15.114:0/1763724475 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff0f806d8a0 0x7ff0f806fd60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff108006010 tx=0x7ff10800b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:51.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.104+0000 7ff102ffd700 1 -- 172.21.15.114:0/1763724475 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff0fc05a830 con 0x7ff10c101e00 2024-02-19T19:05:51.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.343+0000 7ff113c57700 1 -- 172.21.15.114:0/1763724475 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff10c19e9a0 con 0x7ff0f806d8a0 2024-02-19T19:05:51.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.344+0000 7ff102ffd700 1 -- 172.21.15.114:0/1763724475 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7ff10c19e9a0 con 0x7ff0f806d8a0 2024-02-19T19:05:51.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.347+0000 7ff113c57700 1 -- 172.21.15.114:0/1763724475 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff0f806d8a0 msgr2=0x7ff0f806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:51.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.347+0000 7ff113c57700 1 --2- 172.21.15.114:0/1763724475 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff0f806d8a0 0x7ff0f806fd60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff108006010 tx=0x7ff10800b540 comp rx=0 tx=0).stop 2024-02-19T19:05:51.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.347+0000 7ff113c57700 1 -- 172.21.15.114:0/1763724475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff10c101e00 msgr2=0x7ff10c073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:51.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.347+0000 7ff113c57700 1 --2- 172.21.15.114:0/1763724475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff10c101e00 0x7ff10c073180 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00d8d0 tx=0x7ff0fc00dc90 comp rx=0 tx=0).stop 2024-02-19T19:05:51.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.347+0000 7ff113c57700 1 -- 172.21.15.114:0/1763724475 shutdown_connections 2024-02-19T19:05:51.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.347+0000 7ff113c57700 1 --2- 172.21.15.114:0/1763724475 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff0f806d8a0 0x7ff0f806fd60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.347+0000 7ff113c57700 1 --2- 172.21.15.114:0/1763724475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff10c100c00 0x7ff10c074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.347+0000 7ff113c57700 1 --2- 172.21.15.114:0/1763724475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff10c101e00 0x7ff10c073180 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.347+0000 7ff113c57700 1 -- 172.21.15.114:0/1763724475 >> 172.21.15.114:0/1763724475 conn(0x7ff10c0fc180 msgr2=0x7ff10c105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:51.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.347+0000 7ff113c57700 1 -- 172.21.15.114:0/1763724475 shutdown_connections 2024-02-19T19:05:51.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.347+0000 7ff113c57700 1 -- 172.21.15.114:0/1763724475 wait complete. 2024-02-19T19:05:51.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.454+0000 7fb2975e3700 1 -- 172.21.15.114:0/4047049401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb290100cb0 msgr2=0x7fb290101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.454+0000 7fb2975e3700 1 --2- 172.21.15.114:0/4047049401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb290100cb0 0x7fb290101110 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb280009910 tx=0x7fb280009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.454+0000 7fb2975e3700 1 -- 172.21.15.114:0/4047049401 shutdown_connections 2024-02-19T19:05:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.454+0000 7fb2975e3700 1 --2- 172.21.15.114:0/4047049401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb290100cb0 0x7fb290101110 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.454+0000 7fb2975e3700 1 --2- 172.21.15.114:0/4047049401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2900ffab0 0x7fb2900ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.454+0000 7fb2975e3700 1 -- 172.21.15.114:0/4047049401 >> 172.21.15.114:0/4047049401 conn(0x7fb2900fb030 msgr2=0x7fb2900fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.454+0000 7fb2975e3700 1 -- 172.21.15.114:0/4047049401 shutdown_connections 2024-02-19T19:05:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.454+0000 7fb2975e3700 1 -- 172.21.15.114:0/4047049401 wait complete. 2024-02-19T19:05:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.454+0000 7fb2975e3700 1 Processor -- start 2024-02-19T19:05:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.454+0000 7fb2975e3700 1 -- start start 2024-02-19T19:05:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.454+0000 7fb2975e3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2900ffab0 0x7fb290068f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.455+0000 7fb2975e3700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb290100cb0 0x7fb29019b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.455+0000 7fb2975e3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29019b670 con 0x7fb2900ffab0 2024-02-19T19:05:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.455+0000 7fb2975e3700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29019b7e0 con 0x7fb290100cb0 2024-02-19T19:05:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.455+0000 7fb29537f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2900ffab0 0x7fb290068f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.455+0000 7fb29537f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2900ffab0 0x7fb290068f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42692/0 (socket says 172.21.15.114:42692) 2024-02-19T19:05:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.455+0000 7fb29537f700 1 -- 172.21.15.114:0/2727653598 learned_addr learned my addr 172.21.15.114:0/2727653598 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:05:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.455+0000 7fb294b7e700 1 --2- 172.21.15.114:0/2727653598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb290100cb0 0x7fb29019b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.455+0000 7fb29537f700 1 -- 172.21.15.114:0/2727653598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb290100cb0 msgr2=0x7fb29019b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.455+0000 7fb29537f700 1 --2- 172.21.15.114:0/2727653598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb290100cb0 0x7fb29019b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.455+0000 7fb29537f700 1 -- 172.21.15.114:0/2727653598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2800095f0 con 0x7fb2900ffab0 2024-02-19T19:05:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.455+0000 7fb29537f700 1 --2- 172.21.15.114:0/2727653598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2900ffab0 0x7fb290068f90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb28c00d900 tx=0x7fb28c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.455+0000 7fb2867fc700 1 -- 172.21.15.114:0/2727653598 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb28c0041f0 con 0x7fb2900ffab0 2024-02-19T19:05:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.455+0000 7fb2867fc700 1 -- 172.21.15.114:0/2727653598 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb28c004350 con 0x7fb2900ffab0 2024-02-19T19:05:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.455+0000 7fb2975e3700 1 -- 172.21.15.114:0/2727653598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb29019bac0 con 0x7fb2900ffab0 2024-02-19T19:05:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.455+0000 7fb2867fc700 1 -- 172.21.15.114:0/2727653598 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb28c003d20 con 0x7fb2900ffab0 2024-02-19T19:05:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.455+0000 7fb2975e3700 1 -- 172.21.15.114:0/2727653598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb29019c010 con 0x7fb2900ffab0 2024-02-19T19:05:51.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.456+0000 7fb2975e3700 1 -- 172.21.15.114:0/2727653598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb290108610 con 0x7fb2900ffab0 2024-02-19T19:05:51.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.457+0000 7fb2867fc700 1 -- 172.21.15.114:0/2727653598 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7fb28c009730 con 0x7fb2900ffab0 2024-02-19T19:05:51.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.457+0000 7fb2867fc700 1 --2- 172.21.15.114:0/2727653598 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb27c06d8a0 0x7fb27c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:51.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.457+0000 7fb2867fc700 1 -- 172.21.15.114:0/2727653598 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6790+0+0 (secure 0 0 0) 0x7fb28c08d4d0 con 0x7fb2900ffab0 2024-02-19T19:05:51.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.457+0000 7fb294b7e700 1 --2- 172.21.15.114:0/2727653598 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb27c06d8a0 0x7fb27c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:51.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.458+0000 7fb294b7e700 1 --2- 172.21.15.114:0/2727653598 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb27c06d8a0 0x7fb27c06fd60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb2900694f0 tx=0x7fb28000b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:51.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.463+0000 7fb2867fc700 1 -- 172.21.15.114:0/2727653598 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb28c05a6e0 con 0x7fb2900ffab0 2024-02-19T19:05:51.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.679+0000 7fb2975e3700 1 -- 172.21.15.114:0/2727653598 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb290105600 con 0x7fb27c06d8a0 2024-02-19T19:05:51.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.690+0000 7fb2867fc700 1 -- 172.21.15.114:0/2727653598 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fb290105600 con 0x7fb27c06d8a0 2024-02-19T19:05:51.711 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:05:51.712 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (9m) 2s ago 12m 26.4M - 0.25.0 c8568f914cd2 30444b676ca4 2024-02-19T19:05:51.712 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (12m) 2s ago 12m 8212k - 18.2.1 94eadd990e0a 4918cd87e312 2024-02-19T19:05:51.712 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi174 smithi174 running (7m) 63s ago 10m 7847k - 18.2.1 94eadd990e0a 516c34431b93 2024-02-19T19:05:51.712 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (72s) 2s ago 12m 6660k - 19.0.0-1450-gf78a58c0 f081f847070a f10b46e9d636 2024-02-19T19:05:51.712 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi174 smithi174 running (66s) 63s ago 10m 6660k - 19.0.0-1450-gf78a58c0 f081f847070a fb3b17c81bd8 2024-02-19T19:05:51.712 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (9m) 2s ago 11m 85.6M - 9.4.7 954c08fa6188 478bd533e827 2024-02-19T19:05:51.712 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tcwpod smithi114 running (5m) 2s ago 5m 27.5M - 18.2.1 94eadd990e0a 420175777cf4 2024-02-19T19:05:51.712 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.vpoeop smithi114 running (5m) 2s ago 5m 24.5M - 18.2.1 94eadd990e0a f81a79e53ead 2024-02-19T19:05:51.712 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.stnfnf smithi174 running (5m) 63s ago 5m 26.0M - 18.2.1 94eadd990e0a 3df35954a492 2024-02-19T19:05:51.712 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.znhuvd smithi174 running (5m) 63s ago 5m 24.2M - 18.2.1 94eadd990e0a 2d2267494ae1 2024-02-19T19:05:51.712 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.pcvtrk smithi114 *:8443,9283,8765 running (3m) 2s ago 14m 502M - 19.0.0-1450-gf78a58c0 f081f847070a 05b5c06122b1 2024-02-19T19:05:51.712 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi174.lydvnj smithi174 *:8443,9283,8765 running (2m) 63s ago 10m 450M - 19.0.0-1450-gf78a58c0 f081f847070a f962b1af5aed 2024-02-19T19:05:51.712 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (2m) 2s ago 14m 46.8M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e51f650f9be9 2024-02-19T19:05:51.713 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi174 smithi174 running (117s) 63s ago 10m 33.9M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e811581f05fb 2024-02-19T19:05:51.713 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (12m) 2s ago 12m 14.6M - 1.5.0 0da6a335fe13 b6b258074cce 2024-02-19T19:05:51.713 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi174 smithi174 *:9100 running (10m) 63s ago 10m 14.9M - 1.5.0 0da6a335fe13 d0b4fdf67b22 2024-02-19T19:05:51.713 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (45s) 2s ago 9m 51.1M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 764769439e0e 2024-02-19T19:05:51.713 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (7s) 2s ago 9m 12.7M 1216M 19.0.0-1450-gf78a58c0 f081f847070a c1c32dbdf573 2024-02-19T19:05:51.713 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (8m) 2s ago 8m 53.3M 1216M 18.2.1 94eadd990e0a 12953988af86 2024-02-19T19:05:51.713 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi174 running (8m) 63s ago 8m 49.2M 2240M 18.2.1 94eadd990e0a 979f45cabbce 2024-02-19T19:05:51.713 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi174 running (7m) 63s ago 7m 49.7M 2240M 18.2.1 94eadd990e0a 8b8c3745b8c5 2024-02-19T19:05:51.713 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi174 running (7m) 63s ago 7m 52.2M 2240M 18.2.1 94eadd990e0a 5771a995ed75 2024-02-19T19:05:51.713 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (2m) 2s ago 10m 63.1M - 2.43.0 a07b618ecd1d a36922c7c9a1 2024-02-19T19:05:51.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.692+0000 7fb2975e3700 1 -- 172.21.15.114:0/2727653598 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb27c06d8a0 msgr2=0x7fb27c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:51.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.692+0000 7fb2975e3700 1 --2- 172.21.15.114:0/2727653598 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb27c06d8a0 0x7fb27c06fd60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb2900694f0 tx=0x7fb28000b540 comp rx=0 tx=0).stop 2024-02-19T19:05:51.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.692+0000 7fb2975e3700 1 -- 172.21.15.114:0/2727653598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2900ffab0 msgr2=0x7fb290068f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:51.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.692+0000 7fb2975e3700 1 --2- 172.21.15.114:0/2727653598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2900ffab0 0x7fb290068f90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb28c00d900 tx=0x7fb28c00dc10 comp rx=0 tx=0).stop 2024-02-19T19:05:51.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.692+0000 7fb2975e3700 1 -- 172.21.15.114:0/2727653598 shutdown_connections 2024-02-19T19:05:51.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.692+0000 7fb2975e3700 1 --2- 172.21.15.114:0/2727653598 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb27c06d8a0 0x7fb27c06fd60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.692+0000 7fb2975e3700 1 --2- 172.21.15.114:0/2727653598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2900ffab0 0x7fb290068f90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.692+0000 7fb2975e3700 1 --2- 172.21.15.114:0/2727653598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb290100cb0 0x7fb29019b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.692+0000 7fb2975e3700 1 -- 172.21.15.114:0/2727653598 >> 172.21.15.114:0/2727653598 conn(0x7fb2900fb030 msgr2=0x7fb290103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:51.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.692+0000 7fb2975e3700 1 -- 172.21.15.114:0/2727653598 shutdown_connections 2024-02-19T19:05:51.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.692+0000 7fb2975e3700 1 -- 172.21.15.114:0/2727653598 wait complete. 2024-02-19T19:05:51.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.798+0000 7fcbb8a89700 1 -- 172.21.15.114:0/3518717444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb4100bb0 msgr2=0x7fcbb4100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:51.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.798+0000 7fcbb8a89700 1 --2- 172.21.15.114:0/3518717444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb4100bb0 0x7fcbb4100fd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009910 tx=0x7fcba4009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:51.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.798+0000 7fcbb8a89700 1 -- 172.21.15.114:0/3518717444 shutdown_connections 2024-02-19T19:05:51.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.798+0000 7fcbb8a89700 1 --2- 172.21.15.114:0/3518717444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbb4101db0 0x7fcbb4102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.798+0000 7fcbb8a89700 1 --2- 172.21.15.114:0/3518717444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb4100bb0 0x7fcbb4100fd0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.798+0000 7fcbb8a89700 1 -- 172.21.15.114:0/3518717444 >> 172.21.15.114:0/3518717444 conn(0x7fcbb40fc110 msgr2=0x7fcbb40fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:51.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.798+0000 7fcbb8a89700 1 -- 172.21.15.114:0/3518717444 shutdown_connections 2024-02-19T19:05:51.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.798+0000 7fcbb8a89700 1 -- 172.21.15.114:0/3518717444 wait complete. 2024-02-19T19:05:51.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.799+0000 7fcbb8a89700 1 Processor -- start 2024-02-19T19:05:51.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.799+0000 7fcbb8a89700 1 -- start start 2024-02-19T19:05:51.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.799+0000 7fcbb8a89700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb4100bb0 0x7fcbb4196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:51.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.799+0000 7fcbb8a89700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbb4101db0 0x7fcbb4196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:51.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.799+0000 7fcbb8a89700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb4196f60 con 0x7fcbb4100bb0 2024-02-19T19:05:51.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.799+0000 7fcbb8a89700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb41970a0 con 0x7fcbb4101db0 2024-02-19T19:05:51.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.799+0000 7fcbb259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb4100bb0 0x7fcbb4196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:51.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.799+0000 7fcbb259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb4100bb0 0x7fcbb4196400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42704/0 (socket says 172.21.15.114:42704) 2024-02-19T19:05:51.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.799+0000 7fcbb259c700 1 -- 172.21.15.114:0/3549412426 learned_addr learned my addr 172.21.15.114:0/3549412426 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:05:51.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.800+0000 7fcbb1d9b700 1 --2- 172.21.15.114:0/3549412426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbb4101db0 0x7fcbb4196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:51.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.800+0000 7fcbb259c700 1 -- 172.21.15.114:0/3549412426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbb4101db0 msgr2=0x7fcbb4196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:51.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.800+0000 7fcbb259c700 1 --2- 172.21.15.114:0/3549412426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbb4101db0 0x7fcbb4196940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:51.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.800+0000 7fcbb259c700 1 -- 172.21.15.114:0/3549412426 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba40095f0 con 0x7fcbb4100bb0 2024-02-19T19:05:51.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.800+0000 7fcbb259c700 1 --2- 172.21.15.114:0/3549412426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb4100bb0 0x7fcbb4196400 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcba400bda0 tx=0x7fcba400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:51.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.800+0000 7fcba37fe700 1 -- 172.21.15.114:0/3549412426 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcba401d070 con 0x7fcbb4100bb0 2024-02-19T19:05:51.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.800+0000 7fcba37fe700 1 -- 172.21.15.114:0/3549412426 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcba4022470 con 0x7fcbb4100bb0 2024-02-19T19:05:51.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.800+0000 7fcbb8a89700 1 -- 172.21.15.114:0/3549412426 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb419baf0 con 0x7fcbb4100bb0 2024-02-19T19:05:51.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.800+0000 7fcba37fe700 1 -- 172.21.15.114:0/3549412426 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcba400f670 con 0x7fcbb4100bb0 2024-02-19T19:05:51.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.800+0000 7fcbb8a89700 1 -- 172.21.15.114:0/3549412426 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb419c060 con 0x7fcbb4100bb0 2024-02-19T19:05:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.801+0000 7fcbb8a89700 1 -- 172.21.15.114:0/3549412426 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbb4109790 con 0x7fcbb4100bb0 2024-02-19T19:05:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.802+0000 7fcba37fe700 1 -- 172.21.15.114:0/3549412426 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7fcba4004af0 con 0x7fcbb4100bb0 2024-02-19T19:05:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.802+0000 7fcba37fe700 1 --2- 172.21.15.114:0/3549412426 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fcb9c06d8a0 0x7fcb9c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.802+0000 7fcba37fe700 1 -- 172.21.15.114:0/3549412426 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6790+0+0 (secure 0 0 0) 0x7fcba408ec90 con 0x7fcbb4100bb0 2024-02-19T19:05:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.802+0000 7fcbb1d9b700 1 --2- 172.21.15.114:0/3549412426 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fcb9c06d8a0 0x7fcb9c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.802+0000 7fcbb1d9b700 1 --2- 172.21.15.114:0/3549412426 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fcb9c06d8a0 0x7fcb9c06fd60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009a10 tx=0x7fcba8009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:51.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:51.807+0000 7fcba37fe700 1 -- 172.21.15.114:0/3549412426 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcba405c070 con 0x7fcbb4100bb0 2024-02-19T19:05:52.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:51 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:52.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:51 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:52.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:51 smithi114 ceph-mon[92983]: osdmap e62: 6 total, 6 up, 6 in 2024-02-19T19:05:52.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:51 smithi114 ceph-mon[92983]: Health check update: Degraded data redundancy: 18/129 objects degraded (13.953%), 9 pgs degraded (PG_DEGRADED) 2024-02-19T19:05:52.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.096+0000 7fcbb8a89700 1 -- 172.21.15.114:0/3549412426 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcbb419c6a0 con 0x7fcbb4100bb0 2024-02-19T19:05:52.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.096+0000 7fcba37fe700 1 -- 172.21.15.114:0/3549412426 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7fcba4027070 con 0x7fcbb4100bb0 2024-02-19T19:05:52.117 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:05:52.117 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-02-19T19:05:52.117 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:05:52.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:05:52.117 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-02-19T19:05:52.117 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:05:52.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:05:52.117 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-02-19T19:05:52.117 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4, 2024-02-19T19:05:52.118 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:05:52.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:05:52.118 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-02-19T19:05:52.118 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:05:52.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:05:52.118 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-02-19T19:05:52.118 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 8, 2024-02-19T19:05:52.118 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 6 2024-02-19T19:05:52.118 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-02-19T19:05:52.118 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:05:52.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.098+0000 7fcbb8a89700 1 -- 172.21.15.114:0/3549412426 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fcb9c06d8a0 msgr2=0x7fcb9c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:52.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.098+0000 7fcbb8a89700 1 --2- 172.21.15.114:0/3549412426 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fcb9c06d8a0 0x7fcb9c06fd60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009a10 tx=0x7fcba8009190 comp rx=0 tx=0).stop 2024-02-19T19:05:52.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.098+0000 7fcbb8a89700 1 -- 172.21.15.114:0/3549412426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb4100bb0 msgr2=0x7fcbb4196400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:52.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.098+0000 7fcbb8a89700 1 --2- 172.21.15.114:0/3549412426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb4100bb0 0x7fcbb4196400 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcba400bda0 tx=0x7fcba400bdd0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.098+0000 7fcbb8a89700 1 -- 172.21.15.114:0/3549412426 shutdown_connections 2024-02-19T19:05:52.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.098+0000 7fcbb8a89700 1 --2- 172.21.15.114:0/3549412426 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fcb9c06d8a0 0x7fcb9c06fd60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.098+0000 7fcbb8a89700 1 --2- 172.21.15.114:0/3549412426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcbb4100bb0 0x7fcbb4196400 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.098+0000 7fcbb8a89700 1 --2- 172.21.15.114:0/3549412426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbb4101db0 0x7fcbb4196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.098+0000 7fcbb8a89700 1 -- 172.21.15.114:0/3549412426 >> 172.21.15.114:0/3549412426 conn(0x7fcbb40fc110 msgr2=0x7fcbb4104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:52.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.099+0000 7fcbb8a89700 1 -- 172.21.15.114:0/3549412426 shutdown_connections 2024-02-19T19:05:52.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.099+0000 7fcbb8a89700 1 -- 172.21.15.114:0/3549412426 wait complete. 2024-02-19T19:05:52.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:51 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:52.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:51 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:52.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:51 smithi174 ceph-mon[87279]: osdmap e62: 6 total, 6 up, 6 in 2024-02-19T19:05:52.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:51 smithi174 ceph-mon[87279]: Health check update: Degraded data redundancy: 18/129 objects degraded (13.953%), 9 pgs degraded (PG_DEGRADED) 2024-02-19T19:05:52.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.210+0000 7fc384563700 1 -- 172.21.15.114:0/3391466864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc37c0ffab0 msgr2=0x7fc37c0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:52.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.210+0000 7fc384563700 1 --2- 172.21.15.114:0/3391466864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc37c0ffab0 0x7fc37c0ffed0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009910 tx=0x7fc36c009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.210+0000 7fc384563700 1 -- 172.21.15.114:0/3391466864 shutdown_connections 2024-02-19T19:05:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.210+0000 7fc384563700 1 --2- 172.21.15.114:0/3391466864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc37c100cb0 0x7fc37c101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.210+0000 7fc384563700 1 --2- 172.21.15.114:0/3391466864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc37c0ffab0 0x7fc37c0ffed0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.210+0000 7fc384563700 1 -- 172.21.15.114:0/3391466864 >> 172.21.15.114:0/3391466864 conn(0x7fc37c0fb030 msgr2=0x7fc37c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.210+0000 7fc384563700 1 -- 172.21.15.114:0/3391466864 shutdown_connections 2024-02-19T19:05:52.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.210+0000 7fc384563700 1 -- 172.21.15.114:0/3391466864 wait complete. 2024-02-19T19:05:52.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.211+0000 7fc384563700 1 Processor -- start 2024-02-19T19:05:52.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.211+0000 7fc384563700 1 -- start start 2024-02-19T19:05:52.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.211+0000 7fc384563700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc37c0ffab0 0x7fc37c1952e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:52.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.211+0000 7fc384563700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc37c100cb0 0x7fc37c195820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:52.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.211+0000 7fc384563700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c195e40 con 0x7fc37c100cb0 2024-02-19T19:05:52.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.211+0000 7fc384563700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c195f80 con 0x7fc37c0ffab0 2024-02-19T19:05:52.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.211+0000 7fc381afe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc37c100cb0 0x7fc37c195820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:52.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.211+0000 7fc381afe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc37c100cb0 0x7fc37c195820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42716/0 (socket says 172.21.15.114:42716) 2024-02-19T19:05:52.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.211+0000 7fc381afe700 1 -- 172.21.15.114:0/1968564891 learned_addr learned my addr 172.21.15.114:0/1968564891 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:05:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.211+0000 7fc3822ff700 1 --2- 172.21.15.114:0/1968564891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc37c0ffab0 0x7fc37c1952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.211+0000 7fc381afe700 1 -- 172.21.15.114:0/1968564891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc37c0ffab0 msgr2=0x7fc37c1952e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.211+0000 7fc381afe700 1 --2- 172.21.15.114:0/1968564891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc37c0ffab0 0x7fc37c1952e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.211+0000 7fc381afe700 1 -- 172.21.15.114:0/1968564891 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c0095f0 con 0x7fc37c100cb0 2024-02-19T19:05:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.211+0000 7fc3822ff700 1 --2- 172.21.15.114:0/1968564891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc37c0ffab0 0x7fc37c1952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:05:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.212+0000 7fc381afe700 1 --2- 172.21.15.114:0/1968564891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc37c100cb0 0x7fc37c195820 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc37800b4b0 tx=0x7fc37800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.212+0000 7fc3737fe700 1 -- 172.21.15.114:0/1968564891 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc378011800 con 0x7fc37c100cb0 2024-02-19T19:05:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.212+0000 7fc384563700 1 -- 172.21.15.114:0/1968564891 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37c068470 con 0x7fc37c100cb0 2024-02-19T19:05:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.212+0000 7fc3737fe700 1 -- 172.21.15.114:0/1968564891 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc378011e10 con 0x7fc37c100cb0 2024-02-19T19:05:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.212+0000 7fc3737fe700 1 -- 172.21.15.114:0/1968564891 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc37800f2e0 con 0x7fc37c100cb0 2024-02-19T19:05:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.212+0000 7fc384563700 1 -- 172.21.15.114:0/1968564891 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37c068a40 con 0x7fc37c100cb0 2024-02-19T19:05:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.213+0000 7fc384563700 1 -- 172.21.15.114:0/1968564891 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37c108690 con 0x7fc37c100cb0 2024-02-19T19:05:52.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.213+0000 7fc3737fe700 1 -- 172.21.15.114:0/1968564891 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7fc378010380 con 0x7fc37c100cb0 2024-02-19T19:05:52.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.214+0000 7fc3737fe700 1 --2- 172.21.15.114:0/1968564891 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc36806d8a0 0x7fc36806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:52.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.214+0000 7fc3737fe700 1 -- 172.21.15.114:0/1968564891 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6790+0+0 (secure 0 0 0) 0x7fc37808d230 con 0x7fc37c100cb0 2024-02-19T19:05:52.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.214+0000 7fc3822ff700 1 --2- 172.21.15.114:0/1968564891 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc36806d8a0 0x7fc36806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:52.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.214+0000 7fc3822ff700 1 --2- 172.21.15.114:0/1968564891 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc36806d8a0 0x7fc36806fd60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc36c00b5c0 tx=0x7fc36c005e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:52.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.218+0000 7fc3737fe700 1 -- 172.21.15.114:0/1968564891 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc37805a440 con 0x7fc37c100cb0 2024-02-19T19:05:52.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.461+0000 7fc384563700 1 -- 172.21.15.114:0/1968564891 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc37c108850 con 0x7fc37c100cb0 2024-02-19T19:05:52.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.462+0000 7fc3737fe700 1 -- 172.21.15.114:0/1968564891 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fc378059fd0 con 0x7fc37c100cb0 2024-02-19T19:05:52.482 INFO:teuthology.orchestra.run.smithi114.stdout:e18 2024-02-19T19:05:52.482 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:05:52.482 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:05:52.482 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-02-19T19:05:52.483 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:05:52.483 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:05:52.483 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-02-19T19:05:52.483 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 18 2024-02-19T19:05:52.483 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:05:52.483 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:05:52.483 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-02-19T19:00:21.203922+0000 2024-02-19T19:05:52.483 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-02-19T19:05:52.483 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-02-19T19:05:52.483 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-02-19T19:05:52.483 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-02-19T19:05:52.483 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-02-19T19:05:52.483 INFO:teuthology.orchestra.run.smithi114.stdout:max_xattr_size 65536 2024-02-19T19:05:52.483 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-02-19T19:05:52.484 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-02-19T19:05:52.484 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 47 2024-02-19T19:05:52.484 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:05:52.484 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 2 2024-02-19T19:05:52.484 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1 2024-02-19T19:05:52.484 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=14530,1=24269} 2024-02-19T19:05:52.484 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-02-19T19:05:52.484 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-02-19T19:05:52.484 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-02-19T19:05:52.484 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-02-19T19:05:52.484 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-02-19T19:05:52.484 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data enabled 2024-02-19T19:05:52.484 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-02-19T19:05:52.485 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-02-19T19:05:52.485 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 1 2024-02-19T19:05:52.485 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:05:52.485 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.vpoeop{0:14540} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:05:52.485 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.znhuvd{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:05:52.485 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.stnfnf{1:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:05:52.485 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:05:52.485 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:05:52.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.464+0000 7fc384563700 1 -- 172.21.15.114:0/1968564891 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc36806d8a0 msgr2=0x7fc36806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:52.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.464+0000 7fc384563700 1 --2- 172.21.15.114:0/1968564891 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc36806d8a0 0x7fc36806fd60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc36c00b5c0 tx=0x7fc36c005e20 comp rx=0 tx=0).stop 2024-02-19T19:05:52.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.464+0000 7fc384563700 1 -- 172.21.15.114:0/1968564891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc37c100cb0 msgr2=0x7fc37c195820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:52.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.464+0000 7fc384563700 1 --2- 172.21.15.114:0/1968564891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc37c100cb0 0x7fc37c195820 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc37800b4b0 tx=0x7fc37800b870 comp rx=0 tx=0).stop 2024-02-19T19:05:52.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.464+0000 7fc384563700 1 -- 172.21.15.114:0/1968564891 shutdown_connections 2024-02-19T19:05:52.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.464+0000 7fc384563700 1 --2- 172.21.15.114:0/1968564891 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc36806d8a0 0x7fc36806fd60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.464+0000 7fc384563700 1 --2- 172.21.15.114:0/1968564891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc37c0ffab0 0x7fc37c1952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.464+0000 7fc384563700 1 --2- 172.21.15.114:0/1968564891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc37c100cb0 0x7fc37c195820 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.464+0000 7fc384563700 1 -- 172.21.15.114:0/1968564891 >> 172.21.15.114:0/1968564891 conn(0x7fc37c0fb030 msgr2=0x7fc37c103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:52.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.464+0000 7fc384563700 1 -- 172.21.15.114:0/1968564891 shutdown_connections 2024-02-19T19:05:52.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.464+0000 7fc384563700 1 -- 172.21.15.114:0/1968564891 wait complete. 2024-02-19T19:05:52.487 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 18 2024-02-19T19:05:52.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.572+0000 7f82a6d81700 1 -- 172.21.15.114:0/1301445543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82a0103fd0 msgr2=0x7f82a01063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.572+0000 7f82a6d81700 1 --2- 172.21.15.114:0/1301445543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82a0103fd0 0x7f82a01063c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f828c009910 tx=0x7f828c009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.573+0000 7f82a6d81700 1 -- 172.21.15.114:0/1301445543 shutdown_connections 2024-02-19T19:05:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.573+0000 7f82a6d81700 1 --2- 172.21.15.114:0/1301445543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82a0103fd0 0x7f82a01063c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.573+0000 7f82a6d81700 1 --2- 172.21.15.114:0/1301445543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82a01016a0 0x7f82a0103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.573+0000 7f82a6d81700 1 -- 172.21.15.114:0/1301445543 >> 172.21.15.114:0/1301445543 conn(0x7f82a00fb050 msgr2=0x7f82a00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.573+0000 7f82a6d81700 1 -- 172.21.15.114:0/1301445543 shutdown_connections 2024-02-19T19:05:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.573+0000 7f82a6d81700 1 -- 172.21.15.114:0/1301445543 wait complete. 2024-02-19T19:05:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.573+0000 7f82a6d81700 1 Processor -- start 2024-02-19T19:05:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.573+0000 7f82a6d81700 1 -- start start 2024-02-19T19:05:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.573+0000 7f82a6d81700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82a01016a0 0x7f82a0196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.573+0000 7f82a6d81700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82a0103fd0 0x7f82a0196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.573+0000 7f82a6d81700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a0196e80 con 0x7f82a01016a0 2024-02-19T19:05:52.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.573+0000 7f82a6d81700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a0196fc0 con 0x7f82a0103fd0 2024-02-19T19:05:52.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.573+0000 7f82a4b1d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82a01016a0 0x7f82a0196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:52.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.574+0000 7f82a4b1d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82a01016a0 0x7f82a0196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42728/0 (socket says 172.21.15.114:42728) 2024-02-19T19:05:52.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.574+0000 7f82a4b1d700 1 -- 172.21.15.114:0/605057282 learned_addr learned my addr 172.21.15.114:0/605057282 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:05:52.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.574+0000 7f829ffff700 1 --2- 172.21.15.114:0/605057282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82a0103fd0 0x7f82a0196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:52.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.574+0000 7f82a4b1d700 1 -- 172.21.15.114:0/605057282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82a0103fd0 msgr2=0x7f82a0196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:52.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.574+0000 7f82a4b1d700 1 --2- 172.21.15.114:0/605057282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82a0103fd0 0x7f82a0196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.574+0000 7f82a4b1d700 1 -- 172.21.15.114:0/605057282 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c0095f0 con 0x7f82a01016a0 2024-02-19T19:05:52.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.574+0000 7f82a4b1d700 1 --2- 172.21.15.114:0/605057282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82a01016a0 0x7f82a0196320 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f829400d8d0 tx=0x7f829400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:52.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.574+0000 7f829dffb700 1 -- 172.21.15.114:0/605057282 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82940098a0 con 0x7f82a01016a0 2024-02-19T19:05:52.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.574+0000 7f82a6d81700 1 -- 172.21.15.114:0/605057282 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a019ba70 con 0x7f82a01016a0 2024-02-19T19:05:52.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.574+0000 7f829dffb700 1 -- 172.21.15.114:0/605057282 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8294010460 con 0x7f82a01016a0 2024-02-19T19:05:52.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.574+0000 7f829dffb700 1 -- 172.21.15.114:0/605057282 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f829400f5f0 con 0x7f82a01016a0 2024-02-19T19:05:52.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.575+0000 7f82a6d81700 1 -- 172.21.15.114:0/605057282 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a019bfc0 con 0x7f82a01016a0 2024-02-19T19:05:52.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.576+0000 7f82a6d81700 1 -- 172.21.15.114:0/605057282 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a0190520 con 0x7f82a01016a0 2024-02-19T19:05:52.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.576+0000 7f829dffb700 1 -- 172.21.15.114:0/605057282 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f829400f750 con 0x7f82a01016a0 2024-02-19T19:05:52.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.576+0000 7f829dffb700 1 --2- 172.21.15.114:0/605057282 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f829006d8a0 0x7f829006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:52.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.576+0000 7f829dffb700 1 -- 172.21.15.114:0/605057282 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f829408c960 con 0x7f82a01016a0 2024-02-19T19:05:52.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.577+0000 7f829ffff700 1 --2- 172.21.15.114:0/605057282 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f829006d8a0 0x7f829006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:52.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.577+0000 7f829ffff700 1 --2- 172.21.15.114:0/605057282 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f829006d8a0 0x7f829006fd60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f828c006010 tx=0x7f828c00b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:52.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.581+0000 7f829dffb700 1 -- 172.21.15.114:0/605057282 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8294059b60 con 0x7f82a01016a0 2024-02-19T19:05:52.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.803+0000 7f82a6d81700 1 -- 172.21.15.114:0/605057282 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f82a00611d0 con 0x7f829006d8a0 2024-02-19T19:05:52.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.804+0000 7f829dffb700 1 -- 172.21.15.114:0/605057282 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f82a00611d0 con 0x7f829006d8a0 2024-02-19T19:05:52.824 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:05:52.824 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:05:52.824 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-02-19T19:05:52.825 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:05:52.825 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [ 2024-02-19T19:05:52.825 INFO:teuthology.orchestra.run.smithi114.stdout: "crash", 2024-02-19T19:05:52.825 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr", 2024-02-19T19:05:52.825 INFO:teuthology.orchestra.run.smithi114.stdout: "mon" 2024-02-19T19:05:52.825 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-02-19T19:05:52.825 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "8/23 daemons upgraded", 2024-02-19T19:05:52.825 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "Currently upgrading osd daemons", 2024-02-19T19:05:52.825 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-02-19T19:05:52.825 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:05:52.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.806+0000 7f82a6d81700 1 -- 172.21.15.114:0/605057282 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f829006d8a0 msgr2=0x7f829006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:52.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.806+0000 7f82a6d81700 1 --2- 172.21.15.114:0/605057282 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f829006d8a0 0x7f829006fd60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f828c006010 tx=0x7f828c00b540 comp rx=0 tx=0).stop 2024-02-19T19:05:52.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.806+0000 7f82a6d81700 1 -- 172.21.15.114:0/605057282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82a01016a0 msgr2=0x7f82a0196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:52.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.806+0000 7f82a6d81700 1 --2- 172.21.15.114:0/605057282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82a01016a0 0x7f82a0196320 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f829400d8d0 tx=0x7f829400dbe0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.806+0000 7f82a6d81700 1 -- 172.21.15.114:0/605057282 shutdown_connections 2024-02-19T19:05:52.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.806+0000 7f82a6d81700 1 --2- 172.21.15.114:0/605057282 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f829006d8a0 0x7f829006fd60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.806+0000 7f82a6d81700 1 --2- 172.21.15.114:0/605057282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82a01016a0 0x7f82a0196320 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.806+0000 7f82a6d81700 1 --2- 172.21.15.114:0/605057282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82a0103fd0 0x7f82a0196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.806+0000 7f82a6d81700 1 -- 172.21.15.114:0/605057282 >> 172.21.15.114:0/605057282 conn(0x7f82a00fb050 msgr2=0x7f82a00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:52.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.806+0000 7f82a6d81700 1 -- 172.21.15.114:0/605057282 shutdown_connections 2024-02-19T19:05:52.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.806+0000 7f82a6d81700 1 -- 172.21.15.114:0/605057282 wait complete. 2024-02-19T19:05:52.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:52 smithi174 ceph-mon[87279]: from='client.34248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:52.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:52 smithi174 ceph-mon[87279]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:52.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:52 smithi174 ceph-mon[87279]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:52.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:52 smithi174 ceph-mon[87279]: pgmap v81: 65 pgs: 9 remapped+peering, 17 peering, 5 active+undersized+degraded, 7 active+undersized, 27 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 7/129 objects degraded (5.426%) 2024-02-19T19:05:52.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:52 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/3549412426' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:52.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:52 smithi174 ceph-mon[87279]: osdmap e63: 6 total, 6 up, 6 in 2024-02-19T19:05:52.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:52 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/1968564891' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:05:52.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.920+0000 7fcce45b2700 1 -- 172.21.15.114:0/3821340069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccdc100bb0 msgr2=0x7fccdc100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:52.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.920+0000 7fcce45b2700 1 --2- 172.21.15.114:0/3821340069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccdc100bb0 0x7fccdc100fd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fccd8009910 tx=0x7fccd8009c20 comp rx=0 tx=0).stop 2024-02-19T19:05:52.941 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:52 smithi114 ceph-mon[92983]: from='client.34248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:52.941 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:52 smithi114 ceph-mon[92983]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:52.941 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:52 smithi114 ceph-mon[92983]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:52.941 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:52 smithi114 ceph-mon[92983]: pgmap v81: 65 pgs: 9 remapped+peering, 17 peering, 5 active+undersized+degraded, 7 active+undersized, 27 active+clean; 594 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 7/129 objects degraded (5.426%) 2024-02-19T19:05:52.941 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:52 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/3549412426' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:52.941 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:52 smithi114 ceph-mon[92983]: osdmap e63: 6 total, 6 up, 6 in 2024-02-19T19:05:52.941 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:52 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/1968564891' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:05:52.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.920+0000 7fcce45b2700 1 -- 172.21.15.114:0/3821340069 shutdown_connections 2024-02-19T19:05:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.920+0000 7fcce45b2700 1 --2- 172.21.15.114:0/3821340069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccdc101db0 0x7fccdc102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.920+0000 7fcce45b2700 1 --2- 172.21.15.114:0/3821340069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccdc100bb0 0x7fccdc100fd0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.920+0000 7fcce45b2700 1 -- 172.21.15.114:0/3821340069 >> 172.21.15.114:0/3821340069 conn(0x7fccdc0fc110 msgr2=0x7fccdc0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.920+0000 7fcce45b2700 1 -- 172.21.15.114:0/3821340069 shutdown_connections 2024-02-19T19:05:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.920+0000 7fcce45b2700 1 -- 172.21.15.114:0/3821340069 wait complete. 2024-02-19T19:05:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.921+0000 7fcce45b2700 1 Processor -- start 2024-02-19T19:05:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.921+0000 7fcce45b2700 1 -- start start 2024-02-19T19:05:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.921+0000 7fcce45b2700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccdc100bb0 0x7fccdc196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.921+0000 7fcce45b2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccdc101db0 0x7fccdc196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.921+0000 7fcce45b2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc196f60 con 0x7fccdc101db0 2024-02-19T19:05:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.921+0000 7fcce45b2700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc1970a0 con 0x7fccdc100bb0 2024-02-19T19:05:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.921+0000 7fcce1b4d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccdc101db0 0x7fccdc196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.921+0000 7fcce1b4d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccdc101db0 0x7fccdc196940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42740/0 (socket says 172.21.15.114:42740) 2024-02-19T19:05:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.921+0000 7fcce1b4d700 1 -- 172.21.15.114:0/3692502949 learned_addr learned my addr 172.21.15.114:0/3692502949 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:05:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.922+0000 7fcce234e700 1 --2- 172.21.15.114:0/3692502949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccdc100bb0 0x7fccdc196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.922+0000 7fcce1b4d700 1 -- 172.21.15.114:0/3692502949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccdc100bb0 msgr2=0x7fccdc196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.922+0000 7fcce1b4d700 1 --2- 172.21.15.114:0/3692502949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccdc100bb0 0x7fccdc196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.922+0000 7fcce1b4d700 1 -- 172.21.15.114:0/3692502949 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd80095f0 con 0x7fccdc101db0 2024-02-19T19:05:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.922+0000 7fcce1b4d700 1 --2- 172.21.15.114:0/3692502949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccdc101db0 0x7fccdc196940 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00b4b0 tx=0x7fcccc00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.922+0000 7fccd37fe700 1 -- 172.21.15.114:0/3692502949 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcccc011800 con 0x7fccdc101db0 2024-02-19T19:05:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.922+0000 7fccd37fe700 1 -- 172.21.15.114:0/3692502949 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcccc011e10 con 0x7fccdc101db0 2024-02-19T19:05:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.922+0000 7fcce45b2700 1 -- 172.21.15.114:0/3692502949 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccdc19bb50 con 0x7fccdc101db0 2024-02-19T19:05:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.922+0000 7fccd37fe700 1 -- 172.21.15.114:0/3692502949 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcccc00f2e0 con 0x7fccdc101db0 2024-02-19T19:05:52.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.922+0000 7fcce45b2700 1 -- 172.21.15.114:0/3692502949 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccdc19c120 con 0x7fccdc101db0 2024-02-19T19:05:52.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.923+0000 7fcce45b2700 1 -- 172.21.15.114:0/3692502949 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccdc109790 con 0x7fccdc101db0 2024-02-19T19:05:52.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.923+0000 7fccd37fe700 1 -- 172.21.15.114:0/3692502949 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7fcccc010380 con 0x7fccdc101db0 2024-02-19T19:05:52.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.924+0000 7fccd37fe700 1 --2- 172.21.15.114:0/3692502949 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fccc806d8a0 0x7fccc806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:05:52.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.924+0000 7fccd37fe700 1 -- 172.21.15.114:0/3692502949 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6790+0+0 (secure 0 0 0) 0x7fcccc08d260 con 0x7fccdc101db0 2024-02-19T19:05:52.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.924+0000 7fcce234e700 1 --2- 172.21.15.114:0/3692502949 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fccc806d8a0 0x7fccc806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:05:52.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.924+0000 7fcce234e700 1 --2- 172.21.15.114:0/3692502949 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fccc806d8a0 0x7fccc806fd60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fccd8006010 tx=0x7fccd8005920 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:05:52.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:52.928+0000 7fccd37fe700 1 -- 172.21.15.114:0/3692502949 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcccc05a470 con 0x7fccdc101db0 2024-02-19T19:05:53.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:53.219+0000 7fcce45b2700 1 -- 172.21.15.114:0/3692502949 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fccdc109980 con 0x7fccdc101db0 2024-02-19T19:05:53.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:53.219+0000 7fccd37fe700 1 -- 172.21.15.114:0/3692502949 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+645 (secure 0 0 0) 0x7fcccc05a000 con 0x7fccdc101db0 2024-02-19T19:05:53.240 INFO:teuthology.orchestra.run.smithi114.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; Degraded data redundancy: 7/129 objects degraded (5.426%), 5 pgs degraded 2024-02-19T19:05:53.241 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:05:53.241 INFO:teuthology.orchestra.run.smithi114.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:05:53.241 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 7/129 objects degraded (5.426%), 5 pgs degraded 2024-02-19T19:05:53.241 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.6 is active+undersized+degraded, acting [3,4] 2024-02-19T19:05:53.241 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.9 is active+undersized+degraded, acting [4,0] 2024-02-19T19:05:53.241 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.a is active+undersized+degraded, acting [4,3] 2024-02-19T19:05:53.241 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.d is active+undersized+degraded, acting [3,2] 2024-02-19T19:05:53.241 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.1b is active+undersized+degraded, acting [0,5] 2024-02-19T19:05:53.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:53.222+0000 7fcce45b2700 1 -- 172.21.15.114:0/3692502949 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fccc806d8a0 msgr2=0x7fccc806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:53.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:53.222+0000 7fcce45b2700 1 --2- 172.21.15.114:0/3692502949 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fccc806d8a0 0x7fccc806fd60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fccd8006010 tx=0x7fccd8005920 comp rx=0 tx=0).stop 2024-02-19T19:05:53.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:53.222+0000 7fcce45b2700 1 -- 172.21.15.114:0/3692502949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccdc101db0 msgr2=0x7fccdc196940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:05:53.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:53.222+0000 7fcce45b2700 1 --2- 172.21.15.114:0/3692502949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccdc101db0 0x7fccdc196940 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcccc00b4b0 tx=0x7fcccc00b870 comp rx=0 tx=0).stop 2024-02-19T19:05:53.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:53.222+0000 7fcce45b2700 1 -- 172.21.15.114:0/3692502949 shutdown_connections 2024-02-19T19:05:53.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:53.222+0000 7fcce45b2700 1 --2- 172.21.15.114:0/3692502949 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fccc806d8a0 0x7fccc806fd60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:53.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:53.222+0000 7fcce45b2700 1 --2- 172.21.15.114:0/3692502949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccdc100bb0 0x7fccdc196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:53.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:53.222+0000 7fcce45b2700 1 --2- 172.21.15.114:0/3692502949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fccdc101db0 0x7fccdc196940 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:05:53.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:53.222+0000 7fcce45b2700 1 -- 172.21.15.114:0/3692502949 >> 172.21.15.114:0/3692502949 conn(0x7fccdc0fc110 msgr2=0x7fccdc104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:05:53.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:53.222+0000 7fcce45b2700 1 -- 172.21.15.114:0/3692502949 shutdown_connections 2024-02-19T19:05:53.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:05:53.222+0000 7fcce45b2700 1 -- 172.21.15.114:0/3692502949 wait complete. 2024-02-19T19:05:53.988 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:53 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/3692502949' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:05:54.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:53 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/3692502949' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:05:54.952 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:54 smithi114 ceph-mon[92983]: from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:54.952 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:54 smithi114 ceph-mon[92983]: pgmap v83: 65 pgs: 5 remapped+peering, 17 peering, 43 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-02-19T19:05:54.952 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:54 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:54.952 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:54 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:54.953 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:54 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:54.953 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:54 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:05:54.953 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:54 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:54.953 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:54 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:54.953 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:54 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:54.953 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:54 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:54.953 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:54 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:54.953 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:54 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-19T19:05:55.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:54 smithi174 ceph-mon[87279]: from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:05:55.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:54 smithi174 ceph-mon[87279]: pgmap v83: 65 pgs: 5 remapped+peering, 17 peering, 43 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-02-19T19:05:55.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:54 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:55.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:54 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:55.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:54 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:05:55.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:54 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:05:55.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:54 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:55.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:54 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:05:55.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:54 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:55.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:54 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:55.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:54 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:05:55.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:54 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-19T19:05:55.969 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:55 smithi114 ceph-mon[92983]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-19T19:05:55.969 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:55 smithi114 ceph-mon[92983]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-19T19:05:55.969 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:55 smithi114 ceph-mon[92983]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/129 objects degraded (5.426%), 5 pgs degraded) 2024-02-19T19:05:55.969 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:55 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:55.969 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:55 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:05:56.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:55 smithi174 ceph-mon[87279]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-19T19:05:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:55 smithi174 ceph-mon[87279]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-19T19:05:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:55 smithi174 ceph-mon[87279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/129 objects degraded (5.426%), 5 pgs degraded) 2024-02-19T19:05:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:55 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:05:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:55 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:05:57.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:56 smithi114 ceph-mon[92983]: pgmap v84: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s rd, 5 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-02-19T19:05:57.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:56 smithi174 ceph-mon[87279]: pgmap v84: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s rd, 5 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-02-19T19:05:58.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:57 smithi174 ceph-mon[87279]: pgmap v85: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-02-19T19:05:58.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:57 smithi114 ceph-mon[92983]: pgmap v85: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-02-19T19:06:00.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:05:59 smithi174 ceph-mon[87279]: pgmap v86: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-02-19T19:06:00.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:05:59 smithi114 ceph-mon[92983]: pgmap v86: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-02-19T19:06:02.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:01 smithi174 ceph-mon[87279]: pgmap v87: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-19T19:06:02.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:01 smithi114 ceph-mon[92983]: pgmap v87: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-19T19:06:04.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:03 smithi114 ceph-mon[92983]: pgmap v88: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-19T19:06:04.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:03 smithi174 ceph-mon[87279]: pgmap v88: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-19T19:06:06.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:05 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:06.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:05 smithi174 ceph-mon[87279]: pgmap v89: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T19:06:06.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:05 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:06.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:05 smithi114 ceph-mon[92983]: pgmap v89: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T19:06:07.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:07 smithi174 ceph-mon[87279]: pgmap v90: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:06:08.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:07 smithi114 ceph-mon[92983]: pgmap v90: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:06:10.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:09 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-19T19:06:10.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:09 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-19T19:06:10.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:10 smithi114 ceph-mon[92983]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-19T19:06:10.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:10 smithi114 ceph-mon[92983]: Upgrade: osd.2 is safe to restart 2024-02-19T19:06:10.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:10 smithi114 ceph-mon[92983]: pgmap v91: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:06:10.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:10 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:06:11.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:10 smithi174 ceph-mon[87279]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-19T19:06:11.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:10 smithi174 ceph-mon[87279]: Upgrade: osd.2 is safe to restart 2024-02-19T19:06:11.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:10 smithi174 ceph-mon[87279]: pgmap v91: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:06:11.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:10 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:06:12.498 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:12 smithi114 ceph-mon[92983]: Upgrade: Updating osd.2 2024-02-19T19:06:12.498 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:12.498 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-19T19:06:12.498 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:12.498 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:12 smithi114 ceph-mon[92983]: Deploying daemon osd.2 on smithi114 2024-02-19T19:06:12.498 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:12 smithi114 ceph-mon[92983]: pgmap v92: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:06:12.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:12 smithi174 ceph-mon[87279]: Upgrade: Updating osd.2 2024-02-19T19:06:12.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:12.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-19T19:06:12.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:12.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:12 smithi174 ceph-mon[87279]: Deploying daemon osd.2 on smithi114 2024-02-19T19:06:12.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:12 smithi174 ceph-mon[87279]: pgmap v92: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:06:14.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:13 smithi174 ceph-mon[87279]: pgmap v93: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:06:14.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:13 smithi114 ceph-mon[92983]: pgmap v93: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:06:15.039 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:14 smithi114 systemd[1]: Stopping Ceph osd.2 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T19:06:15.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:15 smithi114 ceph-mon[92983]: osd.2 marked itself down and dead 2024-02-19T19:06:15.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:15 smithi114 ceph-mon[92983]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:06:15.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[63366]: 2024-02-19T19:06:15.168+0000 7f13e6368700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T19:06:15.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[63366]: 2024-02-19T19:06:15.168+0000 7f13e6368700 -1 osd.2 63 *** Got signal Terminated *** 2024-02-19T19:06:15.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[63366]: 2024-02-19T19:06:15.168+0000 7f13e6368700 -1 osd.2 63 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T19:06:15.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:15 smithi174 ceph-mon[87279]: osd.2 marked itself down and dead 2024-02-19T19:06:15.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:15 smithi174 ceph-mon[87279]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:06:15.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:15 smithi114 podman[112688]: 2024-02-19 19:06:15.567606094 +0000 UTC m=+0.521074245 container died 12953988af86f6d567dedeaa4246fb05e2c2da04720b056e1125790f9926c559 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, RELEASE=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-02-19T19:06:16.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:16 smithi114 podman[112688]: 2024-02-19 19:06:16.018901728 +0000 UTC m=+0.972369878 container cleanup 12953988af86f6d567dedeaa4246fb05e2c2da04720b056e1125790f9926c559 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-02-19T19:06:16.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:16 smithi114 bash[112688]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2 2024-02-19T19:06:16.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:16 smithi114 ceph-mon[92983]: osdmap e64: 6 total, 5 up, 6 in 2024-02-19T19:06:16.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:16 smithi114 ceph-mon[92983]: pgmap v95: 65 pgs: 13 peering, 8 stale+active+clean, 44 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-02-19T19:06:16.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:16 smithi114 ceph-mon[92983]: osdmap e65: 6 total, 5 up, 6 in 2024-02-19T19:06:16.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:16 smithi114 podman[112706]: 2024-02-19 19:06:16.329433058 +0000 UTC m=+0.761404721 container remove 12953988af86f6d567dedeaa4246fb05e2c2da04720b056e1125790f9926c559 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:06:16.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:16 smithi174 ceph-mon[87279]: osdmap e64: 6 total, 5 up, 6 in 2024-02-19T19:06:16.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:16 smithi174 ceph-mon[87279]: pgmap v95: 65 pgs: 13 peering, 8 stale+active+clean, 44 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-02-19T19:06:16.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:16 smithi174 ceph-mon[87279]: osdmap e65: 6 total, 5 up, 6 in 2024-02-19T19:06:16.944 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:16 smithi114 podman[112797]: 2024-02-19 19:06:16.572986178 +0000 UTC m=+0.018139653 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:06:16.944 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:16 smithi114 podman[112797]: 2024-02-19 19:06:16.688770408 +0000 UTC m=+0.133923872 container create 1a1262fe52d264482655428d2098087f06ab70ee4aafcfaf276e23690802a29d (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240214) 2024-02-19T19:06:16.944 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:16 smithi114 podman[112797]: 2024-02-19 19:06:16.924785606 +0000 UTC m=+0.369939061 container init 1a1262fe52d264482655428d2098087f06ab70ee4aafcfaf276e23690802a29d (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-19T19:06:17.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:16 smithi114 podman[112797]: 2024-02-19 19:06:16.927942015 +0000 UTC m=+0.373095478 container start 1a1262fe52d264482655428d2098087f06ab70ee4aafcfaf276e23690802a29d (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240214) 2024-02-19T19:06:17.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:17 smithi114 podman[112797]: 2024-02-19 19:06:17.041453888 +0000 UTC m=+0.486607344 container attach 1a1262fe52d264482655428d2098087f06ab70ee4aafcfaf276e23690802a29d (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240214, GIT_CLEAN=True) 2024-02-19T19:06:17.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:17 smithi114 ceph-mon[92983]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-19T19:06:17.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:17 smithi114 podman[112797]: 2024-02-19 19:06:17.221019229 +0000 UTC m=+0.666172685 container died 1a1262fe52d264482655428d2098087f06ab70ee4aafcfaf276e23690802a29d (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:06:17.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:17 smithi174 ceph-mon[87279]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-19T19:06:18.039 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:17 smithi114 podman[112823]: 2024-02-19 19:06:17.854381476 +0000 UTC m=+0.627668650 container remove 1a1262fe52d264482655428d2098087f06ab70ee4aafcfaf276e23690802a29d (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-02-19T19:06:18.039 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:17 smithi114 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-19T19:06:18.039 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:17 smithi114 systemd[1]: Stopped Ceph osd.2 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T19:06:18.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:17 smithi114 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.2.service: Consumed 5.078s CPU time. 2024-02-19T19:06:18.435 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:18 smithi114 ceph-mon[92983]: pgmap v97: 65 pgs: 1 active+undersized+degraded, 13 peering, 5 stale+active+clean, 6 active+undersized, 40 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s; 1/129 objects degraded (0.775%) 2024-02-19T19:06:18.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:18 smithi114 systemd[1]: Starting Ceph osd.2 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T19:06:18.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:18 smithi114 podman[112930]: 2024-02-19 19:06:18.201489841 +0000 UTC m=+0.018729049 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:06:18.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:18 smithi114 podman[112930]: 2024-02-19 19:06:18.305473313 +0000 UTC m=+0.122712580 container create 75b4d4c14167937408a935ec7c820d297dec7e166c68a94dae37ba547573460c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux ) 2024-02-19T19:06:18.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:18 smithi174 ceph-mon[87279]: pgmap v97: 65 pgs: 1 active+undersized+degraded, 13 peering, 5 stale+active+clean, 6 active+undersized, 40 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s; 1/129 objects degraded (0.775%) 2024-02-19T19:06:18.789 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:18 smithi114 podman[112930]: 2024-02-19 19:06:18.558913474 +0000 UTC m=+0.376152678 container init 75b4d4c14167937408a935ec7c820d297dec7e166c68a94dae37ba547573460c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.license=GPLv2) 2024-02-19T19:06:18.789 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:18 smithi114 podman[112930]: 2024-02-19 19:06:18.562165501 +0000 UTC m=+0.379404707 container start 75b4d4c14167937408a935ec7c820d297dec7e166c68a94dae37ba547573460c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-activate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-02-19T19:06:18.789 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:18 smithi114 podman[112930]: 2024-02-19 19:06:18.641375754 +0000 UTC m=+0.458614968 container attach 75b4d4c14167937408a935ec7c820d297dec7e166c68a94dae37ba547573460c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-activate, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0) 2024-02-19T19:06:19.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:19 smithi114 ceph-mon[92983]: Health check failed: Degraded data redundancy: 1/129 objects degraded (0.775%), 1 pg degraded (PG_DEGRADED) 2024-02-19T19:06:19.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-activate[112946]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-19T19:06:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-activate[112946]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-19T19:06:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 bash[112930]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-19T19:06:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 bash[112930]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-19T19:06:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 bash[112930]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-19T19:06:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 bash[112930]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-19T19:06:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 bash[112930]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-19T19:06:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 bash[112930]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-19T19:06:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 bash[112930]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-19T19:06:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-activate[112946]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-19T19:06:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-activate[112946]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-19T19:06:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-activate[112946]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-19T19:06:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-activate[112946]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-19T19:06:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-activate[112946]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-19T19:06:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 podman[112930]: 2024-02-19 19:06:19.032401745 +0000 UTC m=+0.849640938 container died 75b4d4c14167937408a935ec7c820d297dec7e166c68a94dae37ba547573460c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-02-19T19:06:19.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:19 smithi174 ceph-mon[87279]: Health check failed: Degraded data redundancy: 1/129 objects degraded (0.775%), 1 pg degraded (PG_DEGRADED) 2024-02-19T19:06:20.039 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 podman[113071]: 2024-02-19 19:06:19.720956688 +0000 UTC m=+0.678822842 container remove 75b4d4c14167937408a935ec7c820d297dec7e166c68a94dae37ba547573460c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-activate, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS) 2024-02-19T19:06:20.039 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 podman[113135]: 2024-02-19 19:06:19.842236657 +0000 UTC m=+0.018693488 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:06:20.039 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:19 smithi114 podman[113135]: 2024-02-19 19:06:19.955156828 +0000 UTC m=+0.131613658 container create 11e64d20e378e5796713cb722c7d426ecb96ab922eda5782300cb59c884e2d6e (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2, org.label-schema.build-date=20240214, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-02-19T19:06:20.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:20 smithi114 ceph-mon[92983]: pgmap v98: 65 pgs: 5 active+undersized+degraded, 13 peering, 11 active+undersized, 36 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 11/129 objects degraded (8.527%) 2024-02-19T19:06:20.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:20 smithi114 podman[113135]: 2024-02-19 19:06:20.19921643 +0000 UTC m=+0.375673260 container init 11e64d20e378e5796713cb722c7d426ecb96ab922eda5782300cb59c884e2d6e (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD) 2024-02-19T19:06:20.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:20 smithi114 podman[113135]: 2024-02-19 19:06:20.202386128 +0000 UTC m=+0.378842959 container start 11e64d20e378e5796713cb722c7d426ecb96ab922eda5782300cb59c884e2d6e (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-19T19:06:20.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:20 smithi114 ceph-osd[113154]: -- 172.21.15.114:0/198157251 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563ff192d1e0 con 0x563ff18fec00 2024-02-19T19:06:20.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:20 smithi114 bash[113135]: 11e64d20e378e5796713cb722c7d426ecb96ab922eda5782300cb59c884e2d6e 2024-02-19T19:06:20.540 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:20 smithi114 systemd[1]: Started Ceph osd.2 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T19:06:20.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:20 smithi174 ceph-mon[87279]: pgmap v98: 65 pgs: 5 active+undersized+degraded, 13 peering, 11 active+undersized, 36 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 11/129 objects degraded (8.527%) 2024-02-19T19:06:21.039 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:06:20.749+0000 7f034b5d77c0 -1 Falling back to public interface 2024-02-19T19:06:21.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:21 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:21.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:21 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:21.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:21 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:06:21.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:21 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:21.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:21 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:21.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:21 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:06:22.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:22 smithi174 ceph-mon[87279]: pgmap v99: 65 pgs: 10 active+undersized+degraded, 19 active+undersized, 36 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 19/129 objects degraded (14.729%) 2024-02-19T19:06:22.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:22 smithi114 ceph-mon[92983]: pgmap v99: 65 pgs: 10 active+undersized+degraded, 19 active+undersized, 36 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 19/129 objects degraded (14.729%) 2024-02-19T19:06:23.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.340+0000 7faa47689700 1 -- 172.21.15.114:0/2257889042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa40101640 msgr2=0x7faa40103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:23.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.340+0000 7faa47689700 1 --2- 172.21.15.114:0/2257889042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa40101640 0x7faa40103a30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009910 tx=0x7faa3c009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:23.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.340+0000 7faa47689700 1 -- 172.21.15.114:0/2257889042 shutdown_connections 2024-02-19T19:06:23.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.340+0000 7faa47689700 1 --2- 172.21.15.114:0/2257889042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa40103f70 0x7faa40106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:23.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.340+0000 7faa47689700 1 --2- 172.21.15.114:0/2257889042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa40101640 0x7faa40103a30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:23.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.340+0000 7faa47689700 1 -- 172.21.15.114:0/2257889042 >> 172.21.15.114:0/2257889042 conn(0x7faa400fb030 msgr2=0x7faa400fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:23.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.340+0000 7faa47689700 1 -- 172.21.15.114:0/2257889042 shutdown_connections 2024-02-19T19:06:23.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.340+0000 7faa47689700 1 -- 172.21.15.114:0/2257889042 wait complete. 2024-02-19T19:06:23.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.340+0000 7faa47689700 1 Processor -- start 2024-02-19T19:06:23.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.341+0000 7faa47689700 1 -- start start 2024-02-19T19:06:23.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.341+0000 7faa47689700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa40101640 0x7faa40196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:23.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.341+0000 7faa47689700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa40103f70 0x7faa401968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:23.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.341+0000 7faa47689700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa40196ee0 con 0x7faa40101640 2024-02-19T19:06:23.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.341+0000 7faa47689700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa40197020 con 0x7faa40103f70 2024-02-19T19:06:23.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.341+0000 7faa45425700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa40101640 0x7faa40196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.341+0000 7faa45425700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa40101640 0x7faa40196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35746/0 (socket says 172.21.15.114:35746) 2024-02-19T19:06:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.341+0000 7faa45425700 1 -- 172.21.15.114:0/1036312047 learned_addr learned my addr 172.21.15.114:0/1036312047 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:06:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.341+0000 7faa45425700 1 -- 172.21.15.114:0/1036312047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa40103f70 msgr2=0x7faa401968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.341+0000 7faa45425700 1 --2- 172.21.15.114:0/1036312047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa40103f70 0x7faa401968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.341+0000 7faa45425700 1 -- 172.21.15.114:0/1036312047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3c0095f0 con 0x7faa40101640 2024-02-19T19:06:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.341+0000 7faa45425700 1 --2- 172.21.15.114:0/1036312047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa40101640 0x7faa40196380 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faa3c006010 tx=0x7faa3c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.341+0000 7faa367fc700 1 -- 172.21.15.114:0/1036312047 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa3c01d070 con 0x7faa40101640 2024-02-19T19:06:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.341+0000 7faa367fc700 1 -- 172.21.15.114:0/1036312047 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faa3c00f460 con 0x7faa40101640 2024-02-19T19:06:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.341+0000 7faa47689700 1 -- 172.21.15.114:0/1036312047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa4019ba70 con 0x7faa40101640 2024-02-19T19:06:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.341+0000 7faa367fc700 1 -- 172.21.15.114:0/1036312047 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa3c021640 con 0x7faa40101640 2024-02-19T19:06:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.342+0000 7faa47689700 1 -- 172.21.15.114:0/1036312047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa4019bfe0 con 0x7faa40101640 2024-02-19T19:06:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.343+0000 7faa47689700 1 -- 172.21.15.114:0/1036312047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa401905a0 con 0x7faa40101640 2024-02-19T19:06:23.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.343+0000 7faa367fc700 1 -- 172.21.15.114:0/1036312047 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7faa3c00fab0 con 0x7faa40101640 2024-02-19T19:06:23.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.343+0000 7faa367fc700 1 --2- 172.21.15.114:0/1036312047 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7faa2c06d8a0 0x7faa2c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:23.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.344+0000 7faa367fc700 1 -- 172.21.15.114:0/1036312047 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6790+0+0 (secure 0 0 0) 0x7faa3c08ead0 con 0x7faa40101640 2024-02-19T19:06:23.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.344+0000 7faa44c24700 1 --2- 172.21.15.114:0/1036312047 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7faa2c06d8a0 0x7faa2c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:23.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.344+0000 7faa44c24700 1 --2- 172.21.15.114:0/1036312047 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7faa2c06d8a0 0x7faa2c06fd60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faa300097b0 tx=0x7faa30006d20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:23.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.348+0000 7faa367fc700 1 -- 172.21.15.114:0/1036312047 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faa3c05bce0 con 0x7faa40101640 2024-02-19T19:06:23.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.582+0000 7faa47689700 1 -- 172.21.15.114:0/1036312047 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faa400611d0 con 0x7faa2c06d8a0 2024-02-19T19:06:23.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.584+0000 7faa367fc700 1 -- 172.21.15.114:0/1036312047 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7faa400611d0 con 0x7faa2c06d8a0 2024-02-19T19:06:23.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.586+0000 7faa47689700 1 -- 172.21.15.114:0/1036312047 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7faa2c06d8a0 msgr2=0x7faa2c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:23.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.586+0000 7faa47689700 1 --2- 172.21.15.114:0/1036312047 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7faa2c06d8a0 0x7faa2c06fd60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faa300097b0 tx=0x7faa30006d20 comp rx=0 tx=0).stop 2024-02-19T19:06:23.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.586+0000 7faa47689700 1 -- 172.21.15.114:0/1036312047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa40101640 msgr2=0x7faa40196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:23.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.586+0000 7faa47689700 1 --2- 172.21.15.114:0/1036312047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa40101640 0x7faa40196380 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faa3c006010 tx=0x7faa3c00be20 comp rx=0 tx=0).stop 2024-02-19T19:06:23.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.586+0000 7faa47689700 1 -- 172.21.15.114:0/1036312047 shutdown_connections 2024-02-19T19:06:23.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.586+0000 7faa47689700 1 --2- 172.21.15.114:0/1036312047 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7faa2c06d8a0 0x7faa2c06fd60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:23.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.586+0000 7faa47689700 1 --2- 172.21.15.114:0/1036312047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa40101640 0x7faa40196380 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:23.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.586+0000 7faa47689700 1 --2- 172.21.15.114:0/1036312047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa40103f70 0x7faa401968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:23.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.586+0000 7faa47689700 1 -- 172.21.15.114:0/1036312047 >> 172.21.15.114:0/1036312047 conn(0x7faa400fb030 msgr2=0x7faa400fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:23.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.586+0000 7faa47689700 1 -- 172.21.15.114:0/1036312047 shutdown_connections 2024-02-19T19:06:23.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.586+0000 7faa47689700 1 -- 172.21.15.114:0/1036312047 wait complete. 2024-02-19T19:06:23.617 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-02-19T19:06:23.617 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:23 smithi114 ceph-mon[92983]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-19T19:06:23.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:23 smithi174 ceph-mon[87279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-19T19:06:23.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.692+0000 7f9fa0a1d700 1 -- 172.21.15.114:0/307219146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f9c0ffab0 msgr2=0x7f9f9c0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:23.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.692+0000 7f9fa0a1d700 1 --2- 172.21.15.114:0/307219146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f9c0ffab0 0x7f9f9c0ffed0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009910 tx=0x7f9f8c009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:23.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.692+0000 7f9fa0a1d700 1 -- 172.21.15.114:0/307219146 shutdown_connections 2024-02-19T19:06:23.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.692+0000 7f9fa0a1d700 1 --2- 172.21.15.114:0/307219146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f9c100cb0 0x7f9f9c101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:23.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.692+0000 7f9fa0a1d700 1 --2- 172.21.15.114:0/307219146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f9c0ffab0 0x7f9f9c0ffed0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:23.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.692+0000 7f9fa0a1d700 1 -- 172.21.15.114:0/307219146 >> 172.21.15.114:0/307219146 conn(0x7f9f9c0fb010 msgr2=0x7f9f9c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:23.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.692+0000 7f9fa0a1d700 1 -- 172.21.15.114:0/307219146 shutdown_connections 2024-02-19T19:06:23.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.692+0000 7f9fa0a1d700 1 -- 172.21.15.114:0/307219146 wait complete. 2024-02-19T19:06:23.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.693+0000 7f9fa0a1d700 1 Processor -- start 2024-02-19T19:06:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.693+0000 7f9fa0a1d700 1 -- start start 2024-02-19T19:06:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.693+0000 7f9fa0a1d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f9c0ffab0 0x7f9f9c1953a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.694+0000 7f9fa0a1d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f9c100cb0 0x7f9f9c1958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.694+0000 7f9fa0a1d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c195f00 con 0x7f9f9c100cb0 2024-02-19T19:06:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.694+0000 7f9fa0a1d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c196040 con 0x7f9f9c0ffab0 2024-02-19T19:06:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.694+0000 7f9f99d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f9c100cb0 0x7f9f9c1958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.694+0000 7f9f99d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f9c100cb0 0x7f9f9c1958e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35750/0 (socket says 172.21.15.114:35750) 2024-02-19T19:06:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.694+0000 7f9f99d9b700 1 -- 172.21.15.114:0/2516515246 learned_addr learned my addr 172.21.15.114:0/2516515246 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:06:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.694+0000 7f9f9a59c700 1 --2- 172.21.15.114:0/2516515246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f9c0ffab0 0x7f9f9c1953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.694+0000 7f9f99d9b700 1 -- 172.21.15.114:0/2516515246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f9c0ffab0 msgr2=0x7f9f9c1953a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.694+0000 7f9f99d9b700 1 --2- 172.21.15.114:0/2516515246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f9c0ffab0 0x7f9f9c1953a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.694+0000 7f9f99d9b700 1 -- 172.21.15.114:0/2516515246 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c0095f0 con 0x7f9f9c100cb0 2024-02-19T19:06:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.694+0000 7f9f99d9b700 1 --2- 172.21.15.114:0/2516515246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f9c100cb0 0x7f9f9c1958e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000d960 tx=0x7f9f9000dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.694+0000 7f9f8b7fe700 1 -- 172.21.15.114:0/2516515246 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f9000ec20 con 0x7f9f9c100cb0 2024-02-19T19:06:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.694+0000 7f9fa0a1d700 1 -- 172.21.15.114:0/2516515246 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9c068e50 con 0x7f9f9c100cb0 2024-02-19T19:06:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.694+0000 7f9f8b7fe700 1 -- 172.21.15.114:0/2516515246 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9f9000ed80 con 0x7f9f9c100cb0 2024-02-19T19:06:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.694+0000 7f9f8b7fe700 1 -- 172.21.15.114:0/2516515246 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f900147b0 con 0x7f9f9c100cb0 2024-02-19T19:06:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.694+0000 7f9fa0a1d700 1 -- 172.21.15.114:0/2516515246 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9c069420 con 0x7f9f9c100cb0 2024-02-19T19:06:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.695+0000 7f9fa0a1d700 1 -- 172.21.15.114:0/2516515246 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f9c108690 con 0x7f9f9c100cb0 2024-02-19T19:06:23.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.696+0000 7f9f8b7fe700 1 -- 172.21.15.114:0/2516515246 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f9f90014950 con 0x7f9f9c100cb0 2024-02-19T19:06:23.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.696+0000 7f9f8b7fe700 1 --2- 172.21.15.114:0/2516515246 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9f8406d8a0 0x7f9f8406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:23.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.696+0000 7f9f8b7fe700 1 -- 172.21.15.114:0/2516515246 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f9f9008efd0 con 0x7f9f9c100cb0 2024-02-19T19:06:23.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.696+0000 7f9f9a59c700 1 --2- 172.21.15.114:0/2516515246 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9f8406d8a0 0x7f9f8406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:23.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.697+0000 7f9f9a59c700 1 --2- 172.21.15.114:0/2516515246 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9f8406d8a0 0x7f9f8406fd60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c006010 tx=0x7f9f8c00b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:23.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.701+0000 7f9f8b7fe700 1 -- 172.21.15.114:0/2516515246 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9f9005c290 con 0x7f9f9c100cb0 2024-02-19T19:06:23.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.917+0000 7f9fa0a1d700 1 -- 172.21.15.114:0/2516515246 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f9c105600 con 0x7f9f8406d8a0 2024-02-19T19:06:23.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.918+0000 7f9f8b7fe700 1 -- 172.21.15.114:0/2516515246 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f9f9c105600 con 0x7f9f8406d8a0 2024-02-19T19:06:23.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.920+0000 7f9fa0a1d700 1 -- 172.21.15.114:0/2516515246 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9f8406d8a0 msgr2=0x7f9f8406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:23.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.920+0000 7f9fa0a1d700 1 --2- 172.21.15.114:0/2516515246 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9f8406d8a0 0x7f9f8406fd60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c006010 tx=0x7f9f8c00b540 comp rx=0 tx=0).stop 2024-02-19T19:06:23.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.921+0000 7f9fa0a1d700 1 -- 172.21.15.114:0/2516515246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f9c100cb0 msgr2=0x7f9f9c1958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:23.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.921+0000 7f9fa0a1d700 1 --2- 172.21.15.114:0/2516515246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f9c100cb0 0x7f9f9c1958e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000d960 tx=0x7f9f9000dd20 comp rx=0 tx=0).stop 2024-02-19T19:06:23.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.921+0000 7f9fa0a1d700 1 -- 172.21.15.114:0/2516515246 shutdown_connections 2024-02-19T19:06:23.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.921+0000 7f9fa0a1d700 1 --2- 172.21.15.114:0/2516515246 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9f8406d8a0 0x7f9f8406fd60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:23.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.921+0000 7f9fa0a1d700 1 --2- 172.21.15.114:0/2516515246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f9c0ffab0 0x7f9f9c1953a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:23.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.921+0000 7f9fa0a1d700 1 --2- 172.21.15.114:0/2516515246 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f9c100cb0 0x7f9f9c1958e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:23.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.921+0000 7f9fa0a1d700 1 -- 172.21.15.114:0/2516515246 >> 172.21.15.114:0/2516515246 conn(0x7f9f9c0fb010 msgr2=0x7f9f9c103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:23.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.921+0000 7f9fa0a1d700 1 -- 172.21.15.114:0/2516515246 shutdown_connections 2024-02-19T19:06:23.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:23.921+0000 7f9fa0a1d700 1 -- 172.21.15.114:0/2516515246 wait complete. 2024-02-19T19:06:24.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.034+0000 7f881636a700 1 -- 172.21.15.114:0/2349370207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88101021a0 msgr2=0x7f88101025c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:24.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.034+0000 7f881636a700 1 --2- 172.21.15.114:0/2349370207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88101021a0 0x7f88101025c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8800009910 tx=0x7f8800009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:24.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.034+0000 7f881636a700 1 -- 172.21.15.114:0/2349370207 shutdown_connections 2024-02-19T19:06:24.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.034+0000 7f881636a700 1 --2- 172.21.15.114:0/2349370207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88100fec20 0x7f88100ff080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:24.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.034+0000 7f881636a700 1 --2- 172.21.15.114:0/2349370207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88101021a0 0x7f88101025c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:24.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.034+0000 7f881636a700 1 -- 172.21.15.114:0/2349370207 >> 172.21.15.114:0/2349370207 conn(0x7f88100fa7c0 msgr2=0x7f88100fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:24.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.034+0000 7f881636a700 1 -- 172.21.15.114:0/2349370207 shutdown_connections 2024-02-19T19:06:24.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.034+0000 7f881636a700 1 -- 172.21.15.114:0/2349370207 wait complete. 2024-02-19T19:06:24.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.035+0000 7f881636a700 1 Processor -- start 2024-02-19T19:06:24.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.035+0000 7f881636a700 1 -- start start 2024-02-19T19:06:24.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.035+0000 7f881636a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88100fec20 0x7f88101963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:24.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.035+0000 7f881636a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88101021a0 0x7f8810196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:24.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.035+0000 7f881636a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8810196f50 con 0x7f88101021a0 2024-02-19T19:06:24.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.035+0000 7f881636a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8810197090 con 0x7f88100fec20 2024-02-19T19:06:24.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.035+0000 7f880f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88101021a0 0x7f8810196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:24.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.035+0000 7f880f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88101021a0 0x7f8810196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35768/0 (socket says 172.21.15.114:35768) 2024-02-19T19:06:24.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.036+0000 7f880f7fe700 1 -- 172.21.15.114:0/1893150887 learned_addr learned my addr 172.21.15.114:0/1893150887 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:06:24.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.036+0000 7f880ffff700 1 --2- 172.21.15.114:0/1893150887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88100fec20 0x7f88101963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:24.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.036+0000 7f880f7fe700 1 -- 172.21.15.114:0/1893150887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88100fec20 msgr2=0x7f88101963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:24.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.036+0000 7f880f7fe700 1 --2- 172.21.15.114:0/1893150887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88100fec20 0x7f88101963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:24.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.036+0000 7f880f7fe700 1 -- 172.21.15.114:0/1893150887 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88000095f0 con 0x7f88101021a0 2024-02-19T19:06:24.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.036+0000 7f880f7fe700 1 --2- 172.21.15.114:0/1893150887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88101021a0 0x7f8810196930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f880400d8d0 tx=0x7f880400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:24.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.036+0000 7f880d7fa700 1 -- 172.21.15.114:0/1893150887 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8804009940 con 0x7f88101021a0 2024-02-19T19:06:24.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.036+0000 7f881636a700 1 -- 172.21.15.114:0/1893150887 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f881019bb40 con 0x7f88101021a0 2024-02-19T19:06:24.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.036+0000 7f881636a700 1 -- 172.21.15.114:0/1893150887 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f881019c110 con 0x7f88101021a0 2024-02-19T19:06:24.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.036+0000 7f880d7fa700 1 -- 172.21.15.114:0/1893150887 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8804010460 con 0x7f88101021a0 2024-02-19T19:06:24.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.037+0000 7f880d7fa700 1 -- 172.21.15.114:0/1893150887 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f880400f5f0 con 0x7f88101021a0 2024-02-19T19:06:24.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.037+0000 7f881636a700 1 -- 172.21.15.114:0/1893150887 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87fc005320 con 0x7f88101021a0 2024-02-19T19:06:24.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.039+0000 7f880d7fa700 1 -- 172.21.15.114:0/1893150887 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f8804009aa0 con 0x7f88101021a0 2024-02-19T19:06:24.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.039+0000 7f880d7fa700 1 --2- 172.21.15.114:0/1893150887 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f87f806d690 0x7f87f806fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:24.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.039+0000 7f880d7fa700 1 -- 172.21.15.114:0/1893150887 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f880408d4e0 con 0x7f88101021a0 2024-02-19T19:06:24.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.039+0000 7f880ffff700 1 --2- 172.21.15.114:0/1893150887 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f87f806d690 0x7f87f806fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:24.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.040+0000 7f880ffff700 1 --2- 172.21.15.114:0/1893150887 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f87f806d690 0x7f87f806fb50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8800006010 tx=0x7f880000b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:24.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.043+0000 7f880d7fa700 1 -- 172.21.15.114:0/1893150887 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f880405a630 con 0x7f88101021a0 2024-02-19T19:06:24.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.263+0000 7f881636a700 1 -- 172.21.15.114:0/1893150887 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f87fc000bf0 con 0x7f87f806d690 2024-02-19T19:06:24.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.275+0000 7f880d7fa700 1 -- 172.21.15.114:0/1893150887 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f87fc000bf0 con 0x7f87f806d690 2024-02-19T19:06:24.296 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:06:24.296 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (10m) 34s ago 13m 26.4M - 0.25.0 c8568f914cd2 30444b676ca4 2024-02-19T19:06:24.296 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (13m) 34s ago 13m 8212k - 18.2.1 94eadd990e0a 4918cd87e312 2024-02-19T19:06:24.296 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi174 smithi174 running (7m) 95s ago 11m 7847k - 18.2.1 94eadd990e0a 516c34431b93 2024-02-19T19:06:24.296 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (104s) 34s ago 13m 6660k - 19.0.0-1450-gf78a58c0 f081f847070a f10b46e9d636 2024-02-19T19:06:24.296 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi174 smithi174 running (99s) 95s ago 11m 6660k - 19.0.0-1450-gf78a58c0 f081f847070a fb3b17c81bd8 2024-02-19T19:06:24.296 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (10m) 34s ago 12m 85.6M - 9.4.7 954c08fa6188 478bd533e827 2024-02-19T19:06:24.296 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tcwpod smithi114 running (6m) 34s ago 6m 27.5M - 18.2.1 94eadd990e0a 420175777cf4 2024-02-19T19:06:24.297 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.vpoeop smithi114 running (6m) 34s ago 6m 24.5M - 18.2.1 94eadd990e0a f81a79e53ead 2024-02-19T19:06:24.297 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.stnfnf smithi174 running (6m) 95s ago 6m 26.0M - 18.2.1 94eadd990e0a 3df35954a492 2024-02-19T19:06:24.297 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.znhuvd smithi174 running (6m) 95s ago 6m 24.2M - 18.2.1 94eadd990e0a 2d2267494ae1 2024-02-19T19:06:24.297 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.pcvtrk smithi114 *:8443,9283,8765 running (3m) 34s ago 14m 502M - 19.0.0-1450-gf78a58c0 f081f847070a 05b5c06122b1 2024-02-19T19:06:24.297 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi174.lydvnj smithi174 *:8443,9283,8765 running (3m) 95s ago 10m 450M - 19.0.0-1450-gf78a58c0 f081f847070a f962b1af5aed 2024-02-19T19:06:24.297 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (2m) 34s ago 14m 46.8M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e51f650f9be9 2024-02-19T19:06:24.297 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi174 smithi174 running (2m) 95s ago 10m 33.9M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e811581f05fb 2024-02-19T19:06:24.297 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (13m) 34s ago 13m 14.6M - 1.5.0 0da6a335fe13 b6b258074cce 2024-02-19T19:06:24.297 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi174 smithi174 *:9100 running (10m) 95s ago 10m 14.9M - 1.5.0 0da6a335fe13 d0b4fdf67b22 2024-02-19T19:06:24.297 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (77s) 34s ago 10m 51.1M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 764769439e0e 2024-02-19T19:06:24.297 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (40s) 34s ago 9m 12.7M 1216M 19.0.0-1450-gf78a58c0 f081f847070a c1c32dbdf573 2024-02-19T19:06:24.297 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 starting - - - 1216M 2024-02-19T19:06:24.297 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi174 running (8m) 95s ago 8m 49.2M 2240M 18.2.1 94eadd990e0a 979f45cabbce 2024-02-19T19:06:24.297 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi174 running (8m) 95s ago 8m 49.7M 2240M 18.2.1 94eadd990e0a 8b8c3745b8c5 2024-02-19T19:06:24.298 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi174 running (7m) 95s ago 7m 52.2M 2240M 18.2.1 94eadd990e0a 5771a995ed75 2024-02-19T19:06:24.298 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (3m) 34s ago 11m 63.1M - 2.43.0 a07b618ecd1d a36922c7c9a1 2024-02-19T19:06:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.277+0000 7f881636a700 1 -- 172.21.15.114:0/1893150887 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f87f806d690 msgr2=0x7f87f806fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.277+0000 7f881636a700 1 --2- 172.21.15.114:0/1893150887 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f87f806d690 0x7f87f806fb50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8800006010 tx=0x7f880000b540 comp rx=0 tx=0).stop 2024-02-19T19:06:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.277+0000 7f881636a700 1 -- 172.21.15.114:0/1893150887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88101021a0 msgr2=0x7f8810196930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.277+0000 7f881636a700 1 --2- 172.21.15.114:0/1893150887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88101021a0 0x7f8810196930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f880400d8d0 tx=0x7f880400dc90 comp rx=0 tx=0).stop 2024-02-19T19:06:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.278+0000 7f881636a700 1 -- 172.21.15.114:0/1893150887 shutdown_connections 2024-02-19T19:06:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.278+0000 7f881636a700 1 --2- 172.21.15.114:0/1893150887 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f87f806d690 0x7f87f806fb50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.278+0000 7f881636a700 1 --2- 172.21.15.114:0/1893150887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88100fec20 0x7f88101963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.278+0000 7f881636a700 1 --2- 172.21.15.114:0/1893150887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88101021a0 0x7f8810196930 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.278+0000 7f881636a700 1 -- 172.21.15.114:0/1893150887 >> 172.21.15.114:0/1893150887 conn(0x7f88100fa7c0 msgr2=0x7f8810105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.278+0000 7f881636a700 1 -- 172.21.15.114:0/1893150887 shutdown_connections 2024-02-19T19:06:24.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.278+0000 7f881636a700 1 -- 172.21.15.114:0/1893150887 wait complete. 2024-02-19T19:06:24.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.391+0000 7f446565a700 1 -- 172.21.15.114:0/2644542307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44600ffd90 msgr2=0x7f44601001f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:24.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.391+0000 7f446565a700 1 --2- 172.21.15.114:0/2644542307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44600ffd90 0x7f44601001f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4454009910 tx=0x7f4454009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:24.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.391+0000 7f446565a700 1 -- 172.21.15.114:0/2644542307 shutdown_connections 2024-02-19T19:06:24.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.391+0000 7f446565a700 1 --2- 172.21.15.114:0/2644542307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44600ffd90 0x7f44601001f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:24.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.391+0000 7f446565a700 1 --2- 172.21.15.114:0/2644542307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4460101050 0x7f4460101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:24.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.391+0000 7f446565a700 1 -- 172.21.15.114:0/2644542307 >> 172.21.15.114:0/2644542307 conn(0x7f44600fa7c0 msgr2=0x7f44600fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:24.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.391+0000 7f446565a700 1 -- 172.21.15.114:0/2644542307 shutdown_connections 2024-02-19T19:06:24.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.391+0000 7f446565a700 1 -- 172.21.15.114:0/2644542307 wait complete. 2024-02-19T19:06:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.392+0000 7f446565a700 1 Processor -- start 2024-02-19T19:06:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.392+0000 7f446565a700 1 -- start start 2024-02-19T19:06:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.392+0000 7f446565a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44600ffd90 0x7f4460195280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.392+0000 7f446565a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460101050 0x7f44601957c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.392+0000 7f446565a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4460195de0 con 0x7f4460101050 2024-02-19T19:06:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.392+0000 7f446565a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4460195f20 con 0x7f44600ffd90 2024-02-19T19:06:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.393+0000 7f445e7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460101050 0x7f44601957c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.393+0000 7f445e7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460101050 0x7f44601957c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35800/0 (socket says 172.21.15.114:35800) 2024-02-19T19:06:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.393+0000 7f445e7fc700 1 -- 172.21.15.114:0/1395637209 learned_addr learned my addr 172.21.15.114:0/1395637209 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:06:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.393+0000 7f445effd700 1 --2- 172.21.15.114:0/1395637209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44600ffd90 0x7f4460195280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.393+0000 7f445e7fc700 1 -- 172.21.15.114:0/1395637209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44600ffd90 msgr2=0x7f4460195280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.393+0000 7f445e7fc700 1 --2- 172.21.15.114:0/1395637209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44600ffd90 0x7f4460195280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.393+0000 7f445e7fc700 1 -- 172.21.15.114:0/1395637209 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44540095f0 con 0x7f4460101050 2024-02-19T19:06:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.393+0000 7f445e7fc700 1 --2- 172.21.15.114:0/1395637209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460101050 0x7f44601957c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4454006010 tx=0x7f445400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.393+0000 7f4447fff700 1 -- 172.21.15.114:0/1395637209 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f445401d070 con 0x7f4460101050 2024-02-19T19:06:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.393+0000 7f4447fff700 1 -- 172.21.15.114:0/1395637209 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f445400f460 con 0x7f4460101050 2024-02-19T19:06:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.393+0000 7f446565a700 1 -- 172.21.15.114:0/1395637209 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44600fedc0 con 0x7f4460101050 2024-02-19T19:06:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.393+0000 7f4447fff700 1 -- 172.21.15.114:0/1395637209 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4454021640 con 0x7f4460101050 2024-02-19T19:06:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.394+0000 7f446565a700 1 -- 172.21.15.114:0/1395637209 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44600ff260 con 0x7f4460101050 2024-02-19T19:06:24.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.395+0000 7f446565a700 1 -- 172.21.15.114:0/1395637209 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4460108630 con 0x7f4460101050 2024-02-19T19:06:24.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.395+0000 7f4447fff700 1 -- 172.21.15.114:0/1395637209 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f445402b440 con 0x7f4460101050 2024-02-19T19:06:24.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.396+0000 7f4447fff700 1 --2- 172.21.15.114:0/1395637209 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f444806d8a0 0x7f444806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:24.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.396+0000 7f4447fff700 1 -- 172.21.15.114:0/1395637209 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f445408e8c0 con 0x7f4460101050 2024-02-19T19:06:24.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.396+0000 7f445effd700 1 --2- 172.21.15.114:0/1395637209 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f444806d8a0 0x7f444806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:24.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.396+0000 7f445effd700 1 --2- 172.21.15.114:0/1395637209 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f444806d8a0 0x7f444806fd60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4460068190 tx=0x7f445000b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:24.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.401+0000 7f4447fff700 1 -- 172.21.15.114:0/1395637209 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f445405bc20 con 0x7f4460101050 2024-02-19T19:06:24.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:24 smithi114 ceph-mon[92983]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:24.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:24 smithi114 ceph-mon[92983]: pgmap v100: 65 pgs: 10 active+undersized+degraded, 19 active+undersized, 36 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 19/129 objects degraded (14.729%) 2024-02-19T19:06:24.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:24 smithi114 ceph-mon[92983]: from='osd.2 [v2:172.21.15.114:6818/3670286774,v1:172.21.15.114:6819/3670286774]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-19T19:06:24.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:06:24.231+0000 7f034b5d77c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-02-19T19:06:24.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:06:24.264+0000 7f034b5d77c0 -1 osd.2 63 log_to_monitors true 2024-02-19T19:06:24.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:06:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:06:24.380+0000 7f033ca2f700 -1 osd.2 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T19:06:24.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:24 smithi174 ceph-mon[87279]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:24.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:24 smithi174 ceph-mon[87279]: pgmap v100: 65 pgs: 10 active+undersized+degraded, 19 active+undersized, 36 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 19/129 objects degraded (14.729%) 2024-02-19T19:06:24.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:24 smithi174 ceph-mon[87279]: from='osd.2 [v2:172.21.15.114:6818/3670286774,v1:172.21.15.114:6819/3670286774]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-19T19:06:24.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.681+0000 7f446565a700 1 -- 172.21.15.114:0/1395637209 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f44601087f0 con 0x7f4460101050 2024-02-19T19:06:24.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.681+0000 7f4447fff700 1 -- 172.21.15.114:0/1395637209 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f445405b7b0 con 0x7f4460101050 2024-02-19T19:06:24.702 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:06:24.702 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-02-19T19:06:24.702 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:06:24.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:06:24.702 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-02-19T19:06:24.702 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:06:24.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:06:24.703 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-02-19T19:06:24.703 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3, 2024-02-19T19:06:24.703 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:06:24.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:06:24.703 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-02-19T19:06:24.703 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:06:24.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:06:24.703 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-02-19T19:06:24.703 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 7, 2024-02-19T19:06:24.703 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 6 2024-02-19T19:06:24.703 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-02-19T19:06:24.703 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:06:24.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.683+0000 7f446565a700 1 -- 172.21.15.114:0/1395637209 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f444806d8a0 msgr2=0x7f444806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:24.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.683+0000 7f446565a700 1 --2- 172.21.15.114:0/1395637209 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f444806d8a0 0x7f444806fd60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4460068190 tx=0x7f445000b410 comp rx=0 tx=0).stop 2024-02-19T19:06:24.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.683+0000 7f446565a700 1 -- 172.21.15.114:0/1395637209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460101050 msgr2=0x7f44601957c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:24.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.683+0000 7f446565a700 1 --2- 172.21.15.114:0/1395637209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460101050 0x7f44601957c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4454006010 tx=0x7f445400bd70 comp rx=0 tx=0).stop 2024-02-19T19:06:24.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.683+0000 7f446565a700 1 -- 172.21.15.114:0/1395637209 shutdown_connections 2024-02-19T19:06:24.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.683+0000 7f446565a700 1 --2- 172.21.15.114:0/1395637209 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f444806d8a0 0x7f444806fd60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:24.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.683+0000 7f446565a700 1 --2- 172.21.15.114:0/1395637209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44600ffd90 0x7f4460195280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:24.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.683+0000 7f446565a700 1 --2- 172.21.15.114:0/1395637209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4460101050 0x7f44601957c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:24.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.683+0000 7f446565a700 1 -- 172.21.15.114:0/1395637209 >> 172.21.15.114:0/1395637209 conn(0x7f44600fa7c0 msgr2=0x7f4460103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:24.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.684+0000 7f446565a700 1 -- 172.21.15.114:0/1395637209 shutdown_connections 2024-02-19T19:06:24.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.684+0000 7f446565a700 1 -- 172.21.15.114:0/1395637209 wait complete. 2024-02-19T19:06:24.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.803+0000 7fb579c45700 1 -- 172.21.15.114:0/2727108515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb574074d60 msgr2=0x7fb5740731c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:24.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.803+0000 7fb579c45700 1 --2- 172.21.15.114:0/2727108515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb574074d60 0x7fb5740731c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb564009910 tx=0x7fb564009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:24.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.803+0000 7fb579c45700 1 -- 172.21.15.114:0/2727108515 shutdown_connections 2024-02-19T19:06:24.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.803+0000 7fb579c45700 1 --2- 172.21.15.114:0/2727108515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb574073790 0x7fb574073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:24.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.803+0000 7fb579c45700 1 --2- 172.21.15.114:0/2727108515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb574074d60 0x7fb5740731c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:24.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.803+0000 7fb579c45700 1 -- 172.21.15.114:0/2727108515 >> 172.21.15.114:0/2727108515 conn(0x7fb5740fc110 msgr2=0x7fb5740fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:24.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.803+0000 7fb579c45700 1 -- 172.21.15.114:0/2727108515 shutdown_connections 2024-02-19T19:06:24.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.803+0000 7fb579c45700 1 -- 172.21.15.114:0/2727108515 wait complete. 2024-02-19T19:06:24.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.804+0000 7fb579c45700 1 Processor -- start 2024-02-19T19:06:24.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.804+0000 7fb579c45700 1 -- start start 2024-02-19T19:06:24.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.804+0000 7fb579c45700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb574073790 0x7fb5741985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:24.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.804+0000 7fb579c45700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb574074d60 0x7fb574198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:24.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.804+0000 7fb579c45700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb574199130 con 0x7fb574073790 2024-02-19T19:06:24.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.804+0000 7fb579c45700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb574199270 con 0x7fb574074d60 2024-02-19T19:06:24.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.805+0000 7fb5737fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb574073790 0x7fb5741985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:24.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.805+0000 7fb5737fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb574073790 0x7fb5741985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35816/0 (socket says 172.21.15.114:35816) 2024-02-19T19:06:24.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.805+0000 7fb5737fe700 1 -- 172.21.15.114:0/3395902624 learned_addr learned my addr 172.21.15.114:0/3395902624 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:06:24.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.805+0000 7fb572ffd700 1 --2- 172.21.15.114:0/3395902624 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb574074d60 0x7fb574198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:24.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.805+0000 7fb5737fe700 1 -- 172.21.15.114:0/3395902624 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb574074d60 msgr2=0x7fb574198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:24.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.805+0000 7fb5737fe700 1 --2- 172.21.15.114:0/3395902624 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb574074d60 0x7fb574198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:24.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.805+0000 7fb5737fe700 1 -- 172.21.15.114:0/3395902624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5640095f0 con 0x7fb574073790 2024-02-19T19:06:24.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.805+0000 7fb5737fe700 1 --2- 172.21.15.114:0/3395902624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb574073790 0x7fb5741985d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb564005b40 tx=0x7fb56400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:24.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.805+0000 7fb570ff9700 1 -- 172.21.15.114:0/3395902624 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb56401d070 con 0x7fb574073790 2024-02-19T19:06:24.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.805+0000 7fb579c45700 1 -- 172.21.15.114:0/3395902624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57419dcc0 con 0x7fb574073790 2024-02-19T19:06:24.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.805+0000 7fb570ff9700 1 -- 172.21.15.114:0/3395902624 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb564022470 con 0x7fb574073790 2024-02-19T19:06:24.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.805+0000 7fb570ff9700 1 -- 172.21.15.114:0/3395902624 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb56400f690 con 0x7fb574073790 2024-02-19T19:06:24.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.805+0000 7fb579c45700 1 -- 172.21.15.114:0/3395902624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57419e1b0 con 0x7fb574073790 2024-02-19T19:06:24.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.806+0000 7fb579c45700 1 -- 172.21.15.114:0/3395902624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57410b980 con 0x7fb574073790 2024-02-19T19:06:24.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.807+0000 7fb570ff9700 1 -- 172.21.15.114:0/3395902624 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7fb564022a10 con 0x7fb574073790 2024-02-19T19:06:24.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.807+0000 7fb570ff9700 1 --2- 172.21.15.114:0/3395902624 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb55c06d8a0 0x7fb55c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:24.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.807+0000 7fb570ff9700 1 -- 172.21.15.114:0/3395902624 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6790+0+0 (secure 0 0 0) 0x7fb56408eee0 con 0x7fb574073790 2024-02-19T19:06:24.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.807+0000 7fb572ffd700 1 --2- 172.21.15.114:0/3395902624 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb55c06d8a0 0x7fb55c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:24.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.808+0000 7fb572ffd700 1 --2- 172.21.15.114:0/3395902624 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb55c06d8a0 0x7fb55c06fd60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb568009a10 tx=0x7fb568009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:24.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:24.812+0000 7fb570ff9700 1 -- 172.21.15.114:0/3395902624 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb56405c340 con 0x7fb574073790 2024-02-19T19:06:25.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.070+0000 7fb579c45700 1 -- 172.21.15.114:0/3395902624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb57419e840 con 0x7fb574073790 2024-02-19T19:06:25.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.070+0000 7fb570ff9700 1 -- 172.21.15.114:0/3395902624 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fb564027070 con 0x7fb574073790 2024-02-19T19:06:25.091 INFO:teuthology.orchestra.run.smithi114.stdout:e18 2024-02-19T19:06:25.091 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:06:25.091 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:06:25.092 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-02-19T19:06:25.092 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:06:25.092 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:06:25.092 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-02-19T19:06:25.092 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 18 2024-02-19T19:06:25.092 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:06:25.092 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:06:25.092 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-02-19T19:00:21.203922+0000 2024-02-19T19:06:25.092 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-02-19T19:06:25.092 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-02-19T19:06:25.092 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-02-19T19:06:25.092 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-02-19T19:06:25.092 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-02-19T19:06:25.092 INFO:teuthology.orchestra.run.smithi114.stdout:max_xattr_size 65536 2024-02-19T19:06:25.092 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-02-19T19:06:25.093 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-02-19T19:06:25.093 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 47 2024-02-19T19:06:25.093 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:06:25.093 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 2 2024-02-19T19:06:25.093 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1 2024-02-19T19:06:25.093 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=14530,1=24269} 2024-02-19T19:06:25.093 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-02-19T19:06:25.093 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-02-19T19:06:25.093 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-02-19T19:06:25.094 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-02-19T19:06:25.094 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-02-19T19:06:25.094 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data enabled 2024-02-19T19:06:25.094 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-02-19T19:06:25.094 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-02-19T19:06:25.094 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 1 2024-02-19T19:06:25.094 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:25.094 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.vpoeop{0:14540} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:25.094 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.znhuvd{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:25.094 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.stnfnf{1:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:25.094 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:06:25.094 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:06:25.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.072+0000 7fb579c45700 1 -- 172.21.15.114:0/3395902624 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb55c06d8a0 msgr2=0x7fb55c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:25.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.072+0000 7fb579c45700 1 --2- 172.21.15.114:0/3395902624 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb55c06d8a0 0x7fb55c06fd60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb568009a10 tx=0x7fb568009190 comp rx=0 tx=0).stop 2024-02-19T19:06:25.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.072+0000 7fb579c45700 1 -- 172.21.15.114:0/3395902624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb574073790 msgr2=0x7fb5741985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:25.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.072+0000 7fb579c45700 1 --2- 172.21.15.114:0/3395902624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb574073790 0x7fb5741985d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb564005b40 tx=0x7fb56400bd70 comp rx=0 tx=0).stop 2024-02-19T19:06:25.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.073+0000 7fb579c45700 1 -- 172.21.15.114:0/3395902624 shutdown_connections 2024-02-19T19:06:25.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.073+0000 7fb579c45700 1 --2- 172.21.15.114:0/3395902624 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb55c06d8a0 0x7fb55c06fd60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:25.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.073+0000 7fb579c45700 1 --2- 172.21.15.114:0/3395902624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb574073790 0x7fb5741985d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:25.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.073+0000 7fb579c45700 1 --2- 172.21.15.114:0/3395902624 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb574074d60 0x7fb574198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:25.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.073+0000 7fb579c45700 1 -- 172.21.15.114:0/3395902624 >> 172.21.15.114:0/3395902624 conn(0x7fb5740fc110 msgr2=0x7fb5741069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:25.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.073+0000 7fb579c45700 1 -- 172.21.15.114:0/3395902624 shutdown_connections 2024-02-19T19:06:25.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.073+0000 7fb579c45700 1 -- 172.21.15.114:0/3395902624 wait complete. 2024-02-19T19:06:25.096 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 18 2024-02-19T19:06:25.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.181+0000 7f5e020ae700 1 -- 172.21.15.114:0/3948580073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dfc100cb0 msgr2=0x7f5dfc101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:25.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.181+0000 7f5e020ae700 1 --2- 172.21.15.114:0/3948580073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dfc100cb0 0x7f5dfc101110 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5df0009910 tx=0x7f5df0009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:25.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.181+0000 7f5e020ae700 1 -- 172.21.15.114:0/3948580073 shutdown_connections 2024-02-19T19:06:25.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.181+0000 7f5e020ae700 1 --2- 172.21.15.114:0/3948580073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dfc100cb0 0x7f5dfc101110 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:25.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.181+0000 7f5e020ae700 1 --2- 172.21.15.114:0/3948580073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5dfc0ffab0 0x7f5dfc0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:25.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.181+0000 7f5e020ae700 1 -- 172.21.15.114:0/3948580073 >> 172.21.15.114:0/3948580073 conn(0x7f5dfc0fb030 msgr2=0x7f5dfc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:25.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.181+0000 7f5e020ae700 1 -- 172.21.15.114:0/3948580073 shutdown_connections 2024-02-19T19:06:25.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.181+0000 7f5e020ae700 1 -- 172.21.15.114:0/3948580073 wait complete. 2024-02-19T19:06:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.182+0000 7f5e020ae700 1 Processor -- start 2024-02-19T19:06:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.182+0000 7f5e020ae700 1 -- start start 2024-02-19T19:06:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.182+0000 7f5e020ae700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dfc0ffab0 0x7f5dfc198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.182+0000 7f5e020ae700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5dfc100cb0 0x7f5dfc198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.182+0000 7f5e020ae700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc199020 con 0x7f5dfc0ffab0 2024-02-19T19:06:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.182+0000 7f5e020ae700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc199160 con 0x7f5dfc100cb0 2024-02-19T19:06:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.182+0000 7f5dfb7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dfc0ffab0 0x7f5dfc198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.182+0000 7f5dfb7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dfc0ffab0 0x7f5dfc198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35836/0 (socket says 172.21.15.114:35836) 2024-02-19T19:06:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.182+0000 7f5dfb7fe700 1 -- 172.21.15.114:0/4258064012 learned_addr learned my addr 172.21.15.114:0/4258064012 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:06:25.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.182+0000 7f5dfaffd700 1 --2- 172.21.15.114:0/4258064012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5dfc100cb0 0x7f5dfc198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.183+0000 7f5dfb7fe700 1 -- 172.21.15.114:0/4258064012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5dfc100cb0 msgr2=0x7f5dfc198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.183+0000 7f5dfb7fe700 1 --2- 172.21.15.114:0/4258064012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5dfc100cb0 0x7f5dfc198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.183+0000 7f5dfb7fe700 1 -- 172.21.15.114:0/4258064012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df00095f0 con 0x7f5dfc0ffab0 2024-02-19T19:06:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.183+0000 7f5dfb7fe700 1 --2- 172.21.15.114:0/4258064012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dfc0ffab0 0x7f5dfc198550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00c910 tx=0x7f5dec00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.183+0000 7f5df8ff9700 1 -- 172.21.15.114:0/4258064012 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dec0041f0 con 0x7f5dfc0ffab0 2024-02-19T19:06:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.183+0000 7f5e020ae700 1 -- 172.21.15.114:0/4258064012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dfc19dc20 con 0x7f5dfc0ffab0 2024-02-19T19:06:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.183+0000 7f5df8ff9700 1 -- 172.21.15.114:0/4258064012 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5dec00ce60 con 0x7f5dfc0ffab0 2024-02-19T19:06:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.183+0000 7f5df8ff9700 1 -- 172.21.15.114:0/4258064012 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dec003d20 con 0x7f5dfc0ffab0 2024-02-19T19:06:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.183+0000 7f5e020ae700 1 -- 172.21.15.114:0/4258064012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dfc19e140 con 0x7f5dfc0ffab0 2024-02-19T19:06:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.184+0000 7f5e020ae700 1 -- 172.21.15.114:0/4258064012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dfc108610 con 0x7f5dfc0ffab0 2024-02-19T19:06:25.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.185+0000 7f5df8ff9700 1 -- 172.21.15.114:0/4258064012 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f5dec007900 con 0x7f5dfc0ffab0 2024-02-19T19:06:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.185+0000 7f5df8ff9700 1 --2- 172.21.15.114:0/4258064012 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f5de406d8a0 0x7f5de406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.185+0000 7f5dfaffd700 1 --2- 172.21.15.114:0/4258064012 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f5de406d8a0 0x7f5de406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.185+0000 7f5df8ff9700 1 -- 172.21.15.114:0/4258064012 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f5dec08d2f0 con 0x7f5dfc0ffab0 2024-02-19T19:06:25.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.186+0000 7f5dfaffd700 1 --2- 172.21.15.114:0/4258064012 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f5de406d8a0 0x7f5de406fd60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5df0006010 tx=0x7f5df000b560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:25.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.194+0000 7f5df8ff9700 1 -- 172.21.15.114:0/4258064012 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5dec05a500 con 0x7f5dfc0ffab0 2024-02-19T19:06:25.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.431+0000 7f5e020ae700 1 -- 172.21.15.114:0/4258064012 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5dfc068b20 con 0x7f5de406d8a0 2024-02-19T19:06:25.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.432+0000 7f5df8ff9700 1 -- 172.21.15.114:0/4258064012 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f5dfc068b20 con 0x7f5de406d8a0 2024-02-19T19:06:25.453 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:06:25.453 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:06:25.453 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-02-19T19:06:25.453 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:06:25.453 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [ 2024-02-19T19:06:25.453 INFO:teuthology.orchestra.run.smithi114.stdout: "crash", 2024-02-19T19:06:25.453 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr", 2024-02-19T19:06:25.454 INFO:teuthology.orchestra.run.smithi114.stdout: "mon" 2024-02-19T19:06:25.454 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-02-19T19:06:25.454 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "8/23 daemons upgraded", 2024-02-19T19:06:25.454 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "Currently upgrading osd daemons", 2024-02-19T19:06:25.454 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-02-19T19:06:25.454 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:06:25.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.434+0000 7f5e020ae700 1 -- 172.21.15.114:0/4258064012 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f5de406d8a0 msgr2=0x7f5de406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:25.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.434+0000 7f5e020ae700 1 --2- 172.21.15.114:0/4258064012 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f5de406d8a0 0x7f5de406fd60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5df0006010 tx=0x7f5df000b560 comp rx=0 tx=0).stop 2024-02-19T19:06:25.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.435+0000 7f5e020ae700 1 -- 172.21.15.114:0/4258064012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dfc0ffab0 msgr2=0x7f5dfc198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:25.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.435+0000 7f5e020ae700 1 --2- 172.21.15.114:0/4258064012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dfc0ffab0 0x7f5dfc198550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00c910 tx=0x7f5dec00cc20 comp rx=0 tx=0).stop 2024-02-19T19:06:25.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.435+0000 7f5e020ae700 1 -- 172.21.15.114:0/4258064012 shutdown_connections 2024-02-19T19:06:25.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.435+0000 7f5e020ae700 1 --2- 172.21.15.114:0/4258064012 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f5de406d8a0 0x7f5de406fd60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:25.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.435+0000 7f5e020ae700 1 --2- 172.21.15.114:0/4258064012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5dfc0ffab0 0x7f5dfc198550 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:25.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.435+0000 7f5e020ae700 1 --2- 172.21.15.114:0/4258064012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5dfc100cb0 0x7f5dfc198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:25.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.435+0000 7f5e020ae700 1 -- 172.21.15.114:0/4258064012 >> 172.21.15.114:0/4258064012 conn(0x7f5dfc0fb030 msgr2=0x7f5dfc068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:25.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.436+0000 7f5e020ae700 1 -- 172.21.15.114:0/4258064012 shutdown_connections 2024-02-19T19:06:25.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.436+0000 7f5e020ae700 1 -- 172.21.15.114:0/4258064012 wait complete. 2024-02-19T19:06:25.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.556+0000 7f8c51aad700 1 -- 172.21.15.114:0/3802859941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c4c100cb0 msgr2=0x7f8c4c101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:25.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.556+0000 7f8c51aad700 1 --2- 172.21.15.114:0/3802859941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c4c100cb0 0x7f8c4c101110 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8c40009910 tx=0x7f8c40009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:25.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.556+0000 7f8c51aad700 1 -- 172.21.15.114:0/3802859941 shutdown_connections 2024-02-19T19:06:25.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.556+0000 7f8c51aad700 1 --2- 172.21.15.114:0/3802859941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c4c100cb0 0x7f8c4c101110 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:25.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.556+0000 7f8c51aad700 1 --2- 172.21.15.114:0/3802859941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c4c0ffab0 0x7f8c4c0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:25.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.556+0000 7f8c51aad700 1 -- 172.21.15.114:0/3802859941 >> 172.21.15.114:0/3802859941 conn(0x7f8c4c0fb030 msgr2=0x7f8c4c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:25.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.556+0000 7f8c51aad700 1 -- 172.21.15.114:0/3802859941 shutdown_connections 2024-02-19T19:06:25.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.556+0000 7f8c51aad700 1 -- 172.21.15.114:0/3802859941 wait complete. 2024-02-19T19:06:25.578 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:25 smithi114 ceph-mon[92983]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:25.578 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:25 smithi114 ceph-mon[92983]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:25.578 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:25 smithi114 ceph-mon[92983]: from='osd.2 [v2:172.21.15.114:6818/3670286774,v1:172.21.15.114:6819/3670286774]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-19T19:06:25.578 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:25 smithi114 ceph-mon[92983]: osdmap e66: 6 total, 5 up, 6 in 2024-02-19T19:06:25.578 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:25 smithi114 ceph-mon[92983]: from='osd.2 [v2:172.21.15.114:6818/3670286774,v1:172.21.15.114:6819/3670286774]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-02-19T19:06:25.578 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:25 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/1395637209' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:06:25.578 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:25.578 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:06:25.578 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:25 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/3395902624' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:06:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.557+0000 7f8c51aad700 1 Processor -- start 2024-02-19T19:06:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.557+0000 7f8c51aad700 1 -- start start 2024-02-19T19:06:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.557+0000 7f8c51aad700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c4c0ffab0 0x7f8c4c068f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.557+0000 7f8c51aad700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c4c100cb0 0x7f8c4c19b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.557+0000 7f8c51aad700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c19b670 con 0x7f8c4c0ffab0 2024-02-19T19:06:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.557+0000 7f8c51aad700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c19b7e0 con 0x7f8c4c100cb0 2024-02-19T19:06:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.558+0000 7f8c4b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c4c0ffab0 0x7f8c4c068f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.558+0000 7f8c4b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c4c0ffab0 0x7f8c4c068f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37006/0 (socket says 172.21.15.114:37006) 2024-02-19T19:06:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.558+0000 7f8c4b7fe700 1 -- 172.21.15.114:0/1487460549 learned_addr learned my addr 172.21.15.114:0/1487460549 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:06:25.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.558+0000 7f8c4affd700 1 --2- 172.21.15.114:0/1487460549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c4c100cb0 0x7f8c4c19b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.558+0000 7f8c4b7fe700 1 -- 172.21.15.114:0/1487460549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c4c100cb0 msgr2=0x7f8c4c19b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.558+0000 7f8c4b7fe700 1 --2- 172.21.15.114:0/1487460549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c4c100cb0 0x7f8c4c19b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.558+0000 7f8c4b7fe700 1 -- 172.21.15.114:0/1487460549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c400095f0 con 0x7f8c4c0ffab0 2024-02-19T19:06:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.558+0000 7f8c4b7fe700 1 --2- 172.21.15.114:0/1487460549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c4c0ffab0 0x7f8c4c068f90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00c8f0 tx=0x7f8c3c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.559+0000 7f8c48ff9700 1 -- 172.21.15.114:0/1487460549 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c3c007a70 con 0x7f8c4c0ffab0 2024-02-19T19:06:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.559+0000 7f8c51aad700 1 -- 172.21.15.114:0/1487460549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4c19bac0 con 0x7f8c4c0ffab0 2024-02-19T19:06:25.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.559+0000 7f8c51aad700 1 -- 172.21.15.114:0/1487460549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4c19c010 con 0x7f8c4c0ffab0 2024-02-19T19:06:25.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.559+0000 7f8c48ff9700 1 -- 172.21.15.114:0/1487460549 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8c3c007bd0 con 0x7f8c4c0ffab0 2024-02-19T19:06:25.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.559+0000 7f8c48ff9700 1 -- 172.21.15.114:0/1487460549 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c3c018630 con 0x7f8c4c0ffab0 2024-02-19T19:06:25.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.560+0000 7f8c51aad700 1 -- 172.21.15.114:0/1487460549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c38005320 con 0x7f8c4c0ffab0 2024-02-19T19:06:25.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.561+0000 7f8c48ff9700 1 -- 172.21.15.114:0/1487460549 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f8c3c00fa00 con 0x7f8c4c0ffab0 2024-02-19T19:06:25.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.562+0000 7f8c48ff9700 1 --2- 172.21.15.114:0/1487460549 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f8c3406d910 0x7f8c3406fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.562+0000 7f8c48ff9700 1 -- 172.21.15.114:0/1487460549 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f8c3c08d540 con 0x7f8c4c0ffab0 2024-02-19T19:06:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.562+0000 7f8c4affd700 1 --2- 172.21.15.114:0/1487460549 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f8c3406d910 0x7f8c3406fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:25.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.563+0000 7f8c4affd700 1 --2- 172.21.15.114:0/1487460549 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f8c3406d910 0x7f8c3406fdd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0694f0 tx=0x7f8c4000b580 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:25.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.571+0000 7f8c48ff9700 1 -- 172.21.15.114:0/1487460549 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8c3c05a750 con 0x7f8c4c0ffab0 2024-02-19T19:06:25.644 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:25 smithi174 ceph-mon[87279]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:25.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:25 smithi174 ceph-mon[87279]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:25.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:25 smithi174 ceph-mon[87279]: from='osd.2 [v2:172.21.15.114:6818/3670286774,v1:172.21.15.114:6819/3670286774]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-19T19:06:25.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:25 smithi174 ceph-mon[87279]: osdmap e66: 6 total, 5 up, 6 in 2024-02-19T19:06:25.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:25 smithi174 ceph-mon[87279]: from='osd.2 [v2:172.21.15.114:6818/3670286774,v1:172.21.15.114:6819/3670286774]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-02-19T19:06:25.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:25 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/1395637209' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:06:25.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:25.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:06:25.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:25 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/3395902624' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:06:25.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.870+0000 7f8c51aad700 1 -- 172.21.15.114:0/1487460549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8c38005190 con 0x7f8c4c0ffab0 2024-02-19T19:06:25.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.870+0000 7f8c48ff9700 1 -- 172.21.15.114:0/1487460549 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+931 (secure 0 0 0) 0x7f8c3c05a2e0 con 0x7f8c4c0ffab0 2024-02-19T19:06:25.891 INFO:teuthology.orchestra.run.smithi114.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; Degraded data redundancy: 19/129 objects degraded (14.729%), 10 pgs degraded 2024-02-19T19:06:25.891 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:06:25.891 INFO:teuthology.orchestra.run.smithi114.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:06:25.891 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 19/129 objects degraded (14.729%), 10 pgs degraded 2024-02-19T19:06:25.892 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.3 is active+undersized+degraded, acting [5,1] 2024-02-19T19:06:25.892 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.7 is active+undersized+degraded, acting [3,4] 2024-02-19T19:06:25.892 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.d is active+undersized+degraded, acting [1,3] 2024-02-19T19:06:25.892 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.e is active+undersized+degraded, acting [0,1] 2024-02-19T19:06:25.892 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.10 is active+undersized+degraded, acting [1,0] 2024-02-19T19:06:25.892 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.12 is active+undersized+degraded, acting [1,0] 2024-02-19T19:06:25.892 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.13 is active+undersized+degraded, acting [1,4] 2024-02-19T19:06:25.892 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.16 is active+undersized+degraded, acting [5,1] 2024-02-19T19:06:25.892 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.17 is active+undersized+degraded, acting [5,1] 2024-02-19T19:06:25.893 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.1c is active+undersized+degraded, acting [4,5] 2024-02-19T19:06:25.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.873+0000 7f8c51aad700 1 -- 172.21.15.114:0/1487460549 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f8c3406d910 msgr2=0x7f8c3406fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:25.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.873+0000 7f8c51aad700 1 --2- 172.21.15.114:0/1487460549 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f8c3406d910 0x7f8c3406fdd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0694f0 tx=0x7f8c4000b580 comp rx=0 tx=0).stop 2024-02-19T19:06:25.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.873+0000 7f8c51aad700 1 -- 172.21.15.114:0/1487460549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c4c0ffab0 msgr2=0x7f8c4c068f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:25.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.873+0000 7f8c51aad700 1 --2- 172.21.15.114:0/1487460549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c4c0ffab0 0x7f8c4c068f90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00c8f0 tx=0x7f8c3c00ccb0 comp rx=0 tx=0).stop 2024-02-19T19:06:25.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.873+0000 7f8c51aad700 1 -- 172.21.15.114:0/1487460549 shutdown_connections 2024-02-19T19:06:25.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.873+0000 7f8c51aad700 1 --2- 172.21.15.114:0/1487460549 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f8c3406d910 0x7f8c3406fdd0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:25.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.873+0000 7f8c51aad700 1 --2- 172.21.15.114:0/1487460549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c4c0ffab0 0x7f8c4c068f90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:25.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.873+0000 7f8c51aad700 1 --2- 172.21.15.114:0/1487460549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c4c100cb0 0x7f8c4c19b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:25.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.873+0000 7f8c51aad700 1 -- 172.21.15.114:0/1487460549 >> 172.21.15.114:0/1487460549 conn(0x7f8c4c0fb030 msgr2=0x7f8c4c103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:25.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.873+0000 7f8c51aad700 1 -- 172.21.15.114:0/1487460549 shutdown_connections 2024-02-19T19:06:25.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:25.873+0000 7f8c51aad700 1 -- 172.21.15.114:0/1487460549 wait complete. 2024-02-19T19:06:26.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:26 smithi114 ceph-mon[92983]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:06:26.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:26 smithi114 ceph-mon[92983]: osd.2 [v2:172.21.15.114:6818/3670286774,v1:172.21.15.114:6819/3670286774] boot 2024-02-19T19:06:26.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:26 smithi114 ceph-mon[92983]: osdmap e67: 6 total, 6 up, 6 in 2024-02-19T19:06:26.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:26 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:06:26.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:26 smithi114 ceph-mon[92983]: from='client.34300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:26.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:26 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:26.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:26 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:26.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:26 smithi114 ceph-mon[92983]: pgmap v103: 65 pgs: 11 peering, 5 active+undersized+degraded, 13 active+undersized, 36 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 11/129 objects degraded (8.527%) 2024-02-19T19:06:26.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:26 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/1487460549' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:06:26.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:26 smithi114 ceph-mon[92983]: Health check update: Degraded data redundancy: 19/129 objects degraded (14.729%), 10 pgs degraded (PG_DEGRADED) 2024-02-19T19:06:26.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:26 smithi174 ceph-mon[87279]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:06:26.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:26 smithi174 ceph-mon[87279]: osd.2 [v2:172.21.15.114:6818/3670286774,v1:172.21.15.114:6819/3670286774] boot 2024-02-19T19:06:26.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:26 smithi174 ceph-mon[87279]: osdmap e67: 6 total, 6 up, 6 in 2024-02-19T19:06:26.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:26 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:06:26.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:26 smithi174 ceph-mon[87279]: from='client.34300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:26.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:26 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:26.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:26 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:26.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:26 smithi174 ceph-mon[87279]: pgmap v103: 65 pgs: 11 peering, 5 active+undersized+degraded, 13 active+undersized, 36 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 11/129 objects degraded (8.527%) 2024-02-19T19:06:26.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:26 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/1487460549' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:06:26.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:26 smithi174 ceph-mon[87279]: Health check update: Degraded data redundancy: 19/129 objects degraded (14.729%), 10 pgs degraded (PG_DEGRADED) 2024-02-19T19:06:27.700 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:27 smithi114 ceph-mon[92983]: osdmap e68: 6 total, 6 up, 6 in 2024-02-19T19:06:27.700 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:27 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:27.700 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:27 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:27.809 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:27 smithi174 ceph-mon[87279]: osdmap e68: 6 total, 6 up, 6 in 2024-02-19T19:06:27.809 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:27 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:27.809 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:27 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:28.747 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:28 smithi114 ceph-mon[92983]: pgmap v105: 65 pgs: 11 peering, 5 active+undersized+degraded, 9 active+undersized, 40 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s; 11/129 objects degraded (8.527%) 2024-02-19T19:06:28.894 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:28 smithi174 ceph-mon[87279]: pgmap v105: 65 pgs: 11 peering, 5 active+undersized+degraded, 9 active+undersized, 40 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s; 11/129 objects degraded (8.527%) 2024-02-19T19:06:29.920 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:29 smithi114 ceph-mon[92983]: pgmap v106: 65 pgs: 11 peering, 54 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-02-19T19:06:29.920 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:29 smithi114 ceph-mon[92983]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/129 objects degraded (8.527%), 5 pgs degraded) 2024-02-19T19:06:30.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:29 smithi174 ceph-mon[87279]: pgmap v106: 65 pgs: 11 peering, 54 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-02-19T19:06:30.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:29 smithi174 ceph-mon[87279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/129 objects degraded (8.527%), 5 pgs degraded) 2024-02-19T19:06:31.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:31 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:31.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:31 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:31.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:31 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:31.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:31 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:06:31.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:31 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:31.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:31 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:06:31.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:31 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:06:31.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:31 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:06:31.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:31 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:06:31.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:31 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-19T19:06:31.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:31 smithi174 ceph-mon[87279]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-19T19:06:31.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:31 smithi174 ceph-mon[87279]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-02-19T19:06:31.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:31 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:31.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:31 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:31.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:31 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:31.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:31 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:06:31.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:31 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:31.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:31 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:06:31.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:31 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:06:31.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:31 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:06:31.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:31 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:06:31.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:31 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-19T19:06:31.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:31 smithi114 ceph-mon[92983]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-19T19:06:31.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:31 smithi114 ceph-mon[92983]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-02-19T19:06:32.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:32 smithi174 ceph-mon[87279]: pgmap v107: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 4 op/s 2024-02-19T19:06:32.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:32 smithi114 ceph-mon[92983]: pgmap v107: 65 pgs: 65 active+clean; 594 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 4 op/s 2024-02-19T19:06:34.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:33 smithi114 ceph-mon[92983]: pgmap v108: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-02-19T19:06:34.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:33 smithi174 ceph-mon[87279]: pgmap v108: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-02-19T19:06:36.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:35 smithi174 ceph-mon[87279]: pgmap v109: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-02-19T19:06:36.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:35 smithi114 ceph-mon[92983]: pgmap v109: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-02-19T19:06:38.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:37 smithi174 ceph-mon[87279]: pgmap v110: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-02-19T19:06:38.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:37 smithi114 ceph-mon[92983]: pgmap v110: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-02-19T19:06:40.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:39 smithi174 ceph-mon[87279]: pgmap v111: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:06:40.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:39 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:40.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:39 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:06:40.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:39 smithi114 ceph-mon[92983]: pgmap v111: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:06:40.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:39 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:40.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:39 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:06:41.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:40 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:41.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:40 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:42.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:41 smithi174 ceph-mon[87279]: pgmap v112: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:06:42.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:41 smithi114 ceph-mon[92983]: pgmap v112: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:06:44.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:43 smithi114 ceph-mon[92983]: pgmap v113: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:06:44.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:43 smithi174 ceph-mon[87279]: pgmap v113: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:06:45.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:45 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-19T19:06:46.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:45 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-19T19:06:46.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:46 smithi174 ceph-mon[87279]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-19T19:06:46.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:46 smithi174 ceph-mon[87279]: Upgrade: osd.3 is safe to restart 2024-02-19T19:06:46.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:46 smithi174 ceph-mon[87279]: pgmap v114: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:06:47.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:46 smithi114 ceph-mon[92983]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-19T19:06:47.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:46 smithi114 ceph-mon[92983]: Upgrade: osd.3 is safe to restart 2024-02-19T19:06:47.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:46 smithi114 ceph-mon[92983]: pgmap v114: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:06:48.359 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:48 smithi174 ceph-mon[87279]: Upgrade: Updating osd.3 2024-02-19T19:06:48.359 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:48 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:48.359 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:48 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-19T19:06:48.359 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:48 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:48.359 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:48 smithi174 ceph-mon[87279]: Deploying daemon osd.3 on smithi174 2024-02-19T19:06:48.359 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:48 smithi174 ceph-mon[87279]: pgmap v115: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:06:48.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:48 smithi114 ceph-mon[92983]: Upgrade: Updating osd.3 2024-02-19T19:06:48.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:48 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:48.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:48 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-19T19:06:48.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:48 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:06:48.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:48 smithi114 ceph-mon[92983]: Deploying daemon osd.3 on smithi174 2024-02-19T19:06:48.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:48 smithi114 ceph-mon[92983]: pgmap v115: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:06:49.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:49 smithi174 ceph-mon[87279]: pgmap v116: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:06:50.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:49 smithi114 ceph-mon[92983]: pgmap v116: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:06:50.645 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:50 smithi174 systemd[1]: Stopping Ceph osd.3 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T19:06:51.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:50 smithi174 ceph-mon[87279]: osd.3 marked itself down and dead 2024-02-19T19:06:51.145 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:50 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3[47098]: 2024-02-19T19:06:50.624+0000 7fc455353700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T19:06:51.145 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:50 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3[47098]: 2024-02-19T19:06:50.624+0000 7fc455353700 -1 osd.3 68 *** Got signal Terminated *** 2024-02-19T19:06:51.145 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:50 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3[47098]: 2024-02-19T19:06:50.625+0000 7fc455353700 -1 osd.3 68 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T19:06:51.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:50 smithi114 ceph-mon[92983]: osd.3 marked itself down and dead 2024-02-19T19:06:51.563 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:51 smithi174 podman[96906]: 2024-02-19 19:06:51.248539833 +0000 UTC m=+0.750571937 container died 979f45cabbce6c1468576d84d7bbf07515252e4d32af71f8ccc4da9fe83d7b53 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=HEAD, io.buildah.version=1.29.1) 2024-02-19T19:06:51.849 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:51 smithi174 podman[96906]: 2024-02-19 19:06:51.668090965 +0000 UTC m=+1.170123080 container cleanup 979f45cabbce6c1468576d84d7bbf07515252e4d32af71f8ccc4da9fe83d7b53 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.buildah.version=1.29.1) 2024-02-19T19:06:51.849 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:51 smithi174 bash[96906]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3 2024-02-19T19:06:52.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:51 smithi114 ceph-mon[92983]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:06:52.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:51 smithi114 ceph-mon[92983]: osdmap e69: 6 total, 5 up, 6 in 2024-02-19T19:06:52.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:51 smithi114 ceph-mon[92983]: pgmap v118: 65 pgs: 14 stale+active+clean, 51 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-02-19T19:06:52.145 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:51 smithi174 podman[96924]: 2024-02-19 19:06:51.894741826 +0000 UTC m=+0.637605896 container remove 979f45cabbce6c1468576d84d7bbf07515252e4d32af71f8ccc4da9fe83d7b53 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , RELEASE=HEAD, GIT_CLEAN=True) 2024-02-19T19:06:52.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:51 smithi174 ceph-mon[87279]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:06:52.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:51 smithi174 ceph-mon[87279]: osdmap e69: 6 total, 5 up, 6 in 2024-02-19T19:06:52.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:51 smithi174 ceph-mon[87279]: pgmap v118: 65 pgs: 14 stale+active+clean, 51 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-02-19T19:06:52.645 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:52 smithi174 podman[97015]: 2024-02-19 19:06:52.232212807 +0000 UTC m=+0.018038433 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:06:52.645 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:52 smithi174 podman[97015]: 2024-02-19 19:06:52.337441336 +0000 UTC m=+0.123267024 container create a91e00e79b44e310b3393ebea337c2d6046b7ee7296f517e1c38141b0323fd83 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-02-19T19:06:52.645 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:52 smithi174 podman[97015]: 2024-02-19 19:06:52.564900442 +0000 UTC m=+0.350726080 container init a91e00e79b44e310b3393ebea337c2d6046b7ee7296f517e1c38141b0323fd83 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214) 2024-02-19T19:06:52.645 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:52 smithi174 podman[97015]: 2024-02-19 19:06:52.567858036 +0000 UTC m=+0.353683667 container start a91e00e79b44e310b3393ebea337c2d6046b7ee7296f517e1c38141b0323fd83 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-deactivate, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:06:53.076 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:52 smithi174 ceph-mon[87279]: osdmap e70: 6 total, 5 up, 6 in 2024-02-19T19:06:53.076 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:52 smithi174 podman[97015]: 2024-02-19 19:06:52.67314891 +0000 UTC m=+0.458974541 container attach a91e00e79b44e310b3393ebea337c2d6046b7ee7296f517e1c38141b0323fd83 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-02-19T19:06:53.077 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:52 smithi174 podman[97015]: 2024-02-19 19:06:52.863937767 +0000 UTC m=+0.649763404 container died a91e00e79b44e310b3393ebea337c2d6046b7ee7296f517e1c38141b0323fd83 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, ceph=True) 2024-02-19T19:06:53.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:52 smithi114 ceph-mon[92983]: osdmap e70: 6 total, 5 up, 6 in 2024-02-19T19:06:53.867 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:53 smithi174 podman[97040]: 2024-02-19 19:06:53.443426784 +0000 UTC m=+0.569180333 container remove a91e00e79b44e310b3393ebea337c2d6046b7ee7296f517e1c38141b0323fd83 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-02-19T19:06:53.867 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:53 smithi174 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.3.service: Deactivated successfully. 2024-02-19T19:06:53.867 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:53 smithi174 systemd[1]: Stopped Ceph osd.3 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T19:06:53.868 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:53 smithi174 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.3.service: Consumed 5.725s CPU time. 2024-02-19T19:06:53.868 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:53 smithi174 systemd[1]: Starting Ceph osd.3 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T19:06:54.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:53 smithi174 ceph-mon[87279]: pgmap v120: 65 pgs: 5 active+undersized+degraded, 6 peering, 10 stale+active+clean, 4 active+undersized, 40 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 8/129 objects degraded (6.202%) 2024-02-19T19:06:54.145 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:53 smithi174 podman[97141]: 2024-02-19 19:06:53.796325473 +0000 UTC m=+0.018587470 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:06:54.145 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:53 smithi174 podman[97141]: 2024-02-19 19:06:53.903403034 +0000 UTC m=+0.125665032 container create 1dfc16b139d4d1bd3dbfdc062e5675df22701c7160d4f57c9a2700722540c762 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-02-19T19:06:54.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:53 smithi114 ceph-mon[92983]: pgmap v120: 65 pgs: 5 active+undersized+degraded, 6 peering, 10 stale+active+clean, 4 active+undersized, 40 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 8/129 objects degraded (6.202%) 2024-02-19T19:06:54.594 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 podman[97141]: 2024-02-19 19:06:54.131086355 +0000 UTC m=+0.353348364 container init 1dfc16b139d4d1bd3dbfdc062e5675df22701c7160d4f57c9a2700722540c762 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-19T19:06:54.595 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 podman[97141]: 2024-02-19 19:06:54.134816039 +0000 UTC m=+0.357078045 container start 1dfc16b139d4d1bd3dbfdc062e5675df22701c7160d4f57c9a2700722540c762 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-19T19:06:54.595 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 podman[97141]: 2024-02-19 19:06:54.23934406 +0000 UTC m=+0.461606068 container attach 1dfc16b139d4d1bd3dbfdc062e5675df22701c7160d4f57c9a2700722540c762 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-02-19T19:06:54.869 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-activate[97156]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-19T19:06:54.869 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-activate[97156]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-19T19:06:54.869 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 bash[97141]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-19T19:06:54.869 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 bash[97141]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-19T19:06:54.869 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 bash[97141]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-19T19:06:54.869 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 bash[97141]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-19T19:06:54.869 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 bash[97141]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-02-19T19:06:54.869 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 bash[97141]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-19T19:06:54.869 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 bash[97141]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-19T19:06:54.869 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-activate[97156]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-19T19:06:54.870 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-activate[97156]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-19T19:06:54.870 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-activate[97156]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-02-19T19:06:54.870 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-activate[97156]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-19T19:06:54.870 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-activate[97156]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-19T19:06:54.870 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:54 smithi174 podman[97141]: 2024-02-19 19:06:54.599287468 +0000 UTC m=+0.821549477 container died 1dfc16b139d4d1bd3dbfdc062e5675df22701c7160d4f57c9a2700722540c762 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T19:06:55.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:54 smithi174 ceph-mon[87279]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-19T19:06:55.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:54 smithi174 ceph-mon[87279]: Health check failed: Degraded data redundancy: 8/129 objects degraded (6.202%), 5 pgs degraded (PG_DEGRADED) 2024-02-19T19:06:55.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:54 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:55.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:54 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:06:55.191 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:54 smithi114 ceph-mon[92983]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-19T19:06:55.191 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:54 smithi114 ceph-mon[92983]: Health check failed: Degraded data redundancy: 8/129 objects degraded (6.202%), 5 pgs degraded (PG_DEGRADED) 2024-02-19T19:06:55.191 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:54 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:55.191 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:54 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:06:55.600 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:55 smithi174 podman[97291]: 2024-02-19 19:06:55.24206017 +0000 UTC m=+0.635252739 container remove 1dfc16b139d4d1bd3dbfdc062e5675df22701c7160d4f57c9a2700722540c762 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3-activate, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240214) 2024-02-19T19:06:55.601 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:55 smithi174 podman[97357]: 2024-02-19 19:06:55.35707879 +0000 UTC m=+0.017835713 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:06:55.601 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:55 smithi174 podman[97357]: 2024-02-19 19:06:55.461361767 +0000 UTC m=+0.122118744 container create cc3a55bd908f4173c3f63a6c0853c372cfbb39a1148d5b6501250a3d898735e2 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T19:06:55.875 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:55 smithi174 podman[97357]: 2024-02-19 19:06:55.69706477 +0000 UTC m=+0.357821692 container init cc3a55bd908f4173c3f63a6c0853c372cfbb39a1148d5b6501250a3d898735e2 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-19T19:06:55.875 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:55 smithi174 podman[97357]: 2024-02-19 19:06:55.700186889 +0000 UTC m=+0.360943812 container start cc3a55bd908f4173c3f63a6c0853c372cfbb39a1148d5b6501250a3d898735e2 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:06:55.875 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:55 smithi174 ceph-osd[97376]: -- 172.21.15.174:0/2914706928 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b86083b1e0 con 0x55b86080cc00 2024-02-19T19:06:55.875 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:55 smithi174 bash[97357]: cc3a55bd908f4173c3f63a6c0853c372cfbb39a1148d5b6501250a3d898735e2 2024-02-19T19:06:55.875 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:55 smithi174 systemd[1]: Started Ceph osd.3 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T19:06:56.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.993+0000 7f1bd7699700 1 -- 172.21.15.114:0/2720106772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bd00ff450 msgr2=0x7f1bd0101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:56.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.993+0000 7f1bd7699700 1 --2- 172.21.15.114:0/2720106772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bd00ff450 0x7f1bd0101880 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1bc800b0a0 tx=0x7f1bc800b3b0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.993+0000 7f1bd7699700 1 -- 172.21.15.114:0/2720106772 shutdown_connections 2024-02-19T19:06:56.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.993+0000 7f1bd7699700 1 --2- 172.21.15.114:0/2720106772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1bd0101dc0 0x7f1bd0104250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.993+0000 7f1bd7699700 1 --2- 172.21.15.114:0/2720106772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bd00ff450 0x7f1bd0101880 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.993+0000 7f1bd7699700 1 -- 172.21.15.114:0/2720106772 >> 172.21.15.114:0/2720106772 conn(0x7f1bd00fb050 msgr2=0x7f1bd00fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:56.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.993+0000 7f1bd7699700 1 -- 172.21.15.114:0/2720106772 shutdown_connections 2024-02-19T19:06:56.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.993+0000 7f1bd7699700 1 -- 172.21.15.114:0/2720106772 wait complete. 2024-02-19T19:06:56.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.994+0000 7f1bd7699700 1 Processor -- start 2024-02-19T19:06:56.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.994+0000 7f1bd7699700 1 -- start start 2024-02-19T19:06:56.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.994+0000 7f1bd7699700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bd00ff450 0x7f1bd019c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:56.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.994+0000 7f1bd7699700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1bd0101dc0 0x7f1bd019ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:56.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.994+0000 7f1bd7699700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd019d4d0 con 0x7f1bd00ff450 2024-02-19T19:06:56.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.994+0000 7f1bd7699700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd019d610 con 0x7f1bd0101dc0 2024-02-19T19:06:56.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.994+0000 7f1bd5435700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bd00ff450 0x7f1bd019c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:56.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.994+0000 7f1bd5435700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bd00ff450 0x7f1bd019c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45664/0 (socket says 172.21.15.114:45664) 2024-02-19T19:06:56.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.994+0000 7f1bd5435700 1 -- 172.21.15.114:0/3329229397 learned_addr learned my addr 172.21.15.114:0/3329229397 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:06:56.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.994+0000 7f1bd4c34700 1 --2- 172.21.15.114:0/3329229397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1bd0101dc0 0x7f1bd019ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:56.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.994+0000 7f1bd5435700 1 -- 172.21.15.114:0/3329229397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1bd0101dc0 msgr2=0x7f1bd019ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:56.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.994+0000 7f1bd5435700 1 --2- 172.21.15.114:0/3329229397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1bd0101dc0 0x7f1bd019ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.995+0000 7f1bd5435700 1 -- 172.21.15.114:0/3329229397 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc8009d00 con 0x7f1bd00ff450 2024-02-19T19:06:56.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.995+0000 7f1bd5435700 1 --2- 172.21.15.114:0/3329229397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bd00ff450 0x7f1bd019c970 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1bc80077b0 tx=0x7f1bc8009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:56.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.995+0000 7f1bce7fc700 1 -- 172.21.15.114:0/3329229397 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bc800e040 con 0x7f1bd00ff450 2024-02-19T19:06:56.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.995+0000 7f1bd7699700 1 -- 172.21.15.114:0/3329229397 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd01a2060 con 0x7f1bd00ff450 2024-02-19T19:06:56.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.995+0000 7f1bce7fc700 1 -- 172.21.15.114:0/3329229397 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1bc8003de0 con 0x7f1bd00ff450 2024-02-19T19:06:56.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.995+0000 7f1bce7fc700 1 -- 172.21.15.114:0/3329229397 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bc801d7f0 con 0x7f1bd00ff450 2024-02-19T19:06:56.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.995+0000 7f1bd7699700 1 -- 172.21.15.114:0/3329229397 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd01a25d0 con 0x7f1bd00ff450 2024-02-19T19:06:56.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.996+0000 7f1bd7699700 1 -- 172.21.15.114:0/3329229397 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd0196b30 con 0x7f1bd00ff450 2024-02-19T19:06:56.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.996+0000 7f1bce7fc700 1 -- 172.21.15.114:0/3329229397 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f1bc8004780 con 0x7f1bd00ff450 2024-02-19T19:06:56.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.997+0000 7f1bce7fc700 1 --2- 172.21.15.114:0/3329229397 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f1bb806d8a0 0x7f1bb806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:56.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.997+0000 7f1bce7fc700 1 -- 172.21.15.114:0/3329229397 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f1bc808f0f0 con 0x7f1bd00ff450 2024-02-19T19:06:56.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.997+0000 7f1bd4c34700 1 --2- 172.21.15.114:0/3329229397 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f1bb806d8a0 0x7f1bb806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:56.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:55.997+0000 7f1bd4c34700 1 --2- 172.21.15.114:0/3329229397 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f1bb806d8a0 0x7f1bb806fd60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4005950 tx=0x7f1bc400b410 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:56.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.002+0000 7f1bce7fc700 1 -- 172.21.15.114:0/3329229397 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1bc805c300 con 0x7f1bd00ff450 2024-02-19T19:06:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:55 smithi174 ceph-mon[87279]: pgmap v121: 65 pgs: 10 active+undersized+degraded, 6 peering, 17 active+undersized, 32 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 19/129 objects degraded (14.729%) 2024-02-19T19:06:56.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:55 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:56.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.214+0000 7f1bd7699700 1 -- 172.21.15.114:0/3329229397 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1bd00611d0 con 0x7f1bb806d8a0 2024-02-19T19:06:56.235 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:55 smithi114 ceph-mon[92983]: pgmap v121: 65 pgs: 10 active+undersized+degraded, 6 peering, 17 active+undersized, 32 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 19/129 objects degraded (14.729%) 2024-02-19T19:06:56.235 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:55 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:56.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.215+0000 7f1bce7fc700 1 -- 172.21.15.114:0/3329229397 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f1bd00611d0 con 0x7f1bb806d8a0 2024-02-19T19:06:56.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.217+0000 7f1bd7699700 1 -- 172.21.15.114:0/3329229397 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f1bb806d8a0 msgr2=0x7f1bb806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:56.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.217+0000 7f1bd7699700 1 --2- 172.21.15.114:0/3329229397 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f1bb806d8a0 0x7f1bb806fd60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4005950 tx=0x7f1bc400b410 comp rx=0 tx=0).stop 2024-02-19T19:06:56.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.217+0000 7f1bd7699700 1 -- 172.21.15.114:0/3329229397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bd00ff450 msgr2=0x7f1bd019c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:56.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.217+0000 7f1bd7699700 1 --2- 172.21.15.114:0/3329229397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bd00ff450 0x7f1bd019c970 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1bc80077b0 tx=0x7f1bc8009760 comp rx=0 tx=0).stop 2024-02-19T19:06:56.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.217+0000 7f1bd7699700 1 -- 172.21.15.114:0/3329229397 shutdown_connections 2024-02-19T19:06:56.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.217+0000 7f1bd7699700 1 --2- 172.21.15.114:0/3329229397 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f1bb806d8a0 0x7f1bb806fd60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.217+0000 7f1bd7699700 1 --2- 172.21.15.114:0/3329229397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bd00ff450 0x7f1bd019c970 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.217+0000 7f1bd7699700 1 --2- 172.21.15.114:0/3329229397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1bd0101dc0 0x7f1bd019ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.217+0000 7f1bd7699700 1 -- 172.21.15.114:0/3329229397 >> 172.21.15.114:0/3329229397 conn(0x7f1bd00fb050 msgr2=0x7f1bd00fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:56.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.217+0000 7f1bd7699700 1 -- 172.21.15.114:0/3329229397 shutdown_connections 2024-02-19T19:06:56.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.217+0000 7f1bd7699700 1 -- 172.21.15.114:0/3329229397 wait complete. 2024-02-19T19:06:56.248 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-02-19T19:06:56.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.322+0000 7f9aba90b700 1 -- 172.21.15.114:0/3128993963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab40ff200 msgr2=0x7f9ab40ff620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:56.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.322+0000 7f9aba90b700 1 --2- 172.21.15.114:0/3128993963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab40ff200 0x7f9ab40ff620 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009910 tx=0x7f9aa4009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:56.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.322+0000 7f9aba90b700 1 -- 172.21.15.114:0/3128993963 shutdown_connections 2024-02-19T19:06:56.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.322+0000 7f9aba90b700 1 --2- 172.21.15.114:0/3128993963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ab4100400 0x7f9ab4100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.322+0000 7f9aba90b700 1 --2- 172.21.15.114:0/3128993963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab40ff200 0x7f9ab40ff620 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.322+0000 7f9aba90b700 1 -- 172.21.15.114:0/3128993963 >> 172.21.15.114:0/3128993963 conn(0x7f9ab40fa780 msgr2=0x7f9ab40fcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:56.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.322+0000 7f9aba90b700 1 -- 172.21.15.114:0/3128993963 shutdown_connections 2024-02-19T19:06:56.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.322+0000 7f9aba90b700 1 -- 172.21.15.114:0/3128993963 wait complete. 2024-02-19T19:06:56.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.322+0000 7f9aba90b700 1 Processor -- start 2024-02-19T19:06:56.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.322+0000 7f9aba90b700 1 -- start start 2024-02-19T19:06:56.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.323+0000 7f9aba90b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ab40ff200 0x7f9ab4102180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:56.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.323+0000 7f9aba90b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab4100400 0x7f9ab419b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:56.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.323+0000 7f9aba90b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab419b650 con 0x7f9ab4100400 2024-02-19T19:06:56.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.323+0000 7f9aba90b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab419b790 con 0x7f9ab40ff200 2024-02-19T19:06:56.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.323+0000 7f9ab37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab4100400 0x7f9ab419b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:56.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.323+0000 7f9ab37fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab4100400 0x7f9ab419b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45676/0 (socket says 172.21.15.114:45676) 2024-02-19T19:06:56.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.323+0000 7f9ab37fe700 1 -- 172.21.15.114:0/1474087813 learned_addr learned my addr 172.21.15.114:0/1474087813 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:06:56.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.323+0000 7f9ab3fff700 1 --2- 172.21.15.114:0/1474087813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ab40ff200 0x7f9ab4102180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:56.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.323+0000 7f9ab37fe700 1 -- 172.21.15.114:0/1474087813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ab40ff200 msgr2=0x7f9ab4102180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:56.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.323+0000 7f9ab37fe700 1 --2- 172.21.15.114:0/1474087813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ab40ff200 0x7f9ab4102180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.323+0000 7f9ab37fe700 1 -- 172.21.15.114:0/1474087813 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa40095f0 con 0x7f9ab4100400 2024-02-19T19:06:56.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.323+0000 7f9ab37fe700 1 --2- 172.21.15.114:0/1474087813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab4100400 0x7f9ab419b080 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9aa800d8d0 tx=0x7f9aa800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:56.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.323+0000 7f9ab17fa700 1 -- 172.21.15.114:0/1474087813 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9aa80098a0 con 0x7f9ab4100400 2024-02-19T19:06:56.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.323+0000 7f9aba90b700 1 -- 172.21.15.114:0/1474087813 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab419ba70 con 0x7f9ab4100400 2024-02-19T19:06:56.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.323+0000 7f9ab17fa700 1 -- 172.21.15.114:0/1474087813 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9aa8010460 con 0x7f9ab4100400 2024-02-19T19:06:56.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.323+0000 7f9ab17fa700 1 -- 172.21.15.114:0/1474087813 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9aa800f5f0 con 0x7f9ab4100400 2024-02-19T19:06:56.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.324+0000 7f9aba90b700 1 -- 172.21.15.114:0/1474087813 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab419bfc0 con 0x7f9ab4100400 2024-02-19T19:06:56.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.325+0000 7f9aba90b700 1 -- 172.21.15.114:0/1474087813 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab4107d60 con 0x7f9ab4100400 2024-02-19T19:06:56.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.325+0000 7f9ab17fa700 1 -- 172.21.15.114:0/1474087813 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f9aa8009a00 con 0x7f9ab4100400 2024-02-19T19:06:56.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.325+0000 7f9ab17fa700 1 --2- 172.21.15.114:0/1474087813 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9a9c06d8a0 0x7f9a9c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:56.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.326+0000 7f9ab17fa700 1 -- 172.21.15.114:0/1474087813 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f9aa808d470 con 0x7f9ab4100400 2024-02-19T19:06:56.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.326+0000 7f9ab3fff700 1 --2- 172.21.15.114:0/1474087813 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9a9c06d8a0 0x7f9a9c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:56.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.326+0000 7f9ab3fff700 1 --2- 172.21.15.114:0/1474087813 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9a9c06d8a0 0x7f9a9c06fd60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4006010 tx=0x7f9aa400b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:56.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.330+0000 7f9ab17fa700 1 -- 172.21.15.114:0/1474087813 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9aa805a680 con 0x7f9ab4100400 2024-02-19T19:06:56.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.541+0000 7f9aba90b700 1 -- 172.21.15.114:0/1474087813 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ab4104740 con 0x7f9a9c06d8a0 2024-02-19T19:06:56.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.543+0000 7f9ab17fa700 1 -- 172.21.15.114:0/1474087813 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f9ab4104740 con 0x7f9a9c06d8a0 2024-02-19T19:06:56.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.545+0000 7f9aba90b700 1 -- 172.21.15.114:0/1474087813 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9a9c06d8a0 msgr2=0x7f9a9c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:56.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.545+0000 7f9aba90b700 1 --2- 172.21.15.114:0/1474087813 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9a9c06d8a0 0x7f9a9c06fd60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4006010 tx=0x7f9aa400b540 comp rx=0 tx=0).stop 2024-02-19T19:06:56.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.545+0000 7f9aba90b700 1 -- 172.21.15.114:0/1474087813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab4100400 msgr2=0x7f9ab419b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:56.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.545+0000 7f9aba90b700 1 --2- 172.21.15.114:0/1474087813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab4100400 0x7f9ab419b080 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9aa800d8d0 tx=0x7f9aa800dbe0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.545+0000 7f9aba90b700 1 -- 172.21.15.114:0/1474087813 shutdown_connections 2024-02-19T19:06:56.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.545+0000 7f9aba90b700 1 --2- 172.21.15.114:0/1474087813 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9a9c06d8a0 0x7f9a9c06fd60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.545+0000 7f9aba90b700 1 --2- 172.21.15.114:0/1474087813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ab40ff200 0x7f9ab4102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.545+0000 7f9aba90b700 1 --2- 172.21.15.114:0/1474087813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ab4100400 0x7f9ab419b080 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.545+0000 7f9aba90b700 1 -- 172.21.15.114:0/1474087813 >> 172.21.15.114:0/1474087813 conn(0x7f9ab40fa780 msgr2=0x7f9ab40fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:56.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.545+0000 7f9aba90b700 1 -- 172.21.15.114:0/1474087813 shutdown_connections 2024-02-19T19:06:56.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.545+0000 7f9aba90b700 1 -- 172.21.15.114:0/1474087813 wait complete. 2024-02-19T19:06:56.645 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:56 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3[97372]: 2024-02-19T19:06:56.239+0000 7f103ad497c0 -1 Falling back to public interface 2024-02-19T19:06:56.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.648+0000 7febdbeca700 1 -- 172.21.15.114:0/2875836092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febd4100be0 msgr2=0x7febd4101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:56.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.648+0000 7febdbeca700 1 --2- 172.21.15.114:0/2875836092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febd4100be0 0x7febd4101000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7febc4009910 tx=0x7febc4009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:56.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.648+0000 7febdbeca700 1 -- 172.21.15.114:0/2875836092 shutdown_connections 2024-02-19T19:06:56.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.648+0000 7febdbeca700 1 --2- 172.21.15.114:0/2875836092 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febd4101de0 0x7febd4102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.648+0000 7febdbeca700 1 --2- 172.21.15.114:0/2875836092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febd4100be0 0x7febd4101000 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.648+0000 7febdbeca700 1 -- 172.21.15.114:0/2875836092 >> 172.21.15.114:0/2875836092 conn(0x7febd40fc140 msgr2=0x7febd40fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:56.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.649+0000 7febdbeca700 1 -- 172.21.15.114:0/2875836092 shutdown_connections 2024-02-19T19:06:56.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.649+0000 7febdbeca700 1 -- 172.21.15.114:0/2875836092 wait complete. 2024-02-19T19:06:56.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.649+0000 7febdbeca700 1 Processor -- start 2024-02-19T19:06:56.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.649+0000 7febdbeca700 1 -- start start 2024-02-19T19:06:56.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.649+0000 7febdbeca700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febd4100be0 0x7febd4196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:56.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.649+0000 7febdbeca700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febd4101de0 0x7febd41968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:56.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.649+0000 7febdbeca700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd4196ef0 con 0x7febd4100be0 2024-02-19T19:06:56.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.649+0000 7febdbeca700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd4197030 con 0x7febd4101de0 2024-02-19T19:06:56.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.649+0000 7febd9c66700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febd4100be0 0x7febd4196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:56.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.649+0000 7febd9c66700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febd4100be0 0x7febd4196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45682/0 (socket says 172.21.15.114:45682) 2024-02-19T19:06:56.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.649+0000 7febd9c66700 1 -- 172.21.15.114:0/3277720627 learned_addr learned my addr 172.21.15.114:0/3277720627 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:06:56.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.649+0000 7febd9465700 1 --2- 172.21.15.114:0/3277720627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febd4101de0 0x7febd41968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:56.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.650+0000 7febd9c66700 1 -- 172.21.15.114:0/3277720627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febd4101de0 msgr2=0x7febd41968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:56.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.650+0000 7febd9c66700 1 --2- 172.21.15.114:0/3277720627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febd4101de0 0x7febd41968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.650+0000 7febd9c66700 1 -- 172.21.15.114:0/3277720627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc40095f0 con 0x7febd4100be0 2024-02-19T19:06:56.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.650+0000 7febd9c66700 1 --2- 172.21.15.114:0/3277720627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febd4100be0 0x7febd4196390 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7febc4000c00 tx=0x7febc400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:56.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.650+0000 7febcaffd700 1 -- 172.21.15.114:0/3277720627 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febc401d070 con 0x7febd4100be0 2024-02-19T19:06:56.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.650+0000 7febcaffd700 1 -- 172.21.15.114:0/3277720627 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7febc4005dc0 con 0x7febd4100be0 2024-02-19T19:06:56.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.650+0000 7febdbeca700 1 -- 172.21.15.114:0/3277720627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd419ba80 con 0x7febd4100be0 2024-02-19T19:06:56.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.650+0000 7febcaffd700 1 -- 172.21.15.114:0/3277720627 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febc4022470 con 0x7febd4100be0 2024-02-19T19:06:56.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.650+0000 7febdbeca700 1 -- 172.21.15.114:0/3277720627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd419bf70 con 0x7febd4100be0 2024-02-19T19:06:56.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.651+0000 7febdbeca700 1 -- 172.21.15.114:0/3277720627 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd41098d0 con 0x7febd4100be0 2024-02-19T19:06:56.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.652+0000 7febcaffd700 1 -- 172.21.15.114:0/3277720627 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7febc4003810 con 0x7febd4100be0 2024-02-19T19:06:56.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.652+0000 7febcaffd700 1 --2- 172.21.15.114:0/3277720627 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7febc006d8a0 0x7febc006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:56.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.652+0000 7febcaffd700 1 -- 172.21.15.114:0/3277720627 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6790+0+0 (secure 0 0 0) 0x7febc408eb70 con 0x7febd4100be0 2024-02-19T19:06:56.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.652+0000 7febd9465700 1 --2- 172.21.15.114:0/3277720627 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7febc006d8a0 0x7febc006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:56.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.653+0000 7febd9465700 1 --2- 172.21.15.114:0/3277720627 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7febc006d8a0 0x7febc006fd60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7febd0009c60 tx=0x7febd0009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:56.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.657+0000 7febcaffd700 1 -- 172.21.15.114:0/3277720627 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7febc405bd80 con 0x7febd4100be0 2024-02-19T19:06:56.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.866+0000 7febdbeca700 1 -- 172.21.15.114:0/3277720627 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7febd4106730 con 0x7febc006d8a0 2024-02-19T19:06:56.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.876+0000 7febcaffd700 1 -- 172.21.15.114:0/3277720627 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7febd4106730 con 0x7febc006d8a0 2024-02-19T19:06:56.897 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:06:56.897 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (10m) 31s ago 13m 26.5M - 0.25.0 c8568f914cd2 30444b676ca4 2024-02-19T19:06:56.897 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (14m) 31s ago 14m 8472k - 18.2.1 94eadd990e0a 4918cd87e312 2024-02-19T19:06:56.897 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi174 smithi174 running (8m) 2m ago 11m 7847k - 18.2.1 94eadd990e0a 516c34431b93 2024-02-19T19:06:56.897 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (2m) 31s ago 13m 6660k - 19.0.0-1450-gf78a58c0 f081f847070a f10b46e9d636 2024-02-19T19:06:56.897 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi174 smithi174 running (2m) 2m ago 11m 6660k - 19.0.0-1450-gf78a58c0 f081f847070a fb3b17c81bd8 2024-02-19T19:06:56.897 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (10m) 31s ago 12m 85.5M - 9.4.7 954c08fa6188 478bd533e827 2024-02-19T19:06:56.897 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tcwpod smithi114 running (6m) 31s ago 6m 27.5M - 18.2.1 94eadd990e0a 420175777cf4 2024-02-19T19:06:56.897 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.vpoeop smithi114 running (6m) 31s ago 6m 24.6M - 18.2.1 94eadd990e0a f81a79e53ead 2024-02-19T19:06:56.898 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.stnfnf smithi174 running (6m) 2m ago 6m 26.0M - 18.2.1 94eadd990e0a 3df35954a492 2024-02-19T19:06:56.898 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.znhuvd smithi174 running (6m) 2m ago 6m 24.2M - 18.2.1 94eadd990e0a 2d2267494ae1 2024-02-19T19:06:56.898 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.pcvtrk smithi114 *:8443,9283,8765 running (4m) 31s ago 15m 503M - 19.0.0-1450-gf78a58c0 f081f847070a 05b5c06122b1 2024-02-19T19:06:56.898 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi174.lydvnj smithi174 *:8443,9283,8765 running (3m) 2m ago 11m 450M - 19.0.0-1450-gf78a58c0 f081f847070a f962b1af5aed 2024-02-19T19:06:56.898 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (3m) 31s ago 15m 48.1M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e51f650f9be9 2024-02-19T19:06:56.898 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi174 smithi174 running (3m) 2m ago 11m 33.9M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e811581f05fb 2024-02-19T19:06:56.898 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (13m) 31s ago 13m 14.3M - 1.5.0 0da6a335fe13 b6b258074cce 2024-02-19T19:06:56.898 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi174 smithi174 *:9100 running (11m) 2m ago 11m 14.9M - 1.5.0 0da6a335fe13 d0b4fdf67b22 2024-02-19T19:06:56.898 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (110s) 31s ago 10m 51.2M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 764769439e0e 2024-02-19T19:06:56.898 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (73s) 31s ago 10m 51.8M 1216M 19.0.0-1450-gf78a58c0 f081f847070a c1c32dbdf573 2024-02-19T19:06:56.898 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (37s) 31s ago 9m 11.5M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 11e64d20e378 2024-02-19T19:06:56.898 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi174 starting - - - 2240M 2024-02-19T19:06:56.898 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi174 running (8m) 2m ago 8m 49.7M 2240M 18.2.1 94eadd990e0a 8b8c3745b8c5 2024-02-19T19:06:56.899 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi174 running (8m) 2m ago 8m 52.2M 2240M 18.2.1 94eadd990e0a 5771a995ed75 2024-02-19T19:06:56.899 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (3m) 31s ago 12m 63.5M - 2.43.0 a07b618ecd1d a36922c7c9a1 2024-02-19T19:06:56.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.878+0000 7febdbeca700 1 -- 172.21.15.114:0/3277720627 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7febc006d8a0 msgr2=0x7febc006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:56.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.878+0000 7febdbeca700 1 --2- 172.21.15.114:0/3277720627 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7febc006d8a0 0x7febc006fd60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7febd0009c60 tx=0x7febd0009260 comp rx=0 tx=0).stop 2024-02-19T19:06:56.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.878+0000 7febdbeca700 1 -- 172.21.15.114:0/3277720627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febd4100be0 msgr2=0x7febd4196390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:56.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.878+0000 7febdbeca700 1 --2- 172.21.15.114:0/3277720627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febd4100be0 0x7febd4196390 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7febc4000c00 tx=0x7febc400f690 comp rx=0 tx=0).stop 2024-02-19T19:06:56.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.878+0000 7febdbeca700 1 -- 172.21.15.114:0/3277720627 shutdown_connections 2024-02-19T19:06:56.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.878+0000 7febdbeca700 1 --2- 172.21.15.114:0/3277720627 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7febc006d8a0 0x7febc006fd60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.878+0000 7febdbeca700 1 --2- 172.21.15.114:0/3277720627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febd4100be0 0x7febd4196390 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.878+0000 7febdbeca700 1 --2- 172.21.15.114:0/3277720627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febd4101de0 0x7febd41968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:56.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.878+0000 7febdbeca700 1 -- 172.21.15.114:0/3277720627 >> 172.21.15.114:0/3277720627 conn(0x7febd40fc140 msgr2=0x7febd4105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:56.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.878+0000 7febdbeca700 1 -- 172.21.15.114:0/3277720627 shutdown_connections 2024-02-19T19:06:56.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.878+0000 7febdbeca700 1 -- 172.21.15.114:0/3277720627 wait complete. 2024-02-19T19:06:57.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.981+0000 7f5519efa700 1 -- 172.21.15.114:0/3797353933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514103f70 msgr2=0x7f5514106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:57.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.981+0000 7f5519efa700 1 --2- 172.21.15.114:0/3797353933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514103f70 0x7f5514106360 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5508009910 tx=0x7f5508009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:57.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.981+0000 7f5519efa700 1 -- 172.21.15.114:0/3797353933 shutdown_connections 2024-02-19T19:06:57.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.981+0000 7f5519efa700 1 --2- 172.21.15.114:0/3797353933 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514103f70 0x7f5514106360 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.981+0000 7f5519efa700 1 --2- 172.21.15.114:0/3797353933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5514101640 0x7f5514103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.981+0000 7f5519efa700 1 -- 172.21.15.114:0/3797353933 >> 172.21.15.114:0/3797353933 conn(0x7f55140fb050 msgr2=0x7f55140fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:57.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.981+0000 7f5519efa700 1 -- 172.21.15.114:0/3797353933 shutdown_connections 2024-02-19T19:06:57.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.981+0000 7f5519efa700 1 -- 172.21.15.114:0/3797353933 wait complete. 2024-02-19T19:06:57.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.982+0000 7f5519efa700 1 Processor -- start 2024-02-19T19:06:57.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.982+0000 7f5519efa700 1 -- start start 2024-02-19T19:06:57.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.982+0000 7f5519efa700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5514101640 0x7f55141963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:57.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.982+0000 7f5519efa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514103f70 0x7f55141968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:57.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.982+0000 7f5519efa700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5514196f10 con 0x7f5514103f70 2024-02-19T19:06:57.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.982+0000 7f5519efa700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5514197050 con 0x7f5514101640 2024-02-19T19:06:57.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.982+0000 7f5512ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514103f70 0x7f55141968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:57.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.983+0000 7f5512ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514103f70 0x7f55141968f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45688/0 (socket says 172.21.15.114:45688) 2024-02-19T19:06:57.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.983+0000 7f5512ffd700 1 -- 172.21.15.114:0/170171228 learned_addr learned my addr 172.21.15.114:0/170171228 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:06:57.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.983+0000 7f55137fe700 1 --2- 172.21.15.114:0/170171228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5514101640 0x7f55141963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:57.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.983+0000 7f5512ffd700 1 -- 172.21.15.114:0/170171228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5514101640 msgr2=0x7f55141963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:57.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.983+0000 7f5512ffd700 1 --2- 172.21.15.114:0/170171228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5514101640 0x7f55141963b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.983+0000 7f5512ffd700 1 -- 172.21.15.114:0/170171228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55080095f0 con 0x7f5514103f70 2024-02-19T19:06:57.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.983+0000 7f5512ffd700 1 --2- 172.21.15.114:0/170171228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514103f70 0x7f55141968f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5508006010 tx=0x7f550800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:57.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.983+0000 7f5510ff9700 1 -- 172.21.15.114:0/170171228 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f550801d070 con 0x7f5514103f70 2024-02-19T19:06:57.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.983+0000 7f5510ff9700 1 -- 172.21.15.114:0/170171228 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f550800f460 con 0x7f5514103f70 2024-02-19T19:06:57.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.983+0000 7f5519efa700 1 -- 172.21.15.114:0/170171228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551419baa0 con 0x7f5514103f70 2024-02-19T19:06:57.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.983+0000 7f5510ff9700 1 -- 172.21.15.114:0/170171228 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5508021640 con 0x7f5514103f70 2024-02-19T19:06:57.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.983+0000 7f5519efa700 1 -- 172.21.15.114:0/170171228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f551419c010 con 0x7f5514103f70 2024-02-19T19:06:57.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.984+0000 7f5519efa700 1 -- 172.21.15.114:0/170171228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55141905a0 con 0x7f5514103f70 2024-02-19T19:06:57.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.985+0000 7f5510ff9700 1 -- 172.21.15.114:0/170171228 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f5508003a60 con 0x7f5514103f70 2024-02-19T19:06:57.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.985+0000 7f5510ff9700 1 --2- 172.21.15.114:0/170171228 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f54fc06d8a0 0x7f54fc06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:57.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.985+0000 7f5510ff9700 1 -- 172.21.15.114:0/170171228 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f550808ea20 con 0x7f5514103f70 2024-02-19T19:06:57.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.985+0000 7f55137fe700 1 --2- 172.21.15.114:0/170171228 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f54fc06d8a0 0x7f54fc06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:57.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.985+0000 7f55137fe700 1 --2- 172.21.15.114:0/170171228 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f54fc06d8a0 0x7f54fc06fd60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5504009a10 tx=0x7f5504009190 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:57.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:56.990+0000 7f5510ff9700 1 -- 172.21.15.114:0/170171228 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f550805bc30 con 0x7f5514103f70 2024-02-19T19:06:57.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:57.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:56 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:06:57.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:56 smithi174 ceph-mon[87279]: from='client.34308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:57.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:56 smithi174 ceph-mon[87279]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:57.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.267+0000 7f5519efa700 1 -- 172.21.15.114:0/170171228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f55140619a0 con 0x7f5514103f70 2024-02-19T19:06:57.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:06:57.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:56 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:06:57.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:56 smithi114 ceph-mon[92983]: from='client.34308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:57.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:56 smithi114 ceph-mon[92983]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:57.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.268+0000 7f5510ff9700 1 -- 172.21.15.114:0/170171228 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f550805b7c0 con 0x7f5514103f70 2024-02-19T19:06:57.289 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:06:57.289 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-02-19T19:06:57.289 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:06:57.289 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:06:57.290 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-02-19T19:06:57.290 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:06:57.290 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:06:57.290 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-02-19T19:06:57.290 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2, 2024-02-19T19:06:57.290 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 3 2024-02-19T19:06:57.290 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:06:57.290 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-02-19T19:06:57.290 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:06:57.290 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:06:57.291 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-02-19T19:06:57.291 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6, 2024-02-19T19:06:57.291 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 7 2024-02-19T19:06:57.291 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-02-19T19:06:57.291 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:06:57.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.270+0000 7f5519efa700 1 -- 172.21.15.114:0/170171228 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f54fc06d8a0 msgr2=0x7f54fc06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:57.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.270+0000 7f5519efa700 1 --2- 172.21.15.114:0/170171228 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f54fc06d8a0 0x7f54fc06fd60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5504009a10 tx=0x7f5504009190 comp rx=0 tx=0).stop 2024-02-19T19:06:57.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.270+0000 7f5519efa700 1 -- 172.21.15.114:0/170171228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514103f70 msgr2=0x7f55141968f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:57.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.270+0000 7f5519efa700 1 --2- 172.21.15.114:0/170171228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514103f70 0x7f55141968f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5508006010 tx=0x7f550800bd70 comp rx=0 tx=0).stop 2024-02-19T19:06:57.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.270+0000 7f5519efa700 1 -- 172.21.15.114:0/170171228 shutdown_connections 2024-02-19T19:06:57.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.270+0000 7f5519efa700 1 --2- 172.21.15.114:0/170171228 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f54fc06d8a0 0x7f54fc06fd60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.270+0000 7f5519efa700 1 --2- 172.21.15.114:0/170171228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5514101640 0x7f55141963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.270+0000 7f5519efa700 1 --2- 172.21.15.114:0/170171228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5514103f70 0x7f55141968f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.270+0000 7f5519efa700 1 -- 172.21.15.114:0/170171228 >> 172.21.15.114:0/170171228 conn(0x7f55140fb050 msgr2=0x7f55140fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:57.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.270+0000 7f5519efa700 1 -- 172.21.15.114:0/170171228 shutdown_connections 2024-02-19T19:06:57.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.270+0000 7f5519efa700 1 -- 172.21.15.114:0/170171228 wait complete. 2024-02-19T19:06:57.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.373+0000 7fc942e7b700 1 -- 172.21.15.114:0/365544528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc93c101dc0 msgr2=0x7fc93c102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:57.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.373+0000 7fc942e7b700 1 --2- 172.21.15.114:0/365544528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc93c101dc0 0x7fc93c102220 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc928009910 tx=0x7fc928009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:57.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.373+0000 7fc942e7b700 1 -- 172.21.15.114:0/365544528 shutdown_connections 2024-02-19T19:06:57.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.373+0000 7fc942e7b700 1 --2- 172.21.15.114:0/365544528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc93c101dc0 0x7fc93c102220 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.373+0000 7fc942e7b700 1 --2- 172.21.15.114:0/365544528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc93c100bc0 0x7fc93c100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.373+0000 7fc942e7b700 1 -- 172.21.15.114:0/365544528 >> 172.21.15.114:0/365544528 conn(0x7fc93c0fc140 msgr2=0x7fc93c0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:57.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.374+0000 7fc942e7b700 1 -- 172.21.15.114:0/365544528 shutdown_connections 2024-02-19T19:06:57.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.374+0000 7fc942e7b700 1 -- 172.21.15.114:0/365544528 wait complete. 2024-02-19T19:06:57.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.374+0000 7fc942e7b700 1 Processor -- start 2024-02-19T19:06:57.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.374+0000 7fc942e7b700 1 -- start start 2024-02-19T19:06:57.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.374+0000 7fc942e7b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc93c100bc0 0x7fc93c1963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:57.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.374+0000 7fc942e7b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc93c101dc0 0x7fc93c196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:57.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.374+0000 7fc942e7b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c196f40 con 0x7fc93c100bc0 2024-02-19T19:06:57.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.374+0000 7fc942e7b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c197080 con 0x7fc93c101dc0 2024-02-19T19:06:57.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.374+0000 7fc940c17700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc93c100bc0 0x7fc93c1963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:57.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.374+0000 7fc940c17700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc93c100bc0 0x7fc93c1963e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45706/0 (socket says 172.21.15.114:45706) 2024-02-19T19:06:57.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.374+0000 7fc940c17700 1 -- 172.21.15.114:0/1169117593 learned_addr learned my addr 172.21.15.114:0/1169117593 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:06:57.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.375+0000 7fc940c17700 1 -- 172.21.15.114:0/1169117593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc93c101dc0 msgr2=0x7fc93c196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:57.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.375+0000 7fc93bfff700 1 --2- 172.21.15.114:0/1169117593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc93c101dc0 0x7fc93c196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:57.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.375+0000 7fc940c17700 1 --2- 172.21.15.114:0/1169117593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc93c101dc0 0x7fc93c196920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.375+0000 7fc940c17700 1 -- 172.21.15.114:0/1169117593 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9280095f0 con 0x7fc93c100bc0 2024-02-19T19:06:57.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.375+0000 7fc940c17700 1 --2- 172.21.15.114:0/1169117593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc93c100bc0 0x7fc93c1963e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc93000b480 tx=0x7fc93000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:57.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.375+0000 7fc939ffb700 1 -- 172.21.15.114:0/1169117593 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9300118c0 con 0x7fc93c100bc0 2024-02-19T19:06:57.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.375+0000 7fc939ffb700 1 -- 172.21.15.114:0/1169117593 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc930010380 con 0x7fc93c100bc0 2024-02-19T19:06:57.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.375+0000 7fc942e7b700 1 -- 172.21.15.114:0/1169117593 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc93c19bb30 con 0x7fc93c100bc0 2024-02-19T19:06:57.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.375+0000 7fc939ffb700 1 -- 172.21.15.114:0/1169117593 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc93000f4f0 con 0x7fc93c100bc0 2024-02-19T19:06:57.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.375+0000 7fc942e7b700 1 -- 172.21.15.114:0/1169117593 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc93c19c080 con 0x7fc93c100bc0 2024-02-19T19:06:57.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.376+0000 7fc942e7b700 1 -- 172.21.15.114:0/1169117593 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc93c1097c0 con 0x7fc93c100bc0 2024-02-19T19:06:57.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.377+0000 7fc939ffb700 1 -- 172.21.15.114:0/1169117593 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7fc9300104f0 con 0x7fc93c100bc0 2024-02-19T19:06:57.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.377+0000 7fc939ffb700 1 --2- 172.21.15.114:0/1169117593 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc92c06d8a0 0x7fc92c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:57.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.377+0000 7fc939ffb700 1 -- 172.21.15.114:0/1169117593 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6790+0+0 (secure 0 0 0) 0x7fc93008d270 con 0x7fc93c100bc0 2024-02-19T19:06:57.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.377+0000 7fc93bfff700 1 --2- 172.21.15.114:0/1169117593 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc92c06d8a0 0x7fc92c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:57.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.377+0000 7fc93bfff700 1 --2- 172.21.15.114:0/1169117593 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc92c06d8a0 0x7fc92c06fd60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc92800b5c0 tx=0x7fc928005960 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:57.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.382+0000 7fc939ffb700 1 -- 172.21.15.114:0/1169117593 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc93005a480 con 0x7fc93c100bc0 2024-02-19T19:06:57.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.625+0000 7fc942e7b700 1 -- 172.21.15.114:0/1169117593 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc93c109980 con 0x7fc93c100bc0 2024-02-19T19:06:57.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.625+0000 7fc939ffb700 1 -- 172.21.15.114:0/1169117593 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fc93005a010 con 0x7fc93c100bc0 2024-02-19T19:06:57.646 INFO:teuthology.orchestra.run.smithi114.stdout:e18 2024-02-19T19:06:57.646 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:06:57.646 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:06:57.646 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-02-19T19:06:57.647 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:06:57.647 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:06:57.647 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-02-19T19:06:57.647 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 18 2024-02-19T19:06:57.647 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:06:57.647 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:06:57.647 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-02-19T19:00:21.203922+0000 2024-02-19T19:06:57.647 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-02-19T19:06:57.647 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-02-19T19:06:57.647 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-02-19T19:06:57.647 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-02-19T19:06:57.647 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-02-19T19:06:57.648 INFO:teuthology.orchestra.run.smithi114.stdout:max_xattr_size 65536 2024-02-19T19:06:57.648 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-02-19T19:06:57.648 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-02-19T19:06:57.648 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 47 2024-02-19T19:06:57.648 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:06:57.648 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 2 2024-02-19T19:06:57.648 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1 2024-02-19T19:06:57.648 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=14530,1=24269} 2024-02-19T19:06:57.648 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-02-19T19:06:57.648 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-02-19T19:06:57.648 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-02-19T19:06:57.648 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-02-19T19:06:57.649 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-02-19T19:06:57.649 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data enabled 2024-02-19T19:06:57.649 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-02-19T19:06:57.649 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-02-19T19:06:57.649 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 1 2024-02-19T19:06:57.649 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:57.649 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.vpoeop{0:14540} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:57.649 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.znhuvd{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:57.649 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.stnfnf{1:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:06:57.649 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:06:57.649 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:06:57.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.627+0000 7fc942e7b700 1 -- 172.21.15.114:0/1169117593 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc92c06d8a0 msgr2=0x7fc92c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:57.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.627+0000 7fc942e7b700 1 --2- 172.21.15.114:0/1169117593 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc92c06d8a0 0x7fc92c06fd60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc92800b5c0 tx=0x7fc928005960 comp rx=0 tx=0).stop 2024-02-19T19:06:57.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.628+0000 7fc942e7b700 1 -- 172.21.15.114:0/1169117593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc93c100bc0 msgr2=0x7fc93c1963e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:57.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.628+0000 7fc942e7b700 1 --2- 172.21.15.114:0/1169117593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc93c100bc0 0x7fc93c1963e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc93000b480 tx=0x7fc93000b840 comp rx=0 tx=0).stop 2024-02-19T19:06:57.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.628+0000 7fc942e7b700 1 -- 172.21.15.114:0/1169117593 shutdown_connections 2024-02-19T19:06:57.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.628+0000 7fc942e7b700 1 --2- 172.21.15.114:0/1169117593 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc92c06d8a0 0x7fc92c06fd60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.628+0000 7fc942e7b700 1 --2- 172.21.15.114:0/1169117593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc93c100bc0 0x7fc93c1963e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.628+0000 7fc942e7b700 1 --2- 172.21.15.114:0/1169117593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc93c101dc0 0x7fc93c196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.628+0000 7fc942e7b700 1 -- 172.21.15.114:0/1169117593 >> 172.21.15.114:0/1169117593 conn(0x7fc93c0fc140 msgr2=0x7fc93c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.628+0000 7fc942e7b700 1 -- 172.21.15.114:0/1169117593 shutdown_connections 2024-02-19T19:06:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.628+0000 7fc942e7b700 1 -- 172.21.15.114:0/1169117593 wait complete. 2024-02-19T19:06:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 18 2024-02-19T19:06:57.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.732+0000 7fba0fb0f700 1 -- 172.21.15.114:0/1498626033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba08103f80 msgr2=0x7fba08106370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:57.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.732+0000 7fba0fb0f700 1 --2- 172.21.15.114:0/1498626033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba08103f80 0x7fba08106370 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8009910 tx=0x7fb9f8009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:57.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.732+0000 7fba0fb0f700 1 -- 172.21.15.114:0/1498626033 shutdown_connections 2024-02-19T19:06:57.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.732+0000 7fba0fb0f700 1 --2- 172.21.15.114:0/1498626033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba08103f80 0x7fba08106370 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.732+0000 7fba0fb0f700 1 --2- 172.21.15.114:0/1498626033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba08101650 0x7fba08103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.732+0000 7fba0fb0f700 1 -- 172.21.15.114:0/1498626033 >> 172.21.15.114:0/1498626033 conn(0x7fba080fb060 msgr2=0x7fba080fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:57.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.732+0000 7fba0fb0f700 1 -- 172.21.15.114:0/1498626033 shutdown_connections 2024-02-19T19:06:57.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.732+0000 7fba0fb0f700 1 -- 172.21.15.114:0/1498626033 wait complete. 2024-02-19T19:06:57.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.733+0000 7fba0fb0f700 1 Processor -- start 2024-02-19T19:06:57.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.733+0000 7fba0fb0f700 1 -- start start 2024-02-19T19:06:57.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.733+0000 7fba0fb0f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba08101650 0x7fba08196350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:57.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.733+0000 7fba0fb0f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba08103f80 0x7fba08196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:57.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.733+0000 7fba0fb0f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba08196e20 con 0x7fba08103f80 2024-02-19T19:06:57.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.733+0000 7fba0fb0f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba08196f60 con 0x7fba08101650 2024-02-19T19:06:57.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.733+0000 7fba0d0aa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba08103f80 0x7fba08196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:57.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.733+0000 7fba0d0aa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba08103f80 0x7fba08196890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45712/0 (socket says 172.21.15.114:45712) 2024-02-19T19:06:57.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.733+0000 7fba0d0aa700 1 -- 172.21.15.114:0/2058956383 learned_addr learned my addr 172.21.15.114:0/2058956383 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:06:57.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.734+0000 7fba0d8ab700 1 --2- 172.21.15.114:0/2058956383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba08101650 0x7fba08196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:57.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.734+0000 7fba0d0aa700 1 -- 172.21.15.114:0/2058956383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba08101650 msgr2=0x7fba08196350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:57.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.734+0000 7fba0d0aa700 1 --2- 172.21.15.114:0/2058956383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba08101650 0x7fba08196350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.734+0000 7fba0d0aa700 1 -- 172.21.15.114:0/2058956383 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f80095f0 con 0x7fba08103f80 2024-02-19T19:06:57.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.734+0000 7fba0d0aa700 1 --2- 172.21.15.114:0/2058956383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba08103f80 0x7fba08196890 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800bd70 tx=0x7fb9f800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:57.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.734+0000 7fb9feffd700 1 -- 172.21.15.114:0/2058956383 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9f801d070 con 0x7fba08103f80 2024-02-19T19:06:57.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.734+0000 7fb9feffd700 1 -- 172.21.15.114:0/2058956383 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb9f8022470 con 0x7fba08103f80 2024-02-19T19:06:57.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.734+0000 7fba0fb0f700 1 -- 172.21.15.114:0/2058956383 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0819b9c0 con 0x7fba08103f80 2024-02-19T19:06:57.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.734+0000 7fb9feffd700 1 -- 172.21.15.114:0/2058956383 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9f800f670 con 0x7fba08103f80 2024-02-19T19:06:57.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.734+0000 7fba0fb0f700 1 -- 172.21.15.114:0/2058956383 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0819be80 con 0x7fba08103f80 2024-02-19T19:06:57.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.735+0000 7fba0fb0f700 1 -- 172.21.15.114:0/2058956383 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba08190530 con 0x7fba08103f80 2024-02-19T19:06:57.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.735+0000 7fb9feffd700 1 -- 172.21.15.114:0/2058956383 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7fb9f8022a50 con 0x7fba08103f80 2024-02-19T19:06:57.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.736+0000 7fb9feffd700 1 --2- 172.21.15.114:0/2058956383 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb9f406d8a0 0x7fb9f406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:57.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.736+0000 7fb9feffd700 1 -- 172.21.15.114:0/2058956383 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6790+0+0 (secure 0 0 0) 0x7fb9f808ed30 con 0x7fba08103f80 2024-02-19T19:06:57.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.736+0000 7fba0d8ab700 1 --2- 172.21.15.114:0/2058956383 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb9f406d8a0 0x7fb9f406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:57.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.736+0000 7fba0d8ab700 1 --2- 172.21.15.114:0/2058956383 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb9f406d8a0 0x7fb9f406fd60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba04005ea0 tx=0x7fba04005e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:57.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.741+0000 7fb9feffd700 1 -- 172.21.15.114:0/2058956383 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb9f805c190 con 0x7fba08103f80 2024-02-19T19:06:57.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.953+0000 7fba0fb0f700 1 -- 172.21.15.114:0/2058956383 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fba080611d0 con 0x7fb9f406d8a0 2024-02-19T19:06:57.975 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:57 smithi114 ceph-mon[92983]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:57.975 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:57 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/170171228' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:06:57.975 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:57 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/1169117593' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:06:57.975 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:57 smithi114 ceph-mon[92983]: pgmap v122: 65 pgs: 12 active+undersized+degraded, 21 active+undersized, 32 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 22/129 objects degraded (17.054%) 2024-02-19T19:06:57.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.954+0000 7fb9feffd700 1 -- 172.21.15.114:0/2058956383 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fba080611d0 con 0x7fb9f406d8a0 2024-02-19T19:06:57.975 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:06:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:06:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-02-19T19:06:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:06:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [ 2024-02-19T19:06:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: "crash", 2024-02-19T19:06:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr", 2024-02-19T19:06:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: "mon" 2024-02-19T19:06:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-02-19T19:06:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "9/23 daemons upgraded", 2024-02-19T19:06:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "Currently upgrading osd daemons", 2024-02-19T19:06:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-02-19T19:06:57.976 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:06:57.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.956+0000 7fba0fb0f700 1 -- 172.21.15.114:0/2058956383 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb9f406d8a0 msgr2=0x7fb9f406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:57.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.956+0000 7fba0fb0f700 1 --2- 172.21.15.114:0/2058956383 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb9f406d8a0 0x7fb9f406fd60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba04005ea0 tx=0x7fba04005e30 comp rx=0 tx=0).stop 2024-02-19T19:06:57.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.956+0000 7fba0fb0f700 1 -- 172.21.15.114:0/2058956383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba08103f80 msgr2=0x7fba08196890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:57.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.956+0000 7fba0fb0f700 1 --2- 172.21.15.114:0/2058956383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba08103f80 0x7fba08196890 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800bd70 tx=0x7fb9f800be50 comp rx=0 tx=0).stop 2024-02-19T19:06:57.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.956+0000 7fba0fb0f700 1 -- 172.21.15.114:0/2058956383 shutdown_connections 2024-02-19T19:06:57.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.956+0000 7fba0fb0f700 1 --2- 172.21.15.114:0/2058956383 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fb9f406d8a0 0x7fb9f406fd60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.956+0000 7fba0fb0f700 1 --2- 172.21.15.114:0/2058956383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba08101650 0x7fba08196350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.956+0000 7fba0fb0f700 1 --2- 172.21.15.114:0/2058956383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fba08103f80 0x7fba08196890 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:57.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.956+0000 7fba0fb0f700 1 -- 172.21.15.114:0/2058956383 >> 172.21.15.114:0/2058956383 conn(0x7fba080fb060 msgr2=0x7fba080fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:57.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.956+0000 7fba0fb0f700 1 -- 172.21.15.114:0/2058956383 shutdown_connections 2024-02-19T19:06:57.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:57.957+0000 7fba0fb0f700 1 -- 172.21.15.114:0/2058956383 wait complete. 2024-02-19T19:06:58.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.060+0000 7fc6f3375700 1 -- 172.21.15.114:0/475091015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6ec100cb0 msgr2=0x7fc6ec101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:58.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.060+0000 7fc6f3375700 1 --2- 172.21.15.114:0/475091015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6ec100cb0 0x7fc6ec101110 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc009910 tx=0x7fc6dc009c20 comp rx=0 tx=0).stop 2024-02-19T19:06:58.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.060+0000 7fc6f3375700 1 -- 172.21.15.114:0/475091015 shutdown_connections 2024-02-19T19:06:58.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.060+0000 7fc6f3375700 1 --2- 172.21.15.114:0/475091015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6ec100cb0 0x7fc6ec101110 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:58.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.060+0000 7fc6f3375700 1 --2- 172.21.15.114:0/475091015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6ec0ffab0 0x7fc6ec0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:58.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.060+0000 7fc6f3375700 1 -- 172.21.15.114:0/475091015 >> 172.21.15.114:0/475091015 conn(0x7fc6ec0fb030 msgr2=0x7fc6ec0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:58.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.060+0000 7fc6f3375700 1 -- 172.21.15.114:0/475091015 shutdown_connections 2024-02-19T19:06:58.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.060+0000 7fc6f3375700 1 -- 172.21.15.114:0/475091015 wait complete. 2024-02-19T19:06:58.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.060+0000 7fc6f3375700 1 Processor -- start 2024-02-19T19:06:58.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.060+0000 7fc6f3375700 1 -- start start 2024-02-19T19:06:58.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.061+0000 7fc6f3375700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6ec0ffab0 0x7fc6ec068f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:58.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.061+0000 7fc6f3375700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6ec100cb0 0x7fc6ec19b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:58.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.061+0000 7fc6f3375700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec19b690 con 0x7fc6ec0ffab0 2024-02-19T19:06:58.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.061+0000 7fc6f3375700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec19b7d0 con 0x7fc6ec100cb0 2024-02-19T19:06:58.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.061+0000 7fc6f1111700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6ec0ffab0 0x7fc6ec068f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:58.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.061+0000 7fc6f1111700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6ec0ffab0 0x7fc6ec068f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45732/0 (socket says 172.21.15.114:45732) 2024-02-19T19:06:58.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.061+0000 7fc6f1111700 1 -- 172.21.15.114:0/4187227449 learned_addr learned my addr 172.21.15.114:0/4187227449 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:06:58.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.061+0000 7fc6f0910700 1 --2- 172.21.15.114:0/4187227449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6ec100cb0 0x7fc6ec19b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:58.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.061+0000 7fc6f1111700 1 -- 172.21.15.114:0/4187227449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6ec100cb0 msgr2=0x7fc6ec19b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:58.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.061+0000 7fc6f1111700 1 --2- 172.21.15.114:0/4187227449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6ec100cb0 0x7fc6ec19b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:58.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.061+0000 7fc6f1111700 1 -- 172.21.15.114:0/4187227449 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc0095f0 con 0x7fc6ec0ffab0 2024-02-19T19:06:58.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.061+0000 7fc6f1111700 1 --2- 172.21.15.114:0/4187227449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6ec0ffab0 0x7fc6ec068f70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc6e800c8f0 tx=0x7fc6e800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:58.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.061+0000 7fc6e27fc700 1 -- 172.21.15.114:0/4187227449 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6e80078e0 con 0x7fc6ec0ffab0 2024-02-19T19:06:58.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.061+0000 7fc6e27fc700 1 -- 172.21.15.114:0/4187227449 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc6e800f450 con 0x7fc6ec0ffab0 2024-02-19T19:06:58.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.061+0000 7fc6f3375700 1 -- 172.21.15.114:0/4187227449 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ec19bab0 con 0x7fc6ec0ffab0 2024-02-19T19:06:58.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.061+0000 7fc6e27fc700 1 -- 172.21.15.114:0/4187227449 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6e8018630 con 0x7fc6ec0ffab0 2024-02-19T19:06:58.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.061+0000 7fc6f3375700 1 -- 172.21.15.114:0/4187227449 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ec19c000 con 0x7fc6ec0ffab0 2024-02-19T19:06:58.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.062+0000 7fc6f3375700 1 -- 172.21.15.114:0/4187227449 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6ec108610 con 0x7fc6ec0ffab0 2024-02-19T19:06:58.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.063+0000 7fc6e27fc700 1 -- 172.21.15.114:0/4187227449 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7fc6e800f5c0 con 0x7fc6ec0ffab0 2024-02-19T19:06:58.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.063+0000 7fc6e27fc700 1 --2- 172.21.15.114:0/4187227449 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc6d806d8a0 0x7fc6d806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:06:58.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.063+0000 7fc6e27fc700 1 -- 172.21.15.114:0/4187227449 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6790+0+0 (secure 0 0 0) 0x7fc6e808d450 con 0x7fc6ec0ffab0 2024-02-19T19:06:58.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.063+0000 7fc6f0910700 1 --2- 172.21.15.114:0/4187227449 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc6d806d8a0 0x7fc6d806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:06:58.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.064+0000 7fc6f0910700 1 --2- 172.21.15.114:0/4187227449 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc6d806d8a0 0x7fc6d806fd60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec0694f0 tx=0x7fc6dc00b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:06:58.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.068+0000 7fc6e27fc700 1 -- 172.21.15.114:0/4187227449 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc6e805a6e0 con 0x7fc6ec0ffab0 2024-02-19T19:06:58.144 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:57 smithi174 ceph-mon[87279]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:58.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:57 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/170171228' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:06:58.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:57 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/1169117593' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:06:58.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:57 smithi174 ceph-mon[87279]: pgmap v122: 65 pgs: 12 active+undersized+degraded, 21 active+undersized, 32 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 22/129 objects degraded (17.054%) 2024-02-19T19:06:58.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.342+0000 7fc6f3375700 1 -- 172.21.15.114:0/4187227449 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc6ec108830 con 0x7fc6ec0ffab0 2024-02-19T19:06:58.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.342+0000 7fc6e27fc700 1 -- 172.21.15.114:0/4187227449 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1128 (secure 0 0 0) 0x7fc6e805a270 con 0x7fc6ec0ffab0 2024-02-19T19:06:58.363 INFO:teuthology.orchestra.run.smithi114.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down; Degraded data redundancy: 22/129 objects degraded (17.054%), 12 pgs degraded 2024-02-19T19:06:58.364 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:06:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:06:58.364 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-19T19:06:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: osd.3 (root=default,host=smithi174) is down 2024-02-19T19:06:58.364 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 22/129 objects degraded (17.054%), 12 pgs degraded 2024-02-19T19:06:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: pg 1.0 is active+undersized+degraded, acting [0,1] 2024-02-19T19:06:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-02-19T19:06:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-02-19T19:06:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.7 is active+undersized+degraded, acting [4,2] 2024-02-19T19:06:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.8 is active+undersized+degraded, acting [5,0] 2024-02-19T19:06:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.a is active+undersized+degraded, acting [1,4] 2024-02-19T19:06:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-02-19T19:06:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.d is active+undersized+degraded, acting [1,2] 2024-02-19T19:06:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.14 is active+undersized+degraded, acting [4,5] 2024-02-19T19:06:58.365 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.15 is active+undersized+degraded, acting [1,5] 2024-02-19T19:06:58.365 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.1d is active+undersized+degraded, acting [5,0] 2024-02-19T19:06:58.365 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.1f is active+undersized+degraded, acting [1,4] 2024-02-19T19:06:58.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.344+0000 7fc6f3375700 1 -- 172.21.15.114:0/4187227449 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc6d806d8a0 msgr2=0x7fc6d806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:58.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.344+0000 7fc6f3375700 1 --2- 172.21.15.114:0/4187227449 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc6d806d8a0 0x7fc6d806fd60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec0694f0 tx=0x7fc6dc00b540 comp rx=0 tx=0).stop 2024-02-19T19:06:58.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.344+0000 7fc6f3375700 1 -- 172.21.15.114:0/4187227449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6ec0ffab0 msgr2=0x7fc6ec068f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:06:58.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.344+0000 7fc6f3375700 1 --2- 172.21.15.114:0/4187227449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6ec0ffab0 0x7fc6ec068f70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc6e800c8f0 tx=0x7fc6e800cc00 comp rx=0 tx=0).stop 2024-02-19T19:06:58.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.345+0000 7fc6f3375700 1 -- 172.21.15.114:0/4187227449 shutdown_connections 2024-02-19T19:06:58.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.345+0000 7fc6f3375700 1 --2- 172.21.15.114:0/4187227449 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc6d806d8a0 0x7fc6d806fd60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:58.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.345+0000 7fc6f3375700 1 --2- 172.21.15.114:0/4187227449 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6ec0ffab0 0x7fc6ec068f70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:58.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.345+0000 7fc6f3375700 1 --2- 172.21.15.114:0/4187227449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6ec100cb0 0x7fc6ec19b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:06:58.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.345+0000 7fc6f3375700 1 -- 172.21.15.114:0/4187227449 >> 172.21.15.114:0/4187227449 conn(0x7fc6ec0fb030 msgr2=0x7fc6ec103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:06:58.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.345+0000 7fc6f3375700 1 -- 172.21.15.114:0/4187227449 shutdown_connections 2024-02-19T19:06:58.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:06:58.345+0000 7fc6f3375700 1 -- 172.21.15.114:0/4187227449 wait complete. 2024-02-19T19:06:59.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:58 smithi174 ceph-mon[87279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-02-19T19:06:59.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:58 smithi174 ceph-mon[87279]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:59.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:58 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/4187227449' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:06:59.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:58 smithi114 ceph-mon[92983]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-02-19T19:06:59.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:58 smithi114 ceph-mon[92983]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:06:59.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:58 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/4187227449' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:07:00.048 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:59 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3[97372]: 2024-02-19T19:06:59.721+0000 7f103ad497c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-02-19T19:07:00.048 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:59 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3[97372]: 2024-02-19T19:06:59.750+0000 7f103ad497c0 -1 osd.3 68 log_to_monitors true 2024-02-19T19:07:00.048 INFO:journalctl@ceph.osd.3.smithi174.stdout:Feb 19 19:06:59 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-3[97372]: 2024-02-19T19:06:59.949+0000 7f102c1a1700 -1 osd.3 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T19:07:00.048 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:59 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:00.048 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:59 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:00.049 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:59 smithi174 ceph-mon[87279]: from='osd.3 [v2:172.21.15.174:6800/790857343,v1:172.21.15.174:6801/790857343]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T19:07:00.049 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:59 smithi174 ceph-mon[87279]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T19:07:00.049 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:06:59 smithi174 ceph-mon[87279]: pgmap v123: 65 pgs: 12 active+undersized+degraded, 21 active+undersized, 32 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 22/129 objects degraded (17.054%) 2024-02-19T19:07:00.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:59 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:00.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:59 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:00.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:59 smithi114 ceph-mon[92983]: from='osd.3 [v2:172.21.15.174:6800/790857343,v1:172.21.15.174:6801/790857343]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T19:07:00.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:59 smithi114 ceph-mon[92983]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T19:07:00.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:06:59 smithi114 ceph-mon[92983]: pgmap v123: 65 pgs: 12 active+undersized+degraded, 21 active+undersized, 32 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 22/129 objects degraded (17.054%) 2024-02-19T19:07:01.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:00 smithi114 ceph-mon[92983]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-19T19:07:01.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:00 smithi114 ceph-mon[92983]: osdmap e71: 6 total, 5 up, 6 in 2024-02-19T19:07:01.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:00 smithi114 ceph-mon[92983]: from='osd.3 [v2:172.21.15.174:6800/790857343,v1:172.21.15.174:6801/790857343]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T19:07:01.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:00 smithi114 ceph-mon[92983]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T19:07:01.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:00 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:01.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:00 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:01.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:00 smithi174 ceph-mon[87279]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-19T19:07:01.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:00 smithi174 ceph-mon[87279]: osdmap e71: 6 total, 5 up, 6 in 2024-02-19T19:07:01.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:00 smithi174 ceph-mon[87279]: from='osd.3 [v2:172.21.15.174:6800/790857343,v1:172.21.15.174:6801/790857343]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T19:07:01.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:00 smithi174 ceph-mon[87279]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T19:07:01.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:00 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:01.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:00 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:02.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:01 smithi114 ceph-mon[92983]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:07:02.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:01 smithi114 ceph-mon[92983]: osd.3 [v2:172.21.15.174:6800/790857343,v1:172.21.15.174:6801/790857343] boot 2024-02-19T19:07:02.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:01 smithi114 ceph-mon[92983]: osdmap e72: 6 total, 6 up, 6 in 2024-02-19T19:07:02.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:01 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:07:02.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:01 smithi114 ceph-mon[92983]: Health check update: Degraded data redundancy: 22/129 objects degraded (17.054%), 12 pgs degraded (PG_DEGRADED) 2024-02-19T19:07:02.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:01 smithi114 ceph-mon[92983]: pgmap v126: 65 pgs: 12 active+undersized+degraded, 21 active+undersized, 32 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 22/129 objects degraded (17.054%) 2024-02-19T19:07:02.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:01 smithi114 ceph-mon[92983]: osdmap e73: 6 total, 6 up, 6 in 2024-02-19T19:07:02.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:01 smithi174 ceph-mon[87279]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:07:02.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:01 smithi174 ceph-mon[87279]: osd.3 [v2:172.21.15.174:6800/790857343,v1:172.21.15.174:6801/790857343] boot 2024-02-19T19:07:02.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:01 smithi174 ceph-mon[87279]: osdmap e72: 6 total, 6 up, 6 in 2024-02-19T19:07:02.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:01 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:07:02.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:01 smithi174 ceph-mon[87279]: Health check update: Degraded data redundancy: 22/129 objects degraded (17.054%), 12 pgs degraded (PG_DEGRADED) 2024-02-19T19:07:02.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:01 smithi174 ceph-mon[87279]: pgmap v126: 65 pgs: 12 active+undersized+degraded, 21 active+undersized, 32 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 22/129 objects degraded (17.054%) 2024-02-19T19:07:02.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:01 smithi174 ceph-mon[87279]: osdmap e73: 6 total, 6 up, 6 in 2024-02-19T19:07:04.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:04.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:04.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:04.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:07:04.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:04.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:07:04.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:04.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:04.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:04.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:04 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-19T19:07:04.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:04 smithi174 ceph-mon[87279]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-19T19:07:04.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:04 smithi174 ceph-mon[87279]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-02-19T19:07:04.646 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:04 smithi174 ceph-mon[87279]: pgmap v128: 65 pgs: 3 peering, 7 active+undersized+degraded, 15 active+undersized, 40 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s; 14/129 objects degraded (10.853%) 2024-02-19T19:07:04.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:04.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:04.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:04.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:07:04.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:04.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:07:04.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:04.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:04.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:04.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:04 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-19T19:07:04.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:04 smithi114 ceph-mon[92983]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-19T19:07:04.791 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:04 smithi114 ceph-mon[92983]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-02-19T19:07:04.791 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:04 smithi114 ceph-mon[92983]: pgmap v128: 65 pgs: 3 peering, 7 active+undersized+degraded, 15 active+undersized, 40 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s; 14/129 objects degraded (10.853%) 2024-02-19T19:07:06.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:05 smithi174 ceph-mon[87279]: pgmap v129: 65 pgs: 3 peering, 62 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-02-19T19:07:06.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:05 smithi174 ceph-mon[87279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/129 objects degraded (10.853%), 7 pgs degraded) 2024-02-19T19:07:06.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:05 smithi114 ceph-mon[92983]: pgmap v129: 65 pgs: 3 peering, 62 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-02-19T19:07:06.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:05 smithi114 ceph-mon[92983]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/129 objects degraded (10.853%), 7 pgs degraded) 2024-02-19T19:07:08.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:07 smithi174 ceph-mon[87279]: pgmap v130: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 3 op/s 2024-02-19T19:07:08.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:07 smithi114 ceph-mon[92983]: pgmap v130: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 3 op/s 2024-02-19T19:07:10.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:09 smithi174 ceph-mon[87279]: pgmap v131: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 926 B/s rd, 2 op/s 2024-02-19T19:07:10.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:09 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:10.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:09 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:07:10.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:09 smithi114 ceph-mon[92983]: pgmap v131: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 926 B/s rd, 2 op/s 2024-02-19T19:07:10.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:09 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:10.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:09 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:07:12.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:11 smithi174 ceph-mon[87279]: pgmap v132: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-02-19T19:07:12.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:11 smithi114 ceph-mon[92983]: pgmap v132: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-02-19T19:07:14.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:13 smithi174 ceph-mon[87279]: pgmap v133: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s 2024-02-19T19:07:14.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:13 smithi114 ceph-mon[92983]: pgmap v133: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s 2024-02-19T19:07:16.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:15 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:16.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:15 smithi174 ceph-mon[87279]: pgmap v134: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-02-19T19:07:16.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:15 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:16.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:15 smithi114 ceph-mon[92983]: pgmap v134: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-02-19T19:07:18.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:17 smithi174 ceph-mon[87279]: pgmap v135: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:07:18.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:17 smithi114 ceph-mon[92983]: pgmap v135: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:07:19.107 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:18 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-19T19:07:19.107 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:18 smithi174 ceph-mon[87279]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-19T19:07:19.107 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:18 smithi174 ceph-mon[87279]: Upgrade: osd.4 is safe to restart 2024-02-19T19:07:19.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:18 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-19T19:07:19.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:18 smithi114 ceph-mon[92983]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-19T19:07:19.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:18 smithi114 ceph-mon[92983]: Upgrade: osd.4 is safe to restart 2024-02-19T19:07:20.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:19 smithi174 ceph-mon[87279]: pgmap v136: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:07:20.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:19 smithi114 ceph-mon[92983]: pgmap v136: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:07:21.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:21 smithi174 ceph-mon[87279]: Upgrade: Updating osd.4 2024-02-19T19:07:21.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:21 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:21.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:21 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-19T19:07:21.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:21 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:21.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:21 smithi174 ceph-mon[87279]: Deploying daemon osd.4 on smithi174 2024-02-19T19:07:21.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:21 smithi114 ceph-mon[92983]: Upgrade: Updating osd.4 2024-02-19T19:07:21.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:21 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:21.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:21 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-19T19:07:21.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:21 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:21.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:21 smithi114 ceph-mon[92983]: Deploying daemon osd.4 on smithi174 2024-02-19T19:07:22.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:22 smithi114 ceph-mon[92983]: pgmap v137: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:07:22.590 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:22 smithi174 ceph-mon[87279]: pgmap v137: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:07:23.645 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:23 smithi174 systemd[1]: Stopping Ceph osd.4 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T19:07:24.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:23 smithi174 ceph-mon[87279]: pgmap v138: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:07:24.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:23 smithi174 ceph-mon[87279]: osd.4 marked itself down and dead 2024-02-19T19:07:24.146 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:23 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4[53547]: 2024-02-19T19:07:23.820+0000 7f9816a23700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T19:07:24.146 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:23 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4[53547]: 2024-02-19T19:07:23.820+0000 7f9816a23700 -1 osd.4 73 *** Got signal Terminated *** 2024-02-19T19:07:24.146 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:23 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4[53547]: 2024-02-19T19:07:23.820+0000 7f9816a23700 -1 osd.4 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T19:07:24.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:23 smithi114 ceph-mon[92983]: pgmap v138: 65 pgs: 65 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:07:24.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:23 smithi114 ceph-mon[92983]: osd.4 marked itself down and dead 2024-02-19T19:07:24.500 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:24 smithi174 podman[101814]: 2024-02-19 19:07:24.208792689 +0000 UTC m=+0.510328366 container died 8b8c3745b8c537df8f3cf28ec1e43d2b53443f967b781a44fa9fd8dffc17a740 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.29.1, RELEASE=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T19:07:24.858 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:24 smithi174 podman[101814]: 2024-02-19 19:07:24.596088904 +0000 UTC m=+0.897624625 container cleanup 8b8c3745b8c537df8f3cf28ec1e43d2b53443f967b781a44fa9fd8dffc17a740 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD) 2024-02-19T19:07:24.859 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:24 smithi174 bash[101814]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4 2024-02-19T19:07:25.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:24 smithi174 ceph-mon[87279]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:07:25.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:24 smithi174 ceph-mon[87279]: osdmap e74: 6 total, 5 up, 6 in 2024-02-19T19:07:25.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:24 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:25.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:24 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:07:25.145 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:24 smithi174 podman[101833]: 2024-02-19 19:07:24.866328907 +0000 UTC m=+0.653075773 container remove 8b8c3745b8c537df8f3cf28ec1e43d2b53443f967b781a44fa9fd8dffc17a740 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) 2024-02-19T19:07:25.191 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:24 smithi114 ceph-mon[92983]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:07:25.191 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:24 smithi114 ceph-mon[92983]: osdmap e74: 6 total, 5 up, 6 in 2024-02-19T19:07:25.191 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:24 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:25.191 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:24 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:07:25.571 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:25 smithi174 podman[101928]: 2024-02-19 19:07:25.115576453 +0000 UTC m=+0.018292420 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:07:25.571 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:25 smithi174 podman[101928]: 2024-02-19 19:07:25.231836955 +0000 UTC m=+0.134552914 container create fa3681f2812abe6a0a4babfb6db2ff3abccb4088f3fee23b0d4aa395fb46494c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T19:07:25.571 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:25 smithi174 podman[101928]: 2024-02-19 19:07:25.451019169 +0000 UTC m=+0.353735129 container init fa3681f2812abe6a0a4babfb6db2ff3abccb4088f3fee23b0d4aa395fb46494c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-19T19:07:25.571 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:25 smithi174 podman[101928]: 2024-02-19 19:07:25.454315394 +0000 UTC m=+0.357031354 container start fa3681f2812abe6a0a4babfb6db2ff3abccb4088f3fee23b0d4aa395fb46494c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-19T19:07:25.572 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:25 smithi174 podman[101928]: 2024-02-19 19:07:25.551076581 +0000 UTC m=+0.453792561 container attach fa3681f2812abe6a0a4babfb6db2ff3abccb4088f3fee23b0d4aa395fb46494c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T19:07:25.895 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:25 smithi174 podman[101928]: 2024-02-19 19:07:25.740791877 +0000 UTC m=+0.643507837 container died fa3681f2812abe6a0a4babfb6db2ff3abccb4088f3fee23b0d4aa395fb46494c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5) 2024-02-19T19:07:26.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:25 smithi114 ceph-mon[92983]: osdmap e75: 6 total, 5 up, 6 in 2024-02-19T19:07:26.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:25 smithi114 ceph-mon[92983]: pgmap v141: 65 pgs: 5 active+undersized+degraded, 2 peering, 3 stale+active+clean, 10 active+undersized, 45 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 2 op/s; 11/129 objects degraded (8.527%) 2024-02-19T19:07:26.358 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:25 smithi174 ceph-mon[87279]: osdmap e75: 6 total, 5 up, 6 in 2024-02-19T19:07:26.359 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:25 smithi174 ceph-mon[87279]: pgmap v141: 65 pgs: 5 active+undersized+degraded, 2 peering, 3 stale+active+clean, 10 active+undersized, 45 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 2 op/s; 11/129 objects degraded (8.527%) 2024-02-19T19:07:26.645 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:26 smithi174 podman[101954]: 2024-02-19 19:07:26.338659512 +0000 UTC m=+0.591918697 container remove fa3681f2812abe6a0a4babfb6db2ff3abccb4088f3fee23b0d4aa395fb46494c (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T19:07:26.645 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:26 smithi174 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.4.service: Deactivated successfully. 2024-02-19T19:07:26.645 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:26 smithi174 systemd[1]: Stopped Ceph osd.4 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T19:07:26.645 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:26 smithi174 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.4.service: Consumed 4.974s CPU time. 2024-02-19T19:07:26.645 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:26 smithi174 systemd[1]: Starting Ceph osd.4 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T19:07:27.055 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:26 smithi174 ceph-mon[87279]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-19T19:07:27.055 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:26 smithi174 ceph-mon[87279]: Health check failed: Degraded data redundancy: 11/129 objects degraded (8.527%), 5 pgs degraded (PG_DEGRADED) 2024-02-19T19:07:27.055 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:26 smithi174 podman[102061]: 2024-02-19 19:07:26.678103226 +0000 UTC m=+0.018525475 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:07:27.055 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:26 smithi174 podman[102061]: 2024-02-19 19:07:26.790243093 +0000 UTC m=+0.130665377 container create 0f9ac4d6a109e0c4ea5c6c87388e98ff16cff1a72b0c193e4211aab2d4600b04 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-activate, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-19T19:07:27.055 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 podman[102061]: 2024-02-19 19:07:27.034634553 +0000 UTC m=+0.375056800 container init 0f9ac4d6a109e0c4ea5c6c87388e98ff16cff1a72b0c193e4211aab2d4600b04 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T19:07:27.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:26 smithi114 ceph-mon[92983]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-19T19:07:27.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:26 smithi114 ceph-mon[92983]: Health check failed: Degraded data redundancy: 11/129 objects degraded (8.527%), 5 pgs degraded (PG_DEGRADED) 2024-02-19T19:07:27.395 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 podman[102061]: 2024-02-19 19:07:27.038543764 +0000 UTC m=+0.378966010 container start 0f9ac4d6a109e0c4ea5c6c87388e98ff16cff1a72b0c193e4211aab2d4600b04 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T19:07:27.395 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 podman[102061]: 2024-02-19 19:07:27.126214602 +0000 UTC m=+0.466636852 container attach 0f9ac4d6a109e0c4ea5c6c87388e98ff16cff1a72b0c193e4211aab2d4600b04 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-activate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:07:27.858 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-activate[102076]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-19T19:07:27.858 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-activate[102076]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-19T19:07:27.858 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 bash[102061]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-19T19:07:27.858 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 bash[102061]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-19T19:07:27.858 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 bash[102061]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-19T19:07:27.858 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 bash[102061]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-19T19:07:27.858 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 bash[102061]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-02-19T19:07:27.859 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 bash[102061]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-19T19:07:27.859 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 bash[102061]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-19T19:07:27.859 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-activate[102076]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-19T19:07:27.859 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-activate[102076]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-19T19:07:27.859 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-activate[102076]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-02-19T19:07:27.859 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-activate[102076]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-19T19:07:27.859 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-activate[102076]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-19T19:07:27.859 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:27 smithi174 podman[102061]: 2024-02-19 19:07:27.504442549 +0000 UTC m=+0.844864801 container died 0f9ac4d6a109e0c4ea5c6c87388e98ff16cff1a72b0c193e4211aab2d4600b04 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-activate, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-02-19T19:07:28.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:27 smithi174 ceph-mon[87279]: pgmap v142: 65 pgs: 6 active+undersized+degraded, 2 peering, 1 stale+active+clean, 15 active+undersized, 41 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 1 op/s; 12/129 objects degraded (9.302%) 2024-02-19T19:07:28.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:27 smithi114 ceph-mon[92983]: pgmap v142: 65 pgs: 6 active+undersized+degraded, 2 peering, 1 stale+active+clean, 15 active+undersized, 41 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 1 op/s; 12/129 objects degraded (9.302%) 2024-02-19T19:07:28.466 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:28 smithi174 podman[102212]: 2024-02-19 19:07:28.192438573 +0000 UTC m=+0.681297538 container remove 0f9ac4d6a109e0c4ea5c6c87388e98ff16cff1a72b0c193e4211aab2d4600b04 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5) 2024-02-19T19:07:28.510 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:28 smithi174 podman[102273]: 2024-02-19 19:07:28.306487657 +0000 UTC m=+0.017701792 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:07:28.510 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:28 smithi174 podman[102273]: 2024-02-19 19:07:28.423043253 +0000 UTC m=+0.134257385 container create b3609ea94466e8847269017eed036c5c1d072227b6acd0e40ed8b3889c73defd (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4, ceph=True, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-02-19T19:07:28.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.461+0000 7f3a63221700 1 -- 172.21.15.114:0/2592361526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a5c101660 msgr2=0x7f3a5c103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:28.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.461+0000 7f3a63221700 1 --2- 172.21.15.114:0/2592361526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a5c101660 0x7f3a5c103a50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400b0a0 tx=0x7f3a5400b3b0 comp rx=0 tx=0).stop 2024-02-19T19:07:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.461+0000 7f3a63221700 1 -- 172.21.15.114:0/2592361526 shutdown_connections 2024-02-19T19:07:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.461+0000 7f3a63221700 1 --2- 172.21.15.114:0/2592361526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a5c103f90 0x7f3a5c106380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.461+0000 7f3a63221700 1 --2- 172.21.15.114:0/2592361526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a5c101660 0x7f3a5c103a50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.461+0000 7f3a63221700 1 -- 172.21.15.114:0/2592361526 >> 172.21.15.114:0/2592361526 conn(0x7f3a5c0fb050 msgr2=0x7f3a5c0fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.461+0000 7f3a63221700 1 -- 172.21.15.114:0/2592361526 shutdown_connections 2024-02-19T19:07:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.462+0000 7f3a63221700 1 -- 172.21.15.114:0/2592361526 wait complete. 2024-02-19T19:07:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.462+0000 7f3a63221700 1 Processor -- start 2024-02-19T19:07:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.462+0000 7f3a63221700 1 -- start start 2024-02-19T19:07:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.462+0000 7f3a63221700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a5c101660 0x7f3a5c1963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.462+0000 7f3a63221700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a5c103f90 0x7f3a5c1968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.462+0000 7f3a63221700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c196f10 con 0x7f3a5c103f90 2024-02-19T19:07:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.462+0000 7f3a63221700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c197050 con 0x7f3a5c101660 2024-02-19T19:07:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.462+0000 7f3a5bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a5c103f90 0x7f3a5c1968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.462+0000 7f3a5bfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a5c103f90 0x7f3a5c1968f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46140/0 (socket says 172.21.15.114:46140) 2024-02-19T19:07:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.462+0000 7f3a5bfff700 1 -- 172.21.15.114:0/3265814754 learned_addr learned my addr 172.21.15.114:0/3265814754 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:07:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.462+0000 7f3a60fbd700 1 --2- 172.21.15.114:0/3265814754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a5c101660 0x7f3a5c1963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.463+0000 7f3a5bfff700 1 -- 172.21.15.114:0/3265814754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a5c101660 msgr2=0x7f3a5c1963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.463+0000 7f3a5bfff700 1 --2- 172.21.15.114:0/3265814754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a5c101660 0x7f3a5c1963b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.463+0000 7f3a5bfff700 1 -- 172.21.15.114:0/3265814754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a54009d00 con 0x7f3a5c103f90 2024-02-19T19:07:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.463+0000 7f3a5bfff700 1 --2- 172.21.15.114:0/3265814754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a5c103f90 0x7f3a5c1968f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c00e920 tx=0x7f3a4c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.463+0000 7f3a59ffb700 1 -- 172.21.15.114:0/3265814754 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a4c00c9d0 con 0x7f3a5c103f90 2024-02-19T19:07:28.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.463+0000 7f3a59ffb700 1 -- 172.21.15.114:0/3265814754 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3a4c010430 con 0x7f3a5c103f90 2024-02-19T19:07:28.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.463+0000 7f3a63221700 1 -- 172.21.15.114:0/3265814754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a5c19bb00 con 0x7f3a5c103f90 2024-02-19T19:07:28.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.463+0000 7f3a59ffb700 1 -- 172.21.15.114:0/3265814754 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a4c018650 con 0x7f3a5c103f90 2024-02-19T19:07:28.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.463+0000 7f3a63221700 1 -- 172.21.15.114:0/3265814754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a5c19c0d0 con 0x7f3a5c103f90 2024-02-19T19:07:28.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.464+0000 7f3a63221700 1 -- 172.21.15.114:0/3265814754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a5c190540 con 0x7f3a5c103f90 2024-02-19T19:07:28.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.465+0000 7f3a59ffb700 1 -- 172.21.15.114:0/3265814754 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f3a4c0105a0 con 0x7f3a5c103f90 2024-02-19T19:07:28.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.465+0000 7f3a59ffb700 1 --2- 172.21.15.114:0/3265814754 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f3a4806d8a0 0x7f3a4806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:28.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.465+0000 7f3a59ffb700 1 -- 172.21.15.114:0/3265814754 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f3a4c014070 con 0x7f3a5c103f90 2024-02-19T19:07:28.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.465+0000 7f3a60fbd700 1 --2- 172.21.15.114:0/3265814754 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f3a4806d8a0 0x7f3a4806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:28.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.466+0000 7f3a60fbd700 1 --2- 172.21.15.114:0/3265814754 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f3a4806d8a0 0x7f3a4806fd60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3a54006070 tx=0x7f3a54006000 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:28.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.470+0000 7f3a59ffb700 1 -- 172.21.15.114:0/3265814754 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3a4c052d20 con 0x7f3a5c103f90 2024-02-19T19:07:28.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.689+0000 7f3a63221700 1 -- 172.21.15.114:0/3265814754 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a5c0611d0 con 0x7f3a4806d8a0 2024-02-19T19:07:28.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.690+0000 7f3a59ffb700 1 -- 172.21.15.114:0/3265814754 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f3a5c0611d0 con 0x7f3a4806d8a0 2024-02-19T19:07:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.692+0000 7f3a63221700 1 -- 172.21.15.114:0/3265814754 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f3a4806d8a0 msgr2=0x7f3a4806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.692+0000 7f3a63221700 1 --2- 172.21.15.114:0/3265814754 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f3a4806d8a0 0x7f3a4806fd60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3a54006070 tx=0x7f3a54006000 comp rx=0 tx=0).stop 2024-02-19T19:07:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.692+0000 7f3a63221700 1 -- 172.21.15.114:0/3265814754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a5c103f90 msgr2=0x7f3a5c1968f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.692+0000 7f3a63221700 1 --2- 172.21.15.114:0/3265814754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a5c103f90 0x7f3a5c1968f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c00e920 tx=0x7f3a4c00ec30 comp rx=0 tx=0).stop 2024-02-19T19:07:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.692+0000 7f3a63221700 1 -- 172.21.15.114:0/3265814754 shutdown_connections 2024-02-19T19:07:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.692+0000 7f3a63221700 1 --2- 172.21.15.114:0/3265814754 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f3a4806d8a0 0x7f3a4806fd60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:28.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.692+0000 7f3a63221700 1 --2- 172.21.15.114:0/3265814754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a5c101660 0x7f3a5c1963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.692+0000 7f3a63221700 1 --2- 172.21.15.114:0/3265814754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3a5c103f90 0x7f3a5c1968f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.692+0000 7f3a63221700 1 -- 172.21.15.114:0/3265814754 >> 172.21.15.114:0/3265814754 conn(0x7f3a5c0fb050 msgr2=0x7f3a5c0fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.692+0000 7f3a63221700 1 -- 172.21.15.114:0/3265814754 shutdown_connections 2024-02-19T19:07:28.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.692+0000 7f3a63221700 1 -- 172.21.15.114:0/3265814754 wait complete. 2024-02-19T19:07:28.723 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-02-19T19:07:28.779 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:28 smithi174 podman[102273]: 2024-02-19 19:07:28.650735992 +0000 UTC m=+0.361950123 container init b3609ea94466e8847269017eed036c5c1d072227b6acd0e40ed8b3889c73defd (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240214) 2024-02-19T19:07:28.779 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:28 smithi174 podman[102273]: 2024-02-19 19:07:28.653788381 +0000 UTC m=+0.365002513 container start b3609ea94466e8847269017eed036c5c1d072227b6acd0e40ed8b3889c73defd (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214) 2024-02-19T19:07:28.779 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:28 smithi174 ceph-osd[102292]: -- 172.21.15.174:0/2892692723 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5574dda5f1e0 con 0x5574dda31000 2024-02-19T19:07:28.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.800+0000 7f40e3b3c700 1 -- 172.21.15.114:0/2845397921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40dc100cb0 msgr2=0x7f40dc101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:28.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.800+0000 7f40e3b3c700 1 --2- 172.21.15.114:0/2845397921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40dc100cb0 0x7f40dc101110 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f40cc009910 tx=0x7f40cc009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:28.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.800+0000 7f40e3b3c700 1 -- 172.21.15.114:0/2845397921 shutdown_connections 2024-02-19T19:07:28.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.800+0000 7f40e3b3c700 1 --2- 172.21.15.114:0/2845397921 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40dc100cb0 0x7f40dc101110 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:28.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.800+0000 7f40e3b3c700 1 --2- 172.21.15.114:0/2845397921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40dc0ffab0 0x7f40dc0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:28.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.800+0000 7f40e3b3c700 1 -- 172.21.15.114:0/2845397921 >> 172.21.15.114:0/2845397921 conn(0x7f40dc0fb050 msgr2=0x7f40dc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:28.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.800+0000 7f40e3b3c700 1 -- 172.21.15.114:0/2845397921 shutdown_connections 2024-02-19T19:07:28.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.800+0000 7f40e3b3c700 1 -- 172.21.15.114:0/2845397921 wait complete. 2024-02-19T19:07:28.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.800+0000 7f40e3b3c700 1 Processor -- start 2024-02-19T19:07:28.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40e3b3c700 1 -- start start 2024-02-19T19:07:28.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40e3b3c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40dc0ffab0 0x7f40dc195260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:28.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40e3b3c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40dc100cb0 0x7f40dc1957a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:28.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40e3b3c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc195dc0 con 0x7f40dc0ffab0 2024-02-19T19:07:28.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40e3b3c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc195f00 con 0x7f40dc100cb0 2024-02-19T19:07:28.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40e18d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40dc0ffab0 0x7f40dc195260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:28.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40e18d8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40dc0ffab0 0x7f40dc195260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46162/0 (socket says 172.21.15.114:46162) 2024-02-19T19:07:28.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40e18d8700 1 -- 172.21.15.114:0/1823342710 learned_addr learned my addr 172.21.15.114:0/1823342710 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:07:28.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40e10d7700 1 --2- 172.21.15.114:0/1823342710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40dc100cb0 0x7f40dc1957a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:28.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40e18d8700 1 -- 172.21.15.114:0/1823342710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40dc100cb0 msgr2=0x7f40dc1957a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:28.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40e18d8700 1 --2- 172.21.15.114:0/1823342710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40dc100cb0 0x7f40dc1957a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:28.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40e18d8700 1 -- 172.21.15.114:0/1823342710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40cc0095f0 con 0x7f40dc0ffab0 2024-02-19T19:07:28.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40e18d8700 1 --2- 172.21.15.114:0/1823342710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40dc0ffab0 0x7f40dc195260 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f40d800e580 tx=0x7f40d800e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:28.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40d2ffd700 1 -- 172.21.15.114:0/1823342710 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40d8003a00 con 0x7f40dc0ffab0 2024-02-19T19:07:28.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40e3b3c700 1 -- 172.21.15.114:0/1823342710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40dc068d00 con 0x7f40dc0ffab0 2024-02-19T19:07:28.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40d2ffd700 1 -- 172.21.15.114:0/1823342710 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f40d8003b60 con 0x7f40dc0ffab0 2024-02-19T19:07:28.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40d2ffd700 1 -- 172.21.15.114:0/1823342710 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40d8007730 con 0x7f40dc0ffab0 2024-02-19T19:07:28.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.801+0000 7f40e3b3c700 1 -- 172.21.15.114:0/1823342710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40dc069250 con 0x7f40dc0ffab0 2024-02-19T19:07:28.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.802+0000 7f40e3b3c700 1 -- 172.21.15.114:0/1823342710 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40dc108610 con 0x7f40dc0ffab0 2024-02-19T19:07:28.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.803+0000 7f40d2ffd700 1 -- 172.21.15.114:0/1823342710 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f40d8003cd0 con 0x7f40dc0ffab0 2024-02-19T19:07:28.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.803+0000 7f40d2ffd700 1 --2- 172.21.15.114:0/1823342710 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f40c806d8a0 0x7f40c806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:28.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.804+0000 7f40d2ffd700 1 -- 172.21.15.114:0/1823342710 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f40d8016070 con 0x7f40dc0ffab0 2024-02-19T19:07:28.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.804+0000 7f40e10d7700 1 --2- 172.21.15.114:0/1823342710 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f40c806d8a0 0x7f40c806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:28.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.804+0000 7f40e10d7700 1 --2- 172.21.15.114:0/1823342710 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f40c806d8a0 0x7f40c806fd60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0059d0 tx=0x7f40cc005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:28.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:28.808+0000 7f40d2ffd700 1 -- 172.21.15.114:0/1823342710 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f40d805d410 con 0x7f40dc0ffab0 2024-02-19T19:07:29.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.031+0000 7f40e3b3c700 1 -- 172.21.15.114:0/1823342710 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f40dc105600 con 0x7f40c806d8a0 2024-02-19T19:07:29.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.032+0000 7f40d2ffd700 1 -- 172.21.15.114:0/1823342710 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f40dc105600 con 0x7f40c806d8a0 2024-02-19T19:07:29.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.034+0000 7f40e3b3c700 1 -- 172.21.15.114:0/1823342710 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f40c806d8a0 msgr2=0x7f40c806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:29.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.034+0000 7f40e3b3c700 1 --2- 172.21.15.114:0/1823342710 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f40c806d8a0 0x7f40c806fd60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0059d0 tx=0x7f40cc005940 comp rx=0 tx=0).stop 2024-02-19T19:07:29.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.034+0000 7f40e3b3c700 1 -- 172.21.15.114:0/1823342710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40dc0ffab0 msgr2=0x7f40dc195260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:29.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.034+0000 7f40e3b3c700 1 --2- 172.21.15.114:0/1823342710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40dc0ffab0 0x7f40dc195260 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f40d800e580 tx=0x7f40d800e940 comp rx=0 tx=0).stop 2024-02-19T19:07:29.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.034+0000 7f40e3b3c700 1 -- 172.21.15.114:0/1823342710 shutdown_connections 2024-02-19T19:07:29.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.034+0000 7f40e3b3c700 1 --2- 172.21.15.114:0/1823342710 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f40c806d8a0 0x7f40c806fd60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.034+0000 7f40e3b3c700 1 --2- 172.21.15.114:0/1823342710 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40dc0ffab0 0x7f40dc195260 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.034+0000 7f40e3b3c700 1 --2- 172.21.15.114:0/1823342710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40dc100cb0 0x7f40dc1957a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.034+0000 7f40e3b3c700 1 -- 172.21.15.114:0/1823342710 >> 172.21.15.114:0/1823342710 conn(0x7f40dc0fb050 msgr2=0x7f40dc103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:29.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.034+0000 7f40e3b3c700 1 -- 172.21.15.114:0/1823342710 shutdown_connections 2024-02-19T19:07:29.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.034+0000 7f40e3b3c700 1 -- 172.21.15.114:0/1823342710 wait complete. 2024-02-19T19:07:29.145 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:28 smithi174 bash[102273]: b3609ea94466e8847269017eed036c5c1d072227b6acd0e40ed8b3889c73defd 2024-02-19T19:07:29.145 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:28 smithi174 systemd[1]: Started Ceph osd.4 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T19:07:29.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:28 smithi174 ceph-mon[87279]: from='client.34338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:29.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:28 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:29.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:28 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:29.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:28 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:07:29.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.141+0000 7f706ef98700 1 -- 172.21.15.114:0/3649091677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7068101dc0 msgr2=0x7f7068102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:29.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.141+0000 7f706ef98700 1 --2- 172.21.15.114:0/3649091677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7068101dc0 0x7f7068102220 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7054009910 tx=0x7f7054009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:29.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.141+0000 7f706ef98700 1 -- 172.21.15.114:0/3649091677 shutdown_connections 2024-02-19T19:07:29.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.141+0000 7f706ef98700 1 --2- 172.21.15.114:0/3649091677 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7068101dc0 0x7f7068102220 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.141+0000 7f706ef98700 1 --2- 172.21.15.114:0/3649091677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7068100bc0 0x7f7068100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.141+0000 7f706ef98700 1 -- 172.21.15.114:0/3649091677 >> 172.21.15.114:0/3649091677 conn(0x7f70680fc140 msgr2=0x7f70680fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:29.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.141+0000 7f706ef98700 1 -- 172.21.15.114:0/3649091677 shutdown_connections 2024-02-19T19:07:29.162 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:28 smithi114 ceph-mon[92983]: from='client.34338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:29.163 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:28 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:29.163 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:28 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:29.163 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:28 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:07:29.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.141+0000 7f706ef98700 1 -- 172.21.15.114:0/3649091677 wait complete. 2024-02-19T19:07:29.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.141+0000 7f706ef98700 1 Processor -- start 2024-02-19T19:07:29.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.142+0000 7f706ef98700 1 -- start start 2024-02-19T19:07:29.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.142+0000 7f706ef98700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7068100bc0 0x7f7068074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:29.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.142+0000 7f706ef98700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7068101dc0 0x7f7068073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:29.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.142+0000 7f706ef98700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70680736c0 con 0x7f7068100bc0 2024-02-19T19:07:29.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.142+0000 7f706ef98700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7068073800 con 0x7f7068101dc0 2024-02-19T19:07:29.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.142+0000 7f706cd34700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7068100bc0 0x7f7068074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:29.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.142+0000 7f706cd34700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7068100bc0 0x7f7068074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46170/0 (socket says 172.21.15.114:46170) 2024-02-19T19:07:29.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.142+0000 7f706cd34700 1 -- 172.21.15.114:0/2259116629 learned_addr learned my addr 172.21.15.114:0/2259116629 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:07:29.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.142+0000 7f7067fff700 1 --2- 172.21.15.114:0/2259116629 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7068101dc0 0x7f7068073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:29.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.142+0000 7f706cd34700 1 -- 172.21.15.114:0/2259116629 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7068101dc0 msgr2=0x7f7068073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:29.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.142+0000 7f706cd34700 1 --2- 172.21.15.114:0/2259116629 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7068101dc0 0x7f7068073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.142+0000 7f706cd34700 1 -- 172.21.15.114:0/2259116629 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70540095f0 con 0x7f7068100bc0 2024-02-19T19:07:29.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.142+0000 7f706cd34700 1 --2- 172.21.15.114:0/2259116629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7068100bc0 0x7f7068074b30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f705c00d900 tx=0x7f705c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:29.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.143+0000 7f7065ffb700 1 -- 172.21.15.114:0/2259116629 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f705c0041f0 con 0x7f7068100bc0 2024-02-19T19:07:29.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.143+0000 7f706ef98700 1 -- 172.21.15.114:0/2259116629 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7068073ae0 con 0x7f7068100bc0 2024-02-19T19:07:29.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.143+0000 7f7065ffb700 1 -- 172.21.15.114:0/2259116629 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f705c004350 con 0x7f7068100bc0 2024-02-19T19:07:29.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.143+0000 7f7065ffb700 1 -- 172.21.15.114:0/2259116629 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f705c003d20 con 0x7f7068100bc0 2024-02-19T19:07:29.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.144+0000 7f706ef98700 1 -- 172.21.15.114:0/2259116629 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f706819e3d0 con 0x7f7068100bc0 2024-02-19T19:07:29.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.144+0000 7f706ef98700 1 -- 172.21.15.114:0/2259116629 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7068109720 con 0x7f7068100bc0 2024-02-19T19:07:29.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.145+0000 7f7065ffb700 1 -- 172.21.15.114:0/2259116629 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f705c009730 con 0x7f7068100bc0 2024-02-19T19:07:29.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.145+0000 7f7065ffb700 1 --2- 172.21.15.114:0/2259116629 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f705806d8a0 0x7f705806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:29.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.145+0000 7f7065ffb700 1 -- 172.21.15.114:0/2259116629 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f705c08d4d0 con 0x7f7068100bc0 2024-02-19T19:07:29.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.145+0000 7f7067fff700 1 --2- 172.21.15.114:0/2259116629 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f705806d8a0 0x7f705806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:29.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.146+0000 7f7067fff700 1 --2- 172.21.15.114:0/2259116629 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f705806d8a0 0x7f705806fd60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f705400b5c0 tx=0x7f7054005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:29.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.154+0000 7f7065ffb700 1 -- 172.21.15.114:0/2259116629 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f705c05a6e0 con 0x7f7068100bc0 2024-02-19T19:07:29.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.366+0000 7f706ef98700 1 -- 172.21.15.114:0/2259116629 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f706819e980 con 0x7f705806d8a0 2024-02-19T19:07:29.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.377+0000 7f7065ffb700 1 -- 172.21.15.114:0/2259116629 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f706819e980 con 0x7f705806d8a0 2024-02-19T19:07:29.398 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:07:29.398 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (11m) 63s ago 14m 26.5M - 0.25.0 c8568f914cd2 30444b676ca4 2024-02-19T19:07:29.398 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (14m) 63s ago 14m 8472k - 18.2.1 94eadd990e0a 4918cd87e312 2024-02-19T19:07:29.398 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi174 smithi174 running (9m) 30s ago 12m 8019k - 18.2.1 94eadd990e0a 516c34431b93 2024-02-19T19:07:29.398 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (2m) 63s ago 14m 6660k - 19.0.0-1450-gf78a58c0 f081f847070a f10b46e9d636 2024-02-19T19:07:29.398 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi174 smithi174 running (2m) 30s ago 12m 6660k - 19.0.0-1450-gf78a58c0 f081f847070a fb3b17c81bd8 2024-02-19T19:07:29.398 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (11m) 63s ago 13m 85.5M - 9.4.7 954c08fa6188 478bd533e827 2024-02-19T19:07:29.398 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tcwpod smithi114 running (7m) 63s ago 7m 27.5M - 18.2.1 94eadd990e0a 420175777cf4 2024-02-19T19:07:29.398 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.vpoeop smithi114 running (7m) 63s ago 7m 24.6M - 18.2.1 94eadd990e0a f81a79e53ead 2024-02-19T19:07:29.398 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.stnfnf smithi174 running (7m) 30s ago 7m 26.4M - 18.2.1 94eadd990e0a 3df35954a492 2024-02-19T19:07:29.399 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.znhuvd smithi174 running (7m) 30s ago 7m 25.1M - 18.2.1 94eadd990e0a 2d2267494ae1 2024-02-19T19:07:29.399 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.pcvtrk smithi114 *:8443,9283,8765 running (4m) 63s ago 15m 503M - 19.0.0-1450-gf78a58c0 f081f847070a 05b5c06122b1 2024-02-19T19:07:29.399 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi174.lydvnj smithi174 *:8443,9283,8765 running (4m) 30s ago 11m 452M - 19.0.0-1450-gf78a58c0 f081f847070a f962b1af5aed 2024-02-19T19:07:29.399 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (3m) 63s ago 15m 48.1M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e51f650f9be9 2024-02-19T19:07:29.399 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi174 smithi174 running (3m) 30s ago 11m 36.1M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e811581f05fb 2024-02-19T19:07:29.399 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (14m) 63s ago 14m 14.3M - 1.5.0 0da6a335fe13 b6b258074cce 2024-02-19T19:07:29.399 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi174 smithi174 *:9100 running (12m) 30s ago 12m 15.0M - 1.5.0 0da6a335fe13 d0b4fdf67b22 2024-02-19T19:07:29.399 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (2m) 63s ago 11m 51.2M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 764769439e0e 2024-02-19T19:07:29.399 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (105s) 63s ago 10m 51.8M 1216M 19.0.0-1450-gf78a58c0 f081f847070a c1c32dbdf573 2024-02-19T19:07:29.399 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (69s) 63s ago 10m 11.5M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 11e64d20e378 2024-02-19T19:07:29.399 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi174 running (34s) 30s ago 9m 12.5M 2240M 19.0.0-1450-gf78a58c0 f081f847070a cc3a55bd908f 2024-02-19T19:07:29.399 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi174 starting - - - 2240M 2024-02-19T19:07:29.399 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi174 running (8m) 30s ago 8m 54.1M 2240M 18.2.1 94eadd990e0a 5771a995ed75 2024-02-19T19:07:29.400 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (4m) 63s ago 12m 63.5M - 2.43.0 a07b618ecd1d a36922c7c9a1 2024-02-19T19:07:29.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.380+0000 7f706ef98700 1 -- 172.21.15.114:0/2259116629 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f705806d8a0 msgr2=0x7f705806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:29.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.380+0000 7f706ef98700 1 --2- 172.21.15.114:0/2259116629 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f705806d8a0 0x7f705806fd60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f705400b5c0 tx=0x7f7054005ee0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.380+0000 7f706ef98700 1 -- 172.21.15.114:0/2259116629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7068100bc0 msgr2=0x7f7068074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:29.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.380+0000 7f706ef98700 1 --2- 172.21.15.114:0/2259116629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7068100bc0 0x7f7068074b30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f705c00d900 tx=0x7f705c00dc10 comp rx=0 tx=0).stop 2024-02-19T19:07:29.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.380+0000 7f706ef98700 1 -- 172.21.15.114:0/2259116629 shutdown_connections 2024-02-19T19:07:29.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.380+0000 7f706ef98700 1 --2- 172.21.15.114:0/2259116629 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f705806d8a0 0x7f705806fd60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.380+0000 7f706ef98700 1 --2- 172.21.15.114:0/2259116629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7068100bc0 0x7f7068074b30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.380+0000 7f706ef98700 1 --2- 172.21.15.114:0/2259116629 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7068101dc0 0x7f7068073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.380+0000 7f706ef98700 1 -- 172.21.15.114:0/2259116629 >> 172.21.15.114:0/2259116629 conn(0x7f70680fc140 msgr2=0x7f7068104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:29.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.380+0000 7f706ef98700 1 -- 172.21.15.114:0/2259116629 shutdown_connections 2024-02-19T19:07:29.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.380+0000 7f706ef98700 1 -- 172.21.15.114:0/2259116629 wait complete. 2024-02-19T19:07:29.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.490+0000 7f34e013e700 1 -- 172.21.15.114:0/3140524322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34d8100cd0 msgr2=0x7f34d8101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:29.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.490+0000 7f34e013e700 1 --2- 172.21.15.114:0/3140524322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34d8100cd0 0x7f34d8101130 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f34c8009910 tx=0x7f34c8009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:29.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.490+0000 7f34e013e700 1 -- 172.21.15.114:0/3140524322 shutdown_connections 2024-02-19T19:07:29.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.490+0000 7f34e013e700 1 --2- 172.21.15.114:0/3140524322 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34d8100cd0 0x7f34d8101130 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.490+0000 7f34e013e700 1 --2- 172.21.15.114:0/3140524322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34d80ffad0 0x7f34d80ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.490+0000 7f34e013e700 1 -- 172.21.15.114:0/3140524322 >> 172.21.15.114:0/3140524322 conn(0x7f34d80fb050 msgr2=0x7f34d80fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:29.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.490+0000 7f34e013e700 1 -- 172.21.15.114:0/3140524322 shutdown_connections 2024-02-19T19:07:29.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.490+0000 7f34e013e700 1 -- 172.21.15.114:0/3140524322 wait complete. 2024-02-19T19:07:29.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.491+0000 7f34e013e700 1 Processor -- start 2024-02-19T19:07:29.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.491+0000 7f34e013e700 1 -- start start 2024-02-19T19:07:29.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.491+0000 7f34e013e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34d80ffad0 0x7f34d8103a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:29.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.491+0000 7f34e013e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34d8100cd0 0x7f34d81020b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:29.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.491+0000 7f34e013e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d81025f0 con 0x7f34d80ffad0 2024-02-19T19:07:29.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.491+0000 7f34e013e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d8102730 con 0x7f34d8100cd0 2024-02-19T19:07:29.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.491+0000 7f34ddeda700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34d80ffad0 0x7f34d8103a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:29.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.491+0000 7f34ddeda700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34d80ffad0 0x7f34d8103a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46178/0 (socket says 172.21.15.114:46178) 2024-02-19T19:07:29.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.491+0000 7f34dd6d9700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34d8100cd0 0x7f34d81020b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:29.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.491+0000 7f34ddeda700 1 -- 172.21.15.114:0/3452453134 learned_addr learned my addr 172.21.15.114:0/3452453134 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:07:29.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.492+0000 7f34ddeda700 1 -- 172.21.15.114:0/3452453134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34d8100cd0 msgr2=0x7f34d81020b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:29.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.492+0000 7f34ddeda700 1 --2- 172.21.15.114:0/3452453134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34d8100cd0 0x7f34d81020b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.492+0000 7f34ddeda700 1 -- 172.21.15.114:0/3452453134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c80095f0 con 0x7f34d80ffad0 2024-02-19T19:07:29.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.492+0000 7f34dd6d9700 1 --2- 172.21.15.114:0/3452453134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34d8100cd0 0x7f34d81020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:07:29.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.492+0000 7f34ddeda700 1 --2- 172.21.15.114:0/3452453134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34d80ffad0 0x7f34d8103a60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f34d400d900 tx=0x7f34d400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:29.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.492+0000 7f34ceffd700 1 -- 172.21.15.114:0/3452453134 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34d40041f0 con 0x7f34d80ffad0 2024-02-19T19:07:29.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.492+0000 7f34ceffd700 1 -- 172.21.15.114:0/3452453134 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f34d4004350 con 0x7f34d80ffad0 2024-02-19T19:07:29.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.492+0000 7f34e013e700 1 -- 172.21.15.114:0/3452453134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d8102a10 con 0x7f34d80ffad0 2024-02-19T19:07:29.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.492+0000 7f34ceffd700 1 -- 172.21.15.114:0/3452453134 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34d4003d20 con 0x7f34d80ffad0 2024-02-19T19:07:29.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.492+0000 7f34e013e700 1 -- 172.21.15.114:0/3452453134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d819e3e0 con 0x7f34d80ffad0 2024-02-19T19:07:29.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.493+0000 7f34e013e700 1 -- 172.21.15.114:0/3452453134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d8108630 con 0x7f34d80ffad0 2024-02-19T19:07:29.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.494+0000 7f34ceffd700 1 -- 172.21.15.114:0/3452453134 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f34d4009730 con 0x7f34d80ffad0 2024-02-19T19:07:29.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.494+0000 7f34ceffd700 1 --2- 172.21.15.114:0/3452453134 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f34c406d8a0 0x7f34c406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:29.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.494+0000 7f34ceffd700 1 -- 172.21.15.114:0/3452453134 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f34d408d4d0 con 0x7f34d80ffad0 2024-02-19T19:07:29.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.494+0000 7f34dd6d9700 1 --2- 172.21.15.114:0/3452453134 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f34c406d8a0 0x7f34c406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:29.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.495+0000 7f34dd6d9700 1 --2- 172.21.15.114:0/3452453134 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f34c406d8a0 0x7f34c406fd60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f34c8006010 tx=0x7f34c800b560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:29.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.499+0000 7f34ceffd700 1 -- 172.21.15.114:0/3452453134 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f34d405a6e0 con 0x7f34d80ffad0 2024-02-19T19:07:29.645 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:29 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4[102288]: 2024-02-19T19:07:29.202+0000 7fe32f5737c0 -1 Falling back to public interface 2024-02-19T19:07:29.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.772+0000 7f34e013e700 1 -- 172.21.15.114:0/3452453134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f34d819ece0 con 0x7f34d80ffad0 2024-02-19T19:07:29.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.772+0000 7f34ceffd700 1 -- 172.21.15.114:0/3452453134 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f34d4003e80 con 0x7f34d80ffad0 2024-02-19T19:07:29.793 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:07:29.793 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-02-19T19:07:29.793 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:07:29.793 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:07:29.794 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-02-19T19:07:29.794 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:07:29.794 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:07:29.794 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-02-19T19:07:29.794 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-02-19T19:07:29.794 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 4 2024-02-19T19:07:29.794 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:07:29.794 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-02-19T19:07:29.794 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:07:29.794 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:07:29.794 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-02-19T19:07:29.794 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 5, 2024-02-19T19:07:29.794 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 8 2024-02-19T19:07:29.794 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-02-19T19:07:29.794 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:07:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.774+0000 7f34e013e700 1 -- 172.21.15.114:0/3452453134 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f34c406d8a0 msgr2=0x7f34c406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.774+0000 7f34e013e700 1 --2- 172.21.15.114:0/3452453134 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f34c406d8a0 0x7f34c406fd60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f34c8006010 tx=0x7f34c800b560 comp rx=0 tx=0).stop 2024-02-19T19:07:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.774+0000 7f34e013e700 1 -- 172.21.15.114:0/3452453134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34d80ffad0 msgr2=0x7f34d8103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.774+0000 7f34e013e700 1 --2- 172.21.15.114:0/3452453134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34d80ffad0 0x7f34d8103a60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f34d400d900 tx=0x7f34d400dc10 comp rx=0 tx=0).stop 2024-02-19T19:07:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.774+0000 7f34e013e700 1 -- 172.21.15.114:0/3452453134 shutdown_connections 2024-02-19T19:07:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.774+0000 7f34e013e700 1 --2- 172.21.15.114:0/3452453134 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f34c406d8a0 0x7f34c406fd60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.774+0000 7f34e013e700 1 --2- 172.21.15.114:0/3452453134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34d80ffad0 0x7f34d8103a60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.774+0000 7f34e013e700 1 --2- 172.21.15.114:0/3452453134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34d8100cd0 0x7f34d81020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.774+0000 7f34e013e700 1 -- 172.21.15.114:0/3452453134 >> 172.21.15.114:0/3452453134 conn(0x7f34d80fb050 msgr2=0x7f34d8068df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.774+0000 7f34e013e700 1 -- 172.21.15.114:0/3452453134 shutdown_connections 2024-02-19T19:07:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.774+0000 7f34e013e700 1 -- 172.21.15.114:0/3452453134 wait complete. 2024-02-19T19:07:29.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.878+0000 7fcdc0053700 1 -- 172.21.15.114:0/1821162706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdb8103f80 msgr2=0x7fcdb8106370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.878+0000 7fcdc0053700 1 --2- 172.21.15.114:0/1821162706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdb8103f80 0x7fcdb8106370 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcda8009910 tx=0x7fcda8009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.878+0000 7fcdc0053700 1 -- 172.21.15.114:0/1821162706 shutdown_connections 2024-02-19T19:07:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.878+0000 7fcdc0053700 1 --2- 172.21.15.114:0/1821162706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdb8103f80 0x7fcdb8106370 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.878+0000 7fcdc0053700 1 --2- 172.21.15.114:0/1821162706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdb8101650 0x7fcdb8103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.878+0000 7fcdc0053700 1 -- 172.21.15.114:0/1821162706 >> 172.21.15.114:0/1821162706 conn(0x7fcdb80fb060 msgr2=0x7fcdb80fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.878+0000 7fcdc0053700 1 -- 172.21.15.114:0/1821162706 shutdown_connections 2024-02-19T19:07:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.878+0000 7fcdc0053700 1 -- 172.21.15.114:0/1821162706 wait complete. 2024-02-19T19:07:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.878+0000 7fcdc0053700 1 Processor -- start 2024-02-19T19:07:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.879+0000 7fcdc0053700 1 -- start start 2024-02-19T19:07:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.879+0000 7fcdc0053700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdb8101650 0x7fcdb81963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.879+0000 7fcdc0053700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdb8103f80 0x7fcdb81968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.879+0000 7fcdc0053700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb8196f00 con 0x7fcdb8103f80 2024-02-19T19:07:29.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.879+0000 7fcdc0053700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb8197040 con 0x7fcdb8101650 2024-02-19T19:07:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.879+0000 7fcdbd5ee700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdb8103f80 0x7fcdb81968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.879+0000 7fcdbd5ee700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdb8103f80 0x7fcdb81968e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46196/0 (socket says 172.21.15.114:46196) 2024-02-19T19:07:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.879+0000 7fcdbd5ee700 1 -- 172.21.15.114:0/1461696244 learned_addr learned my addr 172.21.15.114:0/1461696244 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:07:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.879+0000 7fcdbddef700 1 --2- 172.21.15.114:0/1461696244 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdb8101650 0x7fcdb81963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.879+0000 7fcdbd5ee700 1 -- 172.21.15.114:0/1461696244 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdb8101650 msgr2=0x7fcdb81963a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.879+0000 7fcdbd5ee700 1 --2- 172.21.15.114:0/1461696244 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdb8101650 0x7fcdb81963a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.879+0000 7fcdbd5ee700 1 -- 172.21.15.114:0/1461696244 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda80095f0 con 0x7fcdb8103f80 2024-02-19T19:07:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.879+0000 7fcdbddef700 1 --2- 172.21.15.114:0/1461696244 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdb8101650 0x7fcdb81963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:07:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.879+0000 7fcdbd5ee700 1 --2- 172.21.15.114:0/1461696244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdb8103f80 0x7fcdb81968e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcda800b5c0 tx=0x7fcda800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.879+0000 7fcdaeffd700 1 -- 172.21.15.114:0/1461696244 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcda801d070 con 0x7fcdb8103f80 2024-02-19T19:07:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.879+0000 7fcdaeffd700 1 -- 172.21.15.114:0/1461696244 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fcda8022470 con 0x7fcdb8103f80 2024-02-19T19:07:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.880+0000 7fcdc0053700 1 -- 172.21.15.114:0/1461696244 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb819ba90 con 0x7fcdb8103f80 2024-02-19T19:07:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.880+0000 7fcdaeffd700 1 -- 172.21.15.114:0/1461696244 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcda800f690 con 0x7fcdb8103f80 2024-02-19T19:07:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.880+0000 7fcdc0053700 1 -- 172.21.15.114:0/1461696244 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb819c000 con 0x7fcdb8103f80 2024-02-19T19:07:29.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.881+0000 7fcdc0053700 1 -- 172.21.15.114:0/1461696244 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb81905b0 con 0x7fcdb8103f80 2024-02-19T19:07:29.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.881+0000 7fcdaeffd700 1 -- 172.21.15.114:0/1461696244 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7fcda8004b70 con 0x7fcdb8103f80 2024-02-19T19:07:29.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.881+0000 7fcdaeffd700 1 --2- 172.21.15.114:0/1461696244 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fcda406d8a0 0x7fcda406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:29.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.882+0000 7fcdaeffd700 1 -- 172.21.15.114:0/1461696244 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6790+0+0 (secure 0 0 0) 0x7fcda808ed80 con 0x7fcdb8103f80 2024-02-19T19:07:29.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.882+0000 7fcdbddef700 1 --2- 172.21.15.114:0/1461696244 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fcda406d8a0 0x7fcda406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:29.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.882+0000 7fcdbddef700 1 --2- 172.21.15.114:0/1461696244 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fcda406d8a0 0x7fcda406fd60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4007900 tx=0x7fcdb4008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:29.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:29.886+0000 7fcdaeffd700 1 -- 172.21.15.114:0/1461696244 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcda805c160 con 0x7fcdb8103f80 2024-02-19T19:07:30.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:29 smithi174 ceph-mon[87279]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:30.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:29 smithi174 ceph-mon[87279]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:30.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:29 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/3452453134' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:30.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:29 smithi174 ceph-mon[87279]: pgmap v143: 65 pgs: 11 active+undersized+degraded, 21 active+undersized, 33 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 21/129 objects degraded (16.279%) 2024-02-19T19:07:30.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.125+0000 7fcdc0053700 1 -- 172.21.15.114:0/1461696244 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcdb802cc70 con 0x7fcdb8103f80 2024-02-19T19:07:30.146 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:29 smithi114 ceph-mon[92983]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:30.146 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:29 smithi114 ceph-mon[92983]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:30.147 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:29 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/3452453134' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:30.147 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:29 smithi114 ceph-mon[92983]: pgmap v143: 65 pgs: 11 active+undersized+degraded, 21 active+undersized, 33 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 21/129 objects degraded (16.279%) 2024-02-19T19:07:30.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.126+0000 7fcdaeffd700 1 -- 172.21.15.114:0/1461696244 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fcda8027070 con 0x7fcdb8103f80 2024-02-19T19:07:30.147 INFO:teuthology.orchestra.run.smithi114.stdout:e18 2024-02-19T19:07:30.147 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:07:30.147 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:07:30.147 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-02-19T19:07:30.147 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:07:30.148 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:07:30.148 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-02-19T19:07:30.148 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 18 2024-02-19T19:07:30.148 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:07:30.148 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:07:30.148 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-02-19T19:00:21.203922+0000 2024-02-19T19:07:30.148 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-02-19T19:07:30.148 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-02-19T19:07:30.148 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-02-19T19:07:30.148 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-02-19T19:07:30.148 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-02-19T19:07:30.148 INFO:teuthology.orchestra.run.smithi114.stdout:max_xattr_size 65536 2024-02-19T19:07:30.148 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-02-19T19:07:30.149 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-02-19T19:07:30.149 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 47 2024-02-19T19:07:30.149 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:07:30.149 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 2 2024-02-19T19:07:30.149 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1 2024-02-19T19:07:30.149 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=14530,1=24269} 2024-02-19T19:07:30.149 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-02-19T19:07:30.149 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-02-19T19:07:30.149 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-02-19T19:07:30.149 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-02-19T19:07:30.149 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-02-19T19:07:30.149 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data enabled 2024-02-19T19:07:30.150 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-02-19T19:07:30.150 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-02-19T19:07:30.150 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 1 2024-02-19T19:07:30.150 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:07:30.150 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.vpoeop{0:14540} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:07:30.150 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.znhuvd{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:07:30.150 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.stnfnf{1:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:07:30.150 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:07:30.150 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:07:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.128+0000 7fcdc0053700 1 -- 172.21.15.114:0/1461696244 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fcda406d8a0 msgr2=0x7fcda406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.128+0000 7fcdc0053700 1 --2- 172.21.15.114:0/1461696244 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fcda406d8a0 0x7fcda406fd60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4007900 tx=0x7fcdb4008040 comp rx=0 tx=0).stop 2024-02-19T19:07:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.128+0000 7fcdc0053700 1 -- 172.21.15.114:0/1461696244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdb8103f80 msgr2=0x7fcdb81968e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.128+0000 7fcdc0053700 1 --2- 172.21.15.114:0/1461696244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdb8103f80 0x7fcdb81968e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcda800b5c0 tx=0x7fcda800be20 comp rx=0 tx=0).stop 2024-02-19T19:07:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.128+0000 7fcdc0053700 1 -- 172.21.15.114:0/1461696244 shutdown_connections 2024-02-19T19:07:30.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.128+0000 7fcdc0053700 1 --2- 172.21.15.114:0/1461696244 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fcda406d8a0 0x7fcda406fd60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.128+0000 7fcdc0053700 1 --2- 172.21.15.114:0/1461696244 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdb8101650 0x7fcdb81963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.128+0000 7fcdc0053700 1 --2- 172.21.15.114:0/1461696244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcdb8103f80 0x7fcdb81968e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.128+0000 7fcdc0053700 1 -- 172.21.15.114:0/1461696244 >> 172.21.15.114:0/1461696244 conn(0x7fcdb80fb060 msgr2=0x7fcdb80fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.128+0000 7fcdc0053700 1 -- 172.21.15.114:0/1461696244 shutdown_connections 2024-02-19T19:07:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.128+0000 7fcdc0053700 1 -- 172.21.15.114:0/1461696244 wait complete. 2024-02-19T19:07:30.152 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 18 2024-02-19T19:07:30.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.232+0000 7f17ce810700 1 -- 172.21.15.114:0/4203760524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17c80ffab0 msgr2=0x7f17c80ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:30.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.232+0000 7f17ce810700 1 --2- 172.21.15.114:0/4203760524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17c80ffab0 0x7f17c80ffed0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009910 tx=0x7f17bc009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:30.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.232+0000 7f17ce810700 1 -- 172.21.15.114:0/4203760524 shutdown_connections 2024-02-19T19:07:30.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.232+0000 7f17ce810700 1 --2- 172.21.15.114:0/4203760524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17c8100cb0 0x7f17c8101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:30.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.232+0000 7f17ce810700 1 --2- 172.21.15.114:0/4203760524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17c80ffab0 0x7f17c80ffed0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:30.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.232+0000 7f17ce810700 1 -- 172.21.15.114:0/4203760524 >> 172.21.15.114:0/4203760524 conn(0x7f17c80fb030 msgr2=0x7f17c80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:30.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.232+0000 7f17ce810700 1 -- 172.21.15.114:0/4203760524 shutdown_connections 2024-02-19T19:07:30.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.232+0000 7f17ce810700 1 -- 172.21.15.114:0/4203760524 wait complete. 2024-02-19T19:07:30.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.233+0000 7f17ce810700 1 Processor -- start 2024-02-19T19:07:30.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.233+0000 7f17ce810700 1 -- start start 2024-02-19T19:07:30.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.233+0000 7f17ce810700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17c80ffab0 0x7f17c8195240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:30.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.233+0000 7f17ce810700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17c8100cb0 0x7f17c8195780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:30.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.233+0000 7f17ce810700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c8195da0 con 0x7f17c80ffab0 2024-02-19T19:07:30.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.233+0000 7f17ce810700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c8195ee0 con 0x7f17c8100cb0 2024-02-19T19:07:30.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.233+0000 7f17c7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17c80ffab0 0x7f17c8195240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:30.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.233+0000 7f17c7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17c80ffab0 0x7f17c8195240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46206/0 (socket says 172.21.15.114:46206) 2024-02-19T19:07:30.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.233+0000 7f17c7fff700 1 -- 172.21.15.114:0/2315194730 learned_addr learned my addr 172.21.15.114:0/2315194730 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:07:30.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.233+0000 7f17c77fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17c8100cb0 0x7f17c8195780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:30.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.233+0000 7f17c77fe700 1 -- 172.21.15.114:0/2315194730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17c80ffab0 msgr2=0x7f17c8195240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:30.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.233+0000 7f17c77fe700 1 --2- 172.21.15.114:0/2315194730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17c80ffab0 0x7f17c8195240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:30.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.233+0000 7f17c77fe700 1 -- 172.21.15.114:0/2315194730 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17bc0095f0 con 0x7f17c8100cb0 2024-02-19T19:07:30.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.233+0000 7f17c7fff700 1 --2- 172.21.15.114:0/2315194730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17c80ffab0 0x7f17c8195240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:07:30.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.234+0000 7f17c77fe700 1 --2- 172.21.15.114:0/2315194730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17c8100cb0 0x7f17c8195780 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f17b800e970 tx=0x7f17b800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:30.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.234+0000 7f17c57fa700 1 -- 172.21.15.114:0/2315194730 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17b800c9d0 con 0x7f17c8100cb0 2024-02-19T19:07:30.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.234+0000 7f17ce810700 1 -- 172.21.15.114:0/2315194730 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c819a990 con 0x7f17c8100cb0 2024-02-19T19:07:30.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.234+0000 7f17c57fa700 1 -- 172.21.15.114:0/2315194730 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f17b8010430 con 0x7f17c8100cb0 2024-02-19T19:07:30.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.234+0000 7f17c57fa700 1 -- 172.21.15.114:0/2315194730 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17b8018650 con 0x7f17c8100cb0 2024-02-19T19:07:30.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.234+0000 7f17ce810700 1 -- 172.21.15.114:0/2315194730 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c819aee0 con 0x7f17c8100cb0 2024-02-19T19:07:30.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.235+0000 7f17ce810700 1 -- 172.21.15.114:0/2315194730 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c8108610 con 0x7f17c8100cb0 2024-02-19T19:07:30.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.235+0000 7f17c57fa700 1 -- 172.21.15.114:0/2315194730 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f17b80105a0 con 0x7f17c8100cb0 2024-02-19T19:07:30.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.236+0000 7f17c57fa700 1 --2- 172.21.15.114:0/2315194730 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f17b006d8a0 0x7f17b006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:30.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.236+0000 7f17c57fa700 1 -- 172.21.15.114:0/2315194730 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f17b8014070 con 0x7f17c8100cb0 2024-02-19T19:07:30.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.236+0000 7f17c7fff700 1 --2- 172.21.15.114:0/2315194730 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f17b006d8a0 0x7f17b006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:30.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.236+0000 7f17c7fff700 1 --2- 172.21.15.114:0/2315194730 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f17b006d8a0 0x7f17b006fd60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009fd0 tx=0x7f17bc005940 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:30.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.241+0000 7f17c57fa700 1 -- 172.21.15.114:0/2315194730 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f17b8052d20 con 0x7f17c8100cb0 2024-02-19T19:07:30.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.454+0000 7f17ce810700 1 -- 172.21.15.114:0/2315194730 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f17c8105600 con 0x7f17b006d8a0 2024-02-19T19:07:30.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.455+0000 7f17c57fa700 1 -- 172.21.15.114:0/2315194730 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f17c8105600 con 0x7f17b006d8a0 2024-02-19T19:07:30.476 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:07:30.476 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:07:30.476 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-02-19T19:07:30.476 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:07:30.476 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [ 2024-02-19T19:07:30.476 INFO:teuthology.orchestra.run.smithi114.stdout: "crash", 2024-02-19T19:07:30.476 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr", 2024-02-19T19:07:30.476 INFO:teuthology.orchestra.run.smithi114.stdout: "mon" 2024-02-19T19:07:30.477 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-02-19T19:07:30.477 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "10/23 daemons upgraded", 2024-02-19T19:07:30.477 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "Currently upgrading osd daemons", 2024-02-19T19:07:30.477 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-02-19T19:07:30.477 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:07:30.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.457+0000 7f17ce810700 1 -- 172.21.15.114:0/2315194730 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f17b006d8a0 msgr2=0x7f17b006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:30.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.457+0000 7f17ce810700 1 --2- 172.21.15.114:0/2315194730 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f17b006d8a0 0x7f17b006fd60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009fd0 tx=0x7f17bc005940 comp rx=0 tx=0).stop 2024-02-19T19:07:30.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.457+0000 7f17ce810700 1 -- 172.21.15.114:0/2315194730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17c8100cb0 msgr2=0x7f17c8195780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:30.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.457+0000 7f17ce810700 1 --2- 172.21.15.114:0/2315194730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17c8100cb0 0x7f17c8195780 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f17b800e970 tx=0x7f17b800ec80 comp rx=0 tx=0).stop 2024-02-19T19:07:30.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.457+0000 7f17ce810700 1 -- 172.21.15.114:0/2315194730 shutdown_connections 2024-02-19T19:07:30.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.457+0000 7f17ce810700 1 --2- 172.21.15.114:0/2315194730 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f17b006d8a0 0x7f17b006fd60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:30.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.457+0000 7f17ce810700 1 --2- 172.21.15.114:0/2315194730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17c80ffab0 0x7f17c8195240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:30.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.457+0000 7f17ce810700 1 --2- 172.21.15.114:0/2315194730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17c8100cb0 0x7f17c8195780 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:30.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.457+0000 7f17ce810700 1 -- 172.21.15.114:0/2315194730 >> 172.21.15.114:0/2315194730 conn(0x7f17c80fb030 msgr2=0x7f17c8103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:30.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.457+0000 7f17ce810700 1 -- 172.21.15.114:0/2315194730 shutdown_connections 2024-02-19T19:07:30.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.457+0000 7f17ce810700 1 -- 172.21.15.114:0/2315194730 wait complete. 2024-02-19T19:07:30.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.562+0000 7f2b76d26700 1 -- 172.21.15.114:0/3312330726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70101db0 msgr2=0x7f2b70102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:30.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.562+0000 7f2b76d26700 1 --2- 172.21.15.114:0/3312330726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70101db0 0x7f2b70102210 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2b64009910 tx=0x7f2b64009c20 comp rx=0 tx=0).stop 2024-02-19T19:07:30.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.562+0000 7f2b76d26700 1 -- 172.21.15.114:0/3312330726 shutdown_connections 2024-02-19T19:07:30.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.562+0000 7f2b76d26700 1 --2- 172.21.15.114:0/3312330726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70101db0 0x7f2b70102210 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:30.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.562+0000 7f2b76d26700 1 --2- 172.21.15.114:0/3312330726 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b70100bb0 0x7f2b70100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:30.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.562+0000 7f2b76d26700 1 -- 172.21.15.114:0/3312330726 >> 172.21.15.114:0/3312330726 conn(0x7f2b700fc110 msgr2=0x7f2b700fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:30.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.562+0000 7f2b76d26700 1 -- 172.21.15.114:0/3312330726 shutdown_connections 2024-02-19T19:07:30.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.562+0000 7f2b76d26700 1 -- 172.21.15.114:0/3312330726 wait complete. 2024-02-19T19:07:30.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.562+0000 7f2b76d26700 1 Processor -- start 2024-02-19T19:07:30.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b76d26700 1 -- start start 2024-02-19T19:07:30.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b76d26700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70100bb0 0x7f2b70196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:30.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b76d26700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b70101db0 0x7f2b701968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:30.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b76d26700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b70196ec0 con 0x7f2b70100bb0 2024-02-19T19:07:30.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b76d26700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b70197000 con 0x7f2b70101db0 2024-02-19T19:07:30.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b74ac2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70100bb0 0x7f2b70196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:30.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b74ac2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70100bb0 0x7f2b70196360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46212/0 (socket says 172.21.15.114:46212) 2024-02-19T19:07:30.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b74ac2700 1 -- 172.21.15.114:0/2413942957 learned_addr learned my addr 172.21.15.114:0/2413942957 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:07:30.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b6ffff700 1 --2- 172.21.15.114:0/2413942957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b70101db0 0x7f2b701968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:30.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b74ac2700 1 -- 172.21.15.114:0/2413942957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b70101db0 msgr2=0x7f2b701968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:30.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b74ac2700 1 --2- 172.21.15.114:0/2413942957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b70101db0 0x7f2b701968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:30.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b74ac2700 1 -- 172.21.15.114:0/2413942957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b640095f0 con 0x7f2b70100bb0 2024-02-19T19:07:30.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b6ffff700 1 --2- 172.21.15.114:0/2413942957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b70101db0 0x7f2b701968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:07:30.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b74ac2700 1 --2- 172.21.15.114:0/2413942957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70100bb0 0x7f2b70196360 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2b68007e60 tx=0x7f2b6800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:30.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b6dffb700 1 -- 172.21.15.114:0/2413942957 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b6800dda0 con 0x7f2b70100bb0 2024-02-19T19:07:30.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b6dffb700 1 -- 172.21.15.114:0/2413942957 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2b6800f040 con 0x7f2b70100bb0 2024-02-19T19:07:30.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b76d26700 1 -- 172.21.15.114:0/2413942957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b7019bab0 con 0x7f2b70100bb0 2024-02-19T19:07:30.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.563+0000 7f2b6dffb700 1 -- 172.21.15.114:0/2413942957 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b680147d0 con 0x7f2b70100bb0 2024-02-19T19:07:30.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.564+0000 7f2b76d26700 1 -- 172.21.15.114:0/2413942957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7019c000 con 0x7f2b70100bb0 2024-02-19T19:07:30.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.565+0000 7f2b76d26700 1 -- 172.21.15.114:0/2413942957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b70109710 con 0x7f2b70100bb0 2024-02-19T19:07:30.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.565+0000 7f2b6dffb700 1 -- 172.21.15.114:0/2413942957 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f2b68014930 con 0x7f2b70100bb0 2024-02-19T19:07:30.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.565+0000 7f2b6dffb700 1 --2- 172.21.15.114:0/2413942957 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f2b5806d8a0 0x7f2b5806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:07:30.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.566+0000 7f2b6dffb700 1 -- 172.21.15.114:0/2413942957 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f2b6808d280 con 0x7f2b70100bb0 2024-02-19T19:07:30.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.566+0000 7f2b6ffff700 1 --2- 172.21.15.114:0/2413942957 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f2b5806d8a0 0x7f2b5806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:07:30.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.566+0000 7f2b6ffff700 1 --2- 172.21.15.114:0/2413942957 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f2b5806d8a0 0x7f2b5806fd60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2b64005b40 tx=0x7f2b64005a90 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:07:30.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.570+0000 7f2b6dffb700 1 -- 172.21.15.114:0/2413942957 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2b6805a510 con 0x7f2b70100bb0 2024-02-19T19:07:30.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.845+0000 7f2b76d26700 1 -- 172.21.15.114:0/2413942957 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2b7019c670 con 0x7f2b70100bb0 2024-02-19T19:07:30.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.846+0000 7f2b6dffb700 1 -- 172.21.15.114:0/2413942957 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1073 (secure 0 0 0) 0x7f2b6805a0a0 con 0x7f2b70100bb0 2024-02-19T19:07:30.867 INFO:teuthology.orchestra.run.smithi114.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down; Degraded data redundancy: 21/129 objects degraded (16.279%), 11 pgs degraded 2024-02-19T19:07:30.867 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:07:30.867 INFO:teuthology.orchestra.run.smithi114.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:07:30.867 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-19T19:07:30.867 INFO:teuthology.orchestra.run.smithi114.stdout: osd.4 (root=default,host=smithi174) is down 2024-02-19T19:07:30.867 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 21/129 objects degraded (16.279%), 11 pgs degraded 2024-02-19T19:07:30.867 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.5 is active+undersized+degraded, acting [3,0] 2024-02-19T19:07:30.867 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.6 is active+undersized+degraded, acting [1,3] 2024-02-19T19:07:30.868 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.7 is active+undersized+degraded, acting [3,2] 2024-02-19T19:07:30.868 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.9 is active+undersized+degraded, acting [1,0] 2024-02-19T19:07:30.868 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-02-19T19:07:30.868 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.b is active+undersized+degraded, acting [3,5] 2024-02-19T19:07:30.868 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.f is active+undersized+degraded, acting [0,5] 2024-02-19T19:07:30.868 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.13 is active+undersized+degraded, acting [1,2] 2024-02-19T19:07:30.868 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-02-19T19:07:30.868 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-02-19T19:07:30.868 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.1f is active+undersized+degraded, acting [1,3] 2024-02-19T19:07:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.848+0000 7f2b76d26700 1 -- 172.21.15.114:0/2413942957 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f2b5806d8a0 msgr2=0x7f2b5806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.848+0000 7f2b76d26700 1 --2- 172.21.15.114:0/2413942957 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f2b5806d8a0 0x7f2b5806fd60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2b64005b40 tx=0x7f2b64005a90 comp rx=0 tx=0).stop 2024-02-19T19:07:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.848+0000 7f2b76d26700 1 -- 172.21.15.114:0/2413942957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70100bb0 msgr2=0x7f2b70196360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:07:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.848+0000 7f2b76d26700 1 --2- 172.21.15.114:0/2413942957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70100bb0 0x7f2b70196360 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2b68007e60 tx=0x7f2b6800d3b0 comp rx=0 tx=0).stop 2024-02-19T19:07:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.848+0000 7f2b76d26700 1 -- 172.21.15.114:0/2413942957 shutdown_connections 2024-02-19T19:07:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.848+0000 7f2b76d26700 1 --2- 172.21.15.114:0/2413942957 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f2b5806d8a0 0x7f2b5806fd60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.848+0000 7f2b76d26700 1 --2- 172.21.15.114:0/2413942957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b70100bb0 0x7f2b70196360 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:30.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.848+0000 7f2b76d26700 1 --2- 172.21.15.114:0/2413942957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b70101db0 0x7f2b701968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:07:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.848+0000 7f2b76d26700 1 -- 172.21.15.114:0/2413942957 >> 172.21.15.114:0/2413942957 conn(0x7f2b700fc110 msgr2=0x7f2b70104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:07:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.848+0000 7f2b76d26700 1 -- 172.21.15.114:0/2413942957 shutdown_connections 2024-02-19T19:07:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:07:30.848+0000 7f2b76d26700 1 -- 172.21.15.114:0/2413942957 wait complete. 2024-02-19T19:07:31.205 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:30 smithi174 ceph-mon[87279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-19T19:07:31.205 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:30 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/1461696244' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:07:31.205 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:30 smithi174 ceph-mon[87279]: from='client.44249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:31.205 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:30 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/2413942957' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:07:31.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:30 smithi114 ceph-mon[92983]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-19T19:07:31.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:30 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/1461696244' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:07:31.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:30 smithi114 ceph-mon[92983]: from='client.44249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:07:31.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:30 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/2413942957' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:07:32.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:31 smithi114 ceph-mon[92983]: Health check update: Degraded data redundancy: 21/129 objects degraded (16.279%), 11 pgs degraded (PG_DEGRADED) 2024-02-19T19:07:32.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:31 smithi114 ceph-mon[92983]: pgmap v144: 65 pgs: 11 active+undersized+degraded, 21 active+undersized, 33 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 1 op/s; 21/129 objects degraded (16.279%) 2024-02-19T19:07:32.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:31 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:32.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:31 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:32.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:31 smithi174 ceph-mon[87279]: Health check update: Degraded data redundancy: 21/129 objects degraded (16.279%), 11 pgs degraded (PG_DEGRADED) 2024-02-19T19:07:32.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:31 smithi174 ceph-mon[87279]: pgmap v144: 65 pgs: 11 active+undersized+degraded, 21 active+undersized, 33 active+clean; 594 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 767 B/s rd, 1 op/s; 21/129 objects degraded (16.279%) 2024-02-19T19:07:32.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:31 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:32.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:31 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:33.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:32 smithi174 ceph-mon[87279]: from='osd.4 [v2:172.21.15.174:6808/1253972204,v1:172.21.15.174:6809/1253972204]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T19:07:33.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:32 smithi174 ceph-mon[87279]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T19:07:33.145 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:32 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4[102288]: 2024-02-19T19:07:32.694+0000 7fe32f5737c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-02-19T19:07:33.145 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:32 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4[102288]: 2024-02-19T19:07:32.718+0000 7fe32f5737c0 -1 osd.4 73 log_to_monitors true 2024-02-19T19:07:33.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:32 smithi114 ceph-mon[92983]: from='osd.4 [v2:172.21.15.174:6808/1253972204,v1:172.21.15.174:6809/1253972204]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T19:07:33.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:32 smithi114 ceph-mon[92983]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T19:07:34.145 INFO:journalctl@ceph.osd.4.smithi174.stdout:Feb 19 19:07:33 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-4[102288]: 2024-02-19T19:07:33.782+0000 7fe31c9c3700 -1 osd.4 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T19:07:34.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:33 smithi174 ceph-mon[87279]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-19T19:07:34.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:33 smithi174 ceph-mon[87279]: from='osd.4 [v2:172.21.15.174:6808/1253972204,v1:172.21.15.174:6809/1253972204]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T19:07:34.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:33 smithi174 ceph-mon[87279]: osdmap e76: 6 total, 5 up, 6 in 2024-02-19T19:07:34.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:33 smithi174 ceph-mon[87279]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T19:07:34.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:33 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:34.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:33 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:34.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:33 smithi174 ceph-mon[87279]: from='osd.4 ' entity='osd.4' 2024-02-19T19:07:34.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:33 smithi174 ceph-mon[87279]: pgmap v146: 65 pgs: 11 active+undersized+degraded, 21 active+undersized, 33 active+clean; 594 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 690 B/s rd, 2 op/s; 21/129 objects degraded (16.279%) 2024-02-19T19:07:34.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:33 smithi114 ceph-mon[92983]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-19T19:07:34.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:33 smithi114 ceph-mon[92983]: from='osd.4 [v2:172.21.15.174:6808/1253972204,v1:172.21.15.174:6809/1253972204]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T19:07:34.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:33 smithi114 ceph-mon[92983]: osdmap e76: 6 total, 5 up, 6 in 2024-02-19T19:07:34.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:33 smithi114 ceph-mon[92983]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T19:07:34.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:33 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:34.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:33 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:34.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:33 smithi114 ceph-mon[92983]: from='osd.4 ' entity='osd.4' 2024-02-19T19:07:34.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:33 smithi114 ceph-mon[92983]: pgmap v146: 65 pgs: 11 active+undersized+degraded, 21 active+undersized, 33 active+clean; 594 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 690 B/s rd, 2 op/s; 21/129 objects degraded (16.279%) 2024-02-19T19:07:34.979 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:34 smithi174 ceph-mon[87279]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:07:34.979 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:34 smithi174 ceph-mon[87279]: osd.4 [v2:172.21.15.174:6808/1253972204,v1:172.21.15.174:6809/1253972204] boot 2024-02-19T19:07:34.979 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:34 smithi174 ceph-mon[87279]: osdmap e77: 6 total, 6 up, 6 in 2024-02-19T19:07:34.979 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:34 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:07:35.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:34 smithi114 ceph-mon[92983]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:07:35.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:34 smithi114 ceph-mon[92983]: osd.4 [v2:172.21.15.174:6808/1253972204,v1:172.21.15.174:6809/1253972204] boot 2024-02-19T19:07:35.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:34 smithi114 ceph-mon[92983]: osdmap e77: 6 total, 6 up, 6 in 2024-02-19T19:07:35.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:34 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:07:36.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:35 smithi174 ceph-mon[87279]: pgmap v148: 65 pgs: 13 peering, 7 active+undersized+degraded, 12 active+undersized, 33 active+clean; 594 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s; 11/129 objects degraded (8.527%) 2024-02-19T19:07:36.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:35 smithi174 ceph-mon[87279]: osdmap e78: 6 total, 6 up, 6 in 2024-02-19T19:07:36.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:35 smithi114 ceph-mon[92983]: pgmap v148: 65 pgs: 13 peering, 7 active+undersized+degraded, 12 active+undersized, 33 active+clean; 594 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s; 11/129 objects degraded (8.527%) 2024-02-19T19:07:36.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:35 smithi114 ceph-mon[92983]: osdmap e78: 6 total, 6 up, 6 in 2024-02-19T19:07:37.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:36 smithi114 ceph-mon[92983]: Health check update: Degraded data redundancy: 11/129 objects degraded (8.527%), 7 pgs degraded (PG_DEGRADED) 2024-02-19T19:07:37.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:36 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:37.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:36 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:37.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:36 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:37.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:36 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:07:37.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:36 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:37.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:36 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:07:37.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:36 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:37.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:36 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:37.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:36 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:37.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:36 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-19T19:07:37.041 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:36 smithi114 ceph-mon[92983]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-19T19:07:37.041 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:36 smithi114 ceph-mon[92983]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-19T19:07:37.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:36 smithi174 ceph-mon[87279]: Health check update: Degraded data redundancy: 11/129 objects degraded (8.527%), 7 pgs degraded (PG_DEGRADED) 2024-02-19T19:07:37.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:36 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:37.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:36 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:37.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:36 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:37.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:36 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:07:37.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:36 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:37.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:36 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:07:37.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:36 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:37.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:36 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:37.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:36 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:07:37.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:36 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-19T19:07:37.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:36 smithi174 ceph-mon[87279]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-19T19:07:37.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:36 smithi174 ceph-mon[87279]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-19T19:07:38.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:37 smithi114 ceph-mon[92983]: pgmap v150: 65 pgs: 13 peering, 7 active+undersized+degraded, 8 active+undersized, 37 active+clean; 594 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 11/129 objects degraded (8.527%) 2024-02-19T19:07:38.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:37 smithi174 ceph-mon[87279]: pgmap v150: 65 pgs: 13 peering, 7 active+undersized+degraded, 8 active+undersized, 37 active+clean; 594 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 11/129 objects degraded (8.527%) 2024-02-19T19:07:40.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:39 smithi174 ceph-mon[87279]: pgmap v151: 65 pgs: 13 peering, 52 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s 2024-02-19T19:07:40.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:39 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:40.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:39 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:07:40.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:39 smithi174 ceph-mon[87279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/129 objects degraded (8.527%), 7 pgs degraded) 2024-02-19T19:07:40.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:39 smithi114 ceph-mon[92983]: pgmap v151: 65 pgs: 13 peering, 52 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s 2024-02-19T19:07:40.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:39 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:40.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:39 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:07:40.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:39 smithi114 ceph-mon[92983]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/129 objects degraded (8.527%), 7 pgs degraded) 2024-02-19T19:07:43.602 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:43 smithi114 ceph-mon[92983]: pgmap v152: 65 pgs: 65 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-02-19T19:07:43.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:43 smithi174 ceph-mon[87279]: pgmap v152: 65 pgs: 65 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-02-19T19:07:45.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:45 smithi174 ceph-mon[87279]: pgmap v153: 65 pgs: 65 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-02-19T19:07:45.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:45 smithi114 ceph-mon[92983]: pgmap v153: 65 pgs: 65 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-02-19T19:07:47.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:47 smithi114 ceph-mon[92983]: pgmap v154: 65 pgs: 65 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-02-19T19:07:47.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:47 smithi174 ceph-mon[87279]: pgmap v154: 65 pgs: 65 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-02-19T19:07:49.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:49 smithi174 ceph-mon[87279]: pgmap v155: 65 pgs: 65 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:07:49.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:49 smithi114 ceph-mon[92983]: pgmap v155: 65 pgs: 65 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:07:51.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:50 smithi174 ceph-mon[87279]: pgmap v156: 65 pgs: 65 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:07:51.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:50 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:51.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:50 smithi114 ceph-mon[92983]: pgmap v156: 65 pgs: 65 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:07:51.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:50 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:52.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:51 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-19T19:07:52.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:51 smithi174 ceph-mon[87279]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-19T19:07:52.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:51 smithi174 ceph-mon[87279]: Upgrade: osd.5 is safe to restart 2024-02-19T19:07:52.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:51 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-19T19:07:52.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:51 smithi114 ceph-mon[92983]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-19T19:07:52.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:51 smithi114 ceph-mon[92983]: Upgrade: osd.5 is safe to restart 2024-02-19T19:07:53.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:52 smithi174 ceph-mon[87279]: pgmap v157: 65 pgs: 65 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:07:53.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:52 smithi114 ceph-mon[92983]: pgmap v157: 65 pgs: 65 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:07:54.787 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:54 smithi174 ceph-mon[87279]: Upgrade: Updating osd.5 2024-02-19T19:07:54.787 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:54 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:54.788 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:54 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-19T19:07:54.788 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:54 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:54.788 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:54 smithi174 ceph-mon[87279]: Deploying daemon osd.5 on smithi174 2024-02-19T19:07:54.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:54 smithi114 ceph-mon[92983]: Upgrade: Updating osd.5 2024-02-19T19:07:54.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:54 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:54.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:54 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-19T19:07:54.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:54 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:07:54.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:54 smithi114 ceph-mon[92983]: Deploying daemon osd.5 on smithi174 2024-02-19T19:07:55.590 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:55 smithi174 ceph-mon[87279]: pgmap v158: 65 pgs: 65 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:07:55.590 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:55 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:55.591 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:55 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:07:55.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:55 smithi114 ceph-mon[92983]: pgmap v158: 65 pgs: 65 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-19T19:07:55.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:55 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:07:55.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:55 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:07:57.055 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:57 smithi174 systemd[1]: Stopping Ceph osd.5 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T19:07:57.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:57 smithi114 ceph-mon[92983]: pgmap v159: 65 pgs: 65 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:07:57.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:57 smithi114 ceph-mon[92983]: osd.5 marked itself down and dead 2024-02-19T19:07:57.797 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:57 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5[60160]: 2024-02-19T19:07:57.430+0000 7f468b5dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T19:07:57.798 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:57 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5[60160]: 2024-02-19T19:07:57.430+0000 7f468b5dc700 -1 osd.5 78 *** Got signal Terminated *** 2024-02-19T19:07:57.798 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:57 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5[60160]: 2024-02-19T19:07:57.430+0000 7f468b5dc700 -1 osd.5 78 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T19:07:57.798 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:57 smithi174 podman[106867]: 2024-02-19 19:07:57.776845729 +0000 UTC m=+0.527109075 container died 5771a995ed75d3981cc4e10bc1b4f4dd0bf81af72899ace1de964b80e4d52125 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214) 2024-02-19T19:07:57.798 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:57 smithi174 ceph-mon[87279]: pgmap v159: 65 pgs: 65 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-02-19T19:07:57.798 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:57 smithi174 ceph-mon[87279]: osd.5 marked itself down and dead 2024-02-19T19:07:58.395 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:58 smithi174 podman[106867]: 2024-02-19 19:07:58.203745425 +0000 UTC m=+0.954008755 container cleanup 5771a995ed75d3981cc4e10bc1b4f4dd0bf81af72899ace1de964b80e4d52125 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1) 2024-02-19T19:07:58.396 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:58 smithi174 bash[106867]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5 2024-02-19T19:07:58.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:58 smithi114 ceph-mon[92983]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:07:58.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:58 smithi114 ceph-mon[92983]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-19T19:07:58.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:58 smithi114 ceph-mon[92983]: osdmap e79: 6 total, 5 up, 6 in 2024-02-19T19:07:58.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:58 smithi114 ceph-mon[92983]: osdmap e80: 6 total, 5 up, 6 in 2024-02-19T19:07:58.867 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:58 smithi174 podman[106884]: 2024-02-19 19:07:58.396546961 +0000 UTC m=+0.616129643 container remove 5771a995ed75d3981cc4e10bc1b4f4dd0bf81af72899ace1de964b80e4d52125 (image=quay.io/ceph/ceph@sha256:6176526d5ef4047f148d47c951653dcc6b0de49091fda006d3a088e7a1f404d0, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1) 2024-02-19T19:07:58.867 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:58 smithi174 ceph-mon[87279]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-19T19:07:58.867 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:58 smithi174 ceph-mon[87279]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-19T19:07:58.867 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:58 smithi174 ceph-mon[87279]: osdmap e79: 6 total, 5 up, 6 in 2024-02-19T19:07:58.867 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:58 smithi174 ceph-mon[87279]: osdmap e80: 6 total, 5 up, 6 in 2024-02-19T19:07:59.146 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:58 smithi174 podman[106961]: 2024-02-19 19:07:58.847593633 +0000 UTC m=+0.097920753 container create 8c77413f74a17e65e1910e3230db54da76b4b20df56071c81ee546c4e20d3f7f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5) 2024-02-19T19:07:59.146 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:58 smithi174 podman[106961]: 2024-02-19 19:07:58.767839867 +0000 UTC m=+0.018166989 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:07:59.146 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:59 smithi174 podman[106961]: 2024-02-19 19:07:59.075265695 +0000 UTC m=+0.325592805 container init 8c77413f74a17e65e1910e3230db54da76b4b20df56071c81ee546c4e20d3f7f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-02-19T19:07:59.146 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:59 smithi174 podman[106961]: 2024-02-19 19:07:59.079955088 +0000 UTC m=+0.330282206 container start 8c77413f74a17e65e1910e3230db54da76b4b20df56071c81ee546c4e20d3f7f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, org.label-schema.build-date=20240214) 2024-02-19T19:07:59.481 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:59 smithi174 podman[106961]: 2024-02-19 19:07:59.175360277 +0000 UTC m=+0.425687396 container attach 8c77413f74a17e65e1910e3230db54da76b4b20df56071c81ee546c4e20d3f7f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-19T19:07:59.482 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:59 smithi174 podman[106961]: 2024-02-19 19:07:59.381008719 +0000 UTC m=+0.631335862 container died 8c77413f74a17e65e1910e3230db54da76b4b20df56071c81ee546c4e20d3f7f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-02-19T19:07:59.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:07:59 smithi114 ceph-mon[92983]: pgmap v161: 65 pgs: 14 stale+active+clean, 51 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-02-19T19:07:59.872 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:07:59 smithi174 ceph-mon[87279]: pgmap v161: 65 pgs: 14 stale+active+clean, 51 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-02-19T19:08:00.145 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:59 smithi174 podman[106987]: 2024-02-19 19:07:59.95363036 +0000 UTC m=+0.566959690 container remove 8c77413f74a17e65e1910e3230db54da76b4b20df56071c81ee546c4e20d3f7f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-02-19T19:08:00.145 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:59 smithi174 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.5.service: Deactivated successfully. 2024-02-19T19:08:00.145 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:59 smithi174 systemd[1]: Stopped Ceph osd.5 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T19:08:00.145 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:07:59 smithi174 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.5.service: Consumed 5.634s CPU time. 2024-02-19T19:08:00.577 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:00 smithi174 systemd[1]: Starting Ceph osd.5 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T19:08:00.578 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:00 smithi174 ceph-mon[87279]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-19T19:08:00.578 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:00 smithi174 ceph-mon[87279]: Health check failed: Degraded data redundancy: 4/129 objects degraded (3.101%), 2 pgs degraded (PG_DEGRADED) 2024-02-19T19:08:00.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:00 smithi114 ceph-mon[92983]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-02-19T19:08:00.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:00 smithi114 ceph-mon[92983]: Health check failed: Degraded data redundancy: 4/129 objects degraded (3.101%), 2 pgs degraded (PG_DEGRADED) 2024-02-19T19:08:00.886 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:00 smithi174 podman[107082]: 2024-02-19 19:08:00.45977565 +0000 UTC m=+0.023324006 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:08:00.886 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:00 smithi174 podman[107082]: 2024-02-19 19:08:00.563402992 +0000 UTC m=+0.126951355 container create e1912521b2e67370670d61125d58afda96dc4645fd7c71e646859ce8b6e48e3d (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-activate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T19:08:00.887 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:00 smithi174 podman[107082]: 2024-02-19 19:08:00.782800552 +0000 UTC m=+0.346348920 container init e1912521b2e67370670d61125d58afda96dc4645fd7c71e646859ce8b6e48e3d (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:08:00.887 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:00 smithi174 podman[107082]: 2024-02-19 19:08:00.787533083 +0000 UTC m=+0.351081450 container start e1912521b2e67370670d61125d58afda96dc4645fd7c71e646859ce8b6e48e3d (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-19T19:08:00.887 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:00 smithi174 podman[107082]: 2024-02-19 19:08:00.866117493 +0000 UTC m=+0.429665858 container attach e1912521b2e67370670d61125d58afda96dc4645fd7c71e646859ce8b6e48e3d (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240214) 2024-02-19T19:08:00.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.963+0000 7f2b5e9b4700 1 -- 172.21.15.114:0/2416890831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b58100cc0 msgr2=0x7f2b58101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:00.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.963+0000 7f2b5e9b4700 1 --2- 172.21.15.114:0/2416890831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b58100cc0 0x7f2b58101120 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c009910 tx=0x7f2b4c009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.963+0000 7f2b5e9b4700 1 -- 172.21.15.114:0/2416890831 shutdown_connections 2024-02-19T19:08:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.963+0000 7f2b5e9b4700 1 --2- 172.21.15.114:0/2416890831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b58100cc0 0x7f2b58101120 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.963+0000 7f2b5e9b4700 1 --2- 172.21.15.114:0/2416890831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b580ffac0 0x7f2b580ffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.963+0000 7f2b5e9b4700 1 -- 172.21.15.114:0/2416890831 >> 172.21.15.114:0/2416890831 conn(0x7f2b580fb060 msgr2=0x7f2b580fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.963+0000 7f2b5e9b4700 1 -- 172.21.15.114:0/2416890831 shutdown_connections 2024-02-19T19:08:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.963+0000 7f2b5e9b4700 1 -- 172.21.15.114:0/2416890831 wait complete. 2024-02-19T19:08:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.964+0000 7f2b5e9b4700 1 Processor -- start 2024-02-19T19:08:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.964+0000 7f2b5e9b4700 1 -- start start 2024-02-19T19:08:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.964+0000 7f2b5e9b4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b580ffac0 0x7f2b58107a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.964+0000 7f2b5e9b4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b58100cc0 0x7f2b581060c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.964+0000 7f2b5e9b4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b58199fc0 con 0x7f2b580ffac0 2024-02-19T19:08:00.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.964+0000 7f2b5e9b4700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5819a100 con 0x7f2b58100cc0 2024-02-19T19:08:00.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.964+0000 7f2b57fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b580ffac0 0x7f2b58107a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:00.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.964+0000 7f2b57fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b580ffac0 0x7f2b58107a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37292/0 (socket says 172.21.15.114:37292) 2024-02-19T19:08:00.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.964+0000 7f2b57fff700 1 -- 172.21.15.114:0/3350869587 learned_addr learned my addr 172.21.15.114:0/3350869587 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:08:00.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.964+0000 7f2b577fe700 1 --2- 172.21.15.114:0/3350869587 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b58100cc0 0x7f2b581060c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:00.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.965+0000 7f2b57fff700 1 -- 172.21.15.114:0/3350869587 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b58100cc0 msgr2=0x7f2b581060c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:00.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.965+0000 7f2b57fff700 1 --2- 172.21.15.114:0/3350869587 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b58100cc0 0x7f2b581060c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:00.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.965+0000 7f2b57fff700 1 -- 172.21.15.114:0/3350869587 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4c0095f0 con 0x7f2b580ffac0 2024-02-19T19:08:00.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.965+0000 7f2b57fff700 1 --2- 172.21.15.114:0/3350869587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b580ffac0 0x7f2b58107a70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2b4800c910 tx=0x7f2b4800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:00.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.965+0000 7f2b557fa700 1 -- 172.21.15.114:0/3350869587 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b480041f0 con 0x7f2b580ffac0 2024-02-19T19:08:00.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.965+0000 7f2b557fa700 1 -- 172.21.15.114:0/3350869587 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2b4800ce60 con 0x7f2b580ffac0 2024-02-19T19:08:00.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.965+0000 7f2b5e9b4700 1 -- 172.21.15.114:0/3350869587 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b58106600 con 0x7f2b580ffac0 2024-02-19T19:08:00.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.965+0000 7f2b557fa700 1 -- 172.21.15.114:0/3350869587 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b48003d20 con 0x7f2b580ffac0 2024-02-19T19:08:00.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.965+0000 7f2b5e9b4700 1 -- 172.21.15.114:0/3350869587 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b58106b20 con 0x7f2b580ffac0 2024-02-19T19:08:00.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.966+0000 7f2b5e9b4700 1 -- 172.21.15.114:0/3350869587 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5804ead0 con 0x7f2b580ffac0 2024-02-19T19:08:00.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.966+0000 7f2b557fa700 1 -- 172.21.15.114:0/3350869587 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f2b4800f450 con 0x7f2b580ffac0 2024-02-19T19:08:00.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.967+0000 7f2b557fa700 1 --2- 172.21.15.114:0/3350869587 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f2b4006d8a0 0x7f2b4006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:00.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.967+0000 7f2b557fa700 1 -- 172.21.15.114:0/3350869587 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f2b4808d2e0 con 0x7f2b580ffac0 2024-02-19T19:08:00.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.967+0000 7f2b577fe700 1 --2- 172.21.15.114:0/3350869587 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f2b4006d8a0 0x7f2b4006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:00.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.967+0000 7f2b577fe700 1 --2- 172.21.15.114:0/3350869587 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f2b4006d8a0 0x7f2b4006fd60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c006010 tx=0x7f2b4c00b560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:00.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:00.972+0000 7f2b557fa700 1 -- 172.21.15.114:0/3350869587 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2b4805a4f0 con 0x7f2b580ffac0 2024-02-19T19:08:01.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.185+0000 7f2b5e9b4700 1 -- 172.21.15.114:0/3350869587 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b58105610 con 0x7f2b4006d8a0 2024-02-19T19:08:01.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.186+0000 7f2b557fa700 1 -- 172.21.15.114:0/3350869587 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f2b58105610 con 0x7f2b4006d8a0 2024-02-19T19:08:01.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.188+0000 7f2b5e9b4700 1 -- 172.21.15.114:0/3350869587 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f2b4006d8a0 msgr2=0x7f2b4006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.188+0000 7f2b5e9b4700 1 --2- 172.21.15.114:0/3350869587 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f2b4006d8a0 0x7f2b4006fd60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c006010 tx=0x7f2b4c00b560 comp rx=0 tx=0).stop 2024-02-19T19:08:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.188+0000 7f2b5e9b4700 1 -- 172.21.15.114:0/3350869587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b580ffac0 msgr2=0x7f2b58107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.188+0000 7f2b5e9b4700 1 --2- 172.21.15.114:0/3350869587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b580ffac0 0x7f2b58107a70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2b4800c910 tx=0x7f2b4800cc20 comp rx=0 tx=0).stop 2024-02-19T19:08:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.188+0000 7f2b5e9b4700 1 -- 172.21.15.114:0/3350869587 shutdown_connections 2024-02-19T19:08:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.188+0000 7f2b5e9b4700 1 --2- 172.21.15.114:0/3350869587 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f2b4006d8a0 0x7f2b4006fd60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.188+0000 7f2b5e9b4700 1 --2- 172.21.15.114:0/3350869587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2b580ffac0 0x7f2b58107a70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.188+0000 7f2b5e9b4700 1 --2- 172.21.15.114:0/3350869587 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b58100cc0 0x7f2b581060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.188+0000 7f2b5e9b4700 1 -- 172.21.15.114:0/3350869587 >> 172.21.15.114:0/3350869587 conn(0x7f2b580fb060 msgr2=0x7f2b58103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.188+0000 7f2b5e9b4700 1 -- 172.21.15.114:0/3350869587 shutdown_connections 2024-02-19T19:08:01.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.188+0000 7f2b5e9b4700 1 -- 172.21.15.114:0/3350869587 wait complete. 2024-02-19T19:08:01.219 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-02-19T19:08:01.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.293+0000 7ff91fc9f700 1 -- 172.21.15.114:0/2888067746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9181016c0 msgr2=0x7ff918103ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:01.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.293+0000 7ff91fc9f700 1 --2- 172.21.15.114:0/2888067746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9181016c0 0x7ff918103ab0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff90c009910 tx=0x7ff90c009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:01.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.293+0000 7ff91fc9f700 1 -- 172.21.15.114:0/2888067746 shutdown_connections 2024-02-19T19:08:01.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.293+0000 7ff91fc9f700 1 --2- 172.21.15.114:0/2888067746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff918103ff0 0x7ff9181063e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.293+0000 7ff91fc9f700 1 --2- 172.21.15.114:0/2888067746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9181016c0 0x7ff918103ab0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.293+0000 7ff91fc9f700 1 -- 172.21.15.114:0/2888067746 >> 172.21.15.114:0/2888067746 conn(0x7ff9180fb030 msgr2=0x7ff9180fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.293+0000 7ff91fc9f700 1 -- 172.21.15.114:0/2888067746 shutdown_connections 2024-02-19T19:08:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.293+0000 7ff91fc9f700 1 -- 172.21.15.114:0/2888067746 wait complete. 2024-02-19T19:08:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.294+0000 7ff91fc9f700 1 Processor -- start 2024-02-19T19:08:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.294+0000 7ff91fc9f700 1 -- start start 2024-02-19T19:08:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.294+0000 7ff91fc9f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9181016c0 0x7ff918198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.294+0000 7ff91fc9f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff918103ff0 0x7ff918198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:01.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.294+0000 7ff91fc9f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff918199060 con 0x7ff918103ff0 2024-02-19T19:08:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.294+0000 7ff91fc9f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9181991a0 con 0x7ff9181016c0 2024-02-19T19:08:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.294+0000 7ff91d23a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff918103ff0 0x7ff918198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.294+0000 7ff91d23a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff918103ff0 0x7ff918198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37314/0 (socket says 172.21.15.114:37314) 2024-02-19T19:08:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.294+0000 7ff91d23a700 1 -- 172.21.15.114:0/360644250 learned_addr learned my addr 172.21.15.114:0/360644250 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:08:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.294+0000 7ff91da3b700 1 --2- 172.21.15.114:0/360644250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9181016c0 0x7ff918198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.294+0000 7ff91d23a700 1 -- 172.21.15.114:0/360644250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9181016c0 msgr2=0x7ff918198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:01.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.294+0000 7ff91d23a700 1 --2- 172.21.15.114:0/360644250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9181016c0 0x7ff918198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.294+0000 7ff91d23a700 1 -- 172.21.15.114:0/360644250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90c0095f0 con 0x7ff918103ff0 2024-02-19T19:08:01.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.294+0000 7ff91d23a700 1 --2- 172.21.15.114:0/360644250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff918103ff0 0x7ff918198a40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff91400d8d0 tx=0x7ff91400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:01.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.294+0000 7ff90affd700 1 -- 172.21.15.114:0/360644250 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9140098a0 con 0x7ff918103ff0 2024-02-19T19:08:01.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.294+0000 7ff90affd700 1 -- 172.21.15.114:0/360644250 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff914010460 con 0x7ff918103ff0 2024-02-19T19:08:01.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.295+0000 7ff91fc9f700 1 -- 172.21.15.114:0/360644250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff91819dc50 con 0x7ff918103ff0 2024-02-19T19:08:01.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.295+0000 7ff90affd700 1 -- 172.21.15.114:0/360644250 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff91400f5f0 con 0x7ff918103ff0 2024-02-19T19:08:01.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.295+0000 7ff91fc9f700 1 -- 172.21.15.114:0/360644250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff91819e1a0 con 0x7ff918103ff0 2024-02-19T19:08:01.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.295+0000 7ff91fc9f700 1 -- 172.21.15.114:0/360644250 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9180fcc20 con 0x7ff918103ff0 2024-02-19T19:08:01.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.296+0000 7ff90affd700 1 -- 172.21.15.114:0/360644250 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7ff914009a00 con 0x7ff918103ff0 2024-02-19T19:08:01.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.296+0000 7ff90affd700 1 --2- 172.21.15.114:0/360644250 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff90406d8a0 0x7ff90406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:01.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.296+0000 7ff90affd700 1 -- 172.21.15.114:0/360644250 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6790+0+0 (secure 0 0 0) 0x7ff91408d490 con 0x7ff918103ff0 2024-02-19T19:08:01.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.296+0000 7ff91da3b700 1 --2- 172.21.15.114:0/360644250 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff90406d8a0 0x7ff90406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:01.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.297+0000 7ff91da3b700 1 --2- 172.21.15.114:0/360644250 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff90406d8a0 0x7ff90406fd60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff90c006010 tx=0x7ff90c00b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:01.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.301+0000 7ff90affd700 1 -- 172.21.15.114:0/360644250 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff91405a6a0 con 0x7ff918103ff0 2024-02-19T19:08:01.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.516+0000 7ff91fc9f700 1 -- 172.21.15.114:0/360644250 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9180611d0 con 0x7ff90406d8a0 2024-02-19T19:08:01.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.517+0000 7ff90affd700 1 -- 172.21.15.114:0/360644250 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7ff9180611d0 con 0x7ff90406d8a0 2024-02-19T19:08:01.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.520+0000 7ff91fc9f700 1 -- 172.21.15.114:0/360644250 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff90406d8a0 msgr2=0x7ff90406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:01.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.520+0000 7ff91fc9f700 1 --2- 172.21.15.114:0/360644250 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff90406d8a0 0x7ff90406fd60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff90c006010 tx=0x7ff90c00b540 comp rx=0 tx=0).stop 2024-02-19T19:08:01.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.520+0000 7ff91fc9f700 1 -- 172.21.15.114:0/360644250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff918103ff0 msgr2=0x7ff918198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:01.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.520+0000 7ff91fc9f700 1 --2- 172.21.15.114:0/360644250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff918103ff0 0x7ff918198a40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff91400d8d0 tx=0x7ff91400dbe0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.520+0000 7ff91fc9f700 1 -- 172.21.15.114:0/360644250 shutdown_connections 2024-02-19T19:08:01.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.520+0000 7ff91fc9f700 1 --2- 172.21.15.114:0/360644250 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7ff90406d8a0 0x7ff90406fd60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.520+0000 7ff91fc9f700 1 --2- 172.21.15.114:0/360644250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9181016c0 0x7ff918198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.520+0000 7ff91fc9f700 1 --2- 172.21.15.114:0/360644250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff918103ff0 0x7ff918198a40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.520+0000 7ff91fc9f700 1 -- 172.21.15.114:0/360644250 >> 172.21.15.114:0/360644250 conn(0x7ff9180fb030 msgr2=0x7ff9180ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:01.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.520+0000 7ff91fc9f700 1 -- 172.21.15.114:0/360644250 shutdown_connections 2024-02-19T19:08:01.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.520+0000 7ff91fc9f700 1 -- 172.21.15.114:0/360644250 wait complete. 2024-02-19T19:08:01.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.623+0000 7f31fef70700 1 -- 172.21.15.114:0/1683067469 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31f8100bc0 msgr2=0x7f31f8100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:01.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.623+0000 7f31fef70700 1 --2- 172.21.15.114:0/1683067469 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31f8100bc0 0x7f31f8100fe0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009910 tx=0x7f31ec009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:01.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.623+0000 7f31fef70700 1 -- 172.21.15.114:0/1683067469 shutdown_connections 2024-02-19T19:08:01.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.623+0000 7f31fef70700 1 --2- 172.21.15.114:0/1683067469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31f8101dc0 0x7f31f8102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.623+0000 7f31fef70700 1 --2- 172.21.15.114:0/1683067469 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31f8100bc0 0x7f31f8100fe0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.623+0000 7f31fef70700 1 -- 172.21.15.114:0/1683067469 >> 172.21.15.114:0/1683067469 conn(0x7f31f80fc140 msgr2=0x7f31f80fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:01.645 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:01 smithi114 ceph-mon[92983]: pgmap v163: 65 pgs: 2 active+undersized+degraded, 12 peering, 5 stale+active+clean, 3 active+undersized, 43 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 4/129 objects degraded (3.101%) 2024-02-19T19:08:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.623+0000 7f31fef70700 1 -- 172.21.15.114:0/1683067469 shutdown_connections 2024-02-19T19:08:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.623+0000 7f31fef70700 1 -- 172.21.15.114:0/1683067469 wait complete. 2024-02-19T19:08:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.623+0000 7f31fef70700 1 Processor -- start 2024-02-19T19:08:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.623+0000 7f31fef70700 1 -- start start 2024-02-19T19:08:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.624+0000 7f31fef70700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31f8100bc0 0x7f31f8196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.624+0000 7f31fef70700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31f8101dc0 0x7f31f81968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:01.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.624+0000 7f31fef70700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f8196ef0 con 0x7f31f8100bc0 2024-02-19T19:08:01.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.624+0000 7f31fef70700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f8197030 con 0x7f31f8101dc0 2024-02-19T19:08:01.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.624+0000 7f31fcd0c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31f8100bc0 0x7f31f8196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:01.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.624+0000 7f31fcd0c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31f8100bc0 0x7f31f8196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37330/0 (socket says 172.21.15.114:37330) 2024-02-19T19:08:01.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.624+0000 7f31fcd0c700 1 -- 172.21.15.114:0/3893128476 learned_addr learned my addr 172.21.15.114:0/3893128476 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:08:01.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.624+0000 7f31f7fff700 1 --2- 172.21.15.114:0/3893128476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31f8101dc0 0x7f31f81968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:01.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.624+0000 7f31fcd0c700 1 -- 172.21.15.114:0/3893128476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31f8101dc0 msgr2=0x7f31f81968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:01.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.624+0000 7f31fcd0c700 1 --2- 172.21.15.114:0/3893128476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31f8101dc0 0x7f31f81968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.624+0000 7f31fcd0c700 1 -- 172.21.15.114:0/3893128476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec0095f0 con 0x7f31f8100bc0 2024-02-19T19:08:01.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.624+0000 7f31fcd0c700 1 --2- 172.21.15.114:0/3893128476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31f8100bc0 0x7f31f8196390 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f31ec000c00 tx=0x7f31ec00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:01.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.624+0000 7f31f5ffb700 1 -- 172.21.15.114:0/3893128476 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31ec01d070 con 0x7f31f8100bc0 2024-02-19T19:08:01.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.624+0000 7f31f5ffb700 1 -- 172.21.15.114:0/3893128476 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f31ec005dc0 con 0x7f31f8100bc0 2024-02-19T19:08:01.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.624+0000 7f31fef70700 1 -- 172.21.15.114:0/3893128476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f819ba80 con 0x7f31f8100bc0 2024-02-19T19:08:01.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.624+0000 7f31f5ffb700 1 -- 172.21.15.114:0/3893128476 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31ec022470 con 0x7f31f8100bc0 2024-02-19T19:08:01.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.625+0000 7f31fef70700 1 -- 172.21.15.114:0/3893128476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f819bf70 con 0x7f31f8100bc0 2024-02-19T19:08:01.648 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:01 smithi174 ceph-mon[87279]: pgmap v163: 65 pgs: 2 active+undersized+degraded, 12 peering, 5 stale+active+clean, 3 active+undersized, 43 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 4/129 objects degraded (3.101%) 2024-02-19T19:08:01.648 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:01 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-activate[107096]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-19T19:08:01.649 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:01 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-activate[107096]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-19T19:08:01.649 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:01 smithi174 bash[107082]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-19T19:08:01.649 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:01 smithi174 bash[107082]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-19T19:08:01.649 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:01 smithi174 bash[107082]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-19T19:08:01.649 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:01 smithi174 bash[107082]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-19T19:08:01.649 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:01 smithi174 bash[107082]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-02-19T19:08:01.649 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:01 smithi174 bash[107082]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-19T19:08:01.649 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:01 smithi174 bash[107082]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-19T19:08:01.649 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:01 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-activate[107096]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-19T19:08:01.649 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:01 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-activate[107096]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-19T19:08:01.649 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:01 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-activate[107096]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-02-19T19:08:01.649 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:01 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-activate[107096]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-19T19:08:01.649 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:01 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-activate[107096]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-19T19:08:01.649 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:01 smithi174 podman[107082]: 2024-02-19 19:08:01.391134975 +0000 UTC m=+0.954683340 container died e1912521b2e67370670d61125d58afda96dc4645fd7c71e646859ce8b6e48e3d (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-activate, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214) 2024-02-19T19:08:01.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.626+0000 7f31fef70700 1 -- 172.21.15.114:0/3893128476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f8109720 con 0x7f31f8100bc0 2024-02-19T19:08:01.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.626+0000 7f31f5ffb700 1 -- 172.21.15.114:0/3893128476 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f31ec017400 con 0x7f31f8100bc0 2024-02-19T19:08:01.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.626+0000 7f31f5ffb700 1 --2- 172.21.15.114:0/3893128476 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f31e806d8a0 0x7f31e806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:01.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.627+0000 7f31f5ffb700 1 -- 172.21.15.114:0/3893128476 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f31ec08eab0 con 0x7f31f8100bc0 2024-02-19T19:08:01.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.627+0000 7f31f7fff700 1 --2- 172.21.15.114:0/3893128476 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f31e806d8a0 0x7f31e806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:01.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.627+0000 7f31f7fff700 1 --2- 172.21.15.114:0/3893128476 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f31e806d8a0 0x7f31e806fd60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f31e4009c60 tx=0x7f31e4009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:01.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.631+0000 7f31f5ffb700 1 -- 172.21.15.114:0/3893128476 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f31ec05bdb0 con 0x7f31f8100bc0 2024-02-19T19:08:01.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.838+0000 7f31fef70700 1 -- 172.21.15.114:0/3893128476 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f31f8106710 con 0x7f31e806d8a0 2024-02-19T19:08:01.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.848+0000 7f31f5ffb700 1 -- 172.21.15.114:0/3893128476 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f31f8106710 con 0x7f31e806d8a0 2024-02-19T19:08:01.869 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:08:01.869 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (12m) 96s ago 14m 26.5M - 0.25.0 c8568f914cd2 30444b676ca4 2024-02-19T19:08:01.869 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (15m) 96s ago 15m 8472k - 18.2.1 94eadd990e0a 4918cd87e312 2024-02-19T19:08:01.870 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi174 smithi174 running (9m) 30s ago 12m 8212k - 18.2.1 94eadd990e0a 516c34431b93 2024-02-19T19:08:01.870 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (3m) 96s ago 15m 6660k - 19.0.0-1450-gf78a58c0 f081f847070a f10b46e9d636 2024-02-19T19:08:01.870 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi174 smithi174 running (3m) 30s ago 12m 6660k - 19.0.0-1450-gf78a58c0 f081f847070a fb3b17c81bd8 2024-02-19T19:08:01.870 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (11m) 96s ago 13m 85.5M - 9.4.7 954c08fa6188 478bd533e827 2024-02-19T19:08:01.870 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tcwpod smithi114 running (7m) 96s ago 7m 27.5M - 18.2.1 94eadd990e0a 420175777cf4 2024-02-19T19:08:01.870 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.vpoeop smithi114 running (7m) 96s ago 7m 24.6M - 18.2.1 94eadd990e0a f81a79e53ead 2024-02-19T19:08:01.870 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.stnfnf smithi174 running (7m) 30s ago 7m 26.4M - 18.2.1 94eadd990e0a 3df35954a492 2024-02-19T19:08:01.870 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.znhuvd smithi174 running (7m) 30s ago 7m 25.5M - 18.2.1 94eadd990e0a 2d2267494ae1 2024-02-19T19:08:01.870 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.pcvtrk smithi114 *:8443,9283,8765 running (5m) 96s ago 16m 503M - 19.0.0-1450-gf78a58c0 f081f847070a 05b5c06122b1 2024-02-19T19:08:01.870 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi174.lydvnj smithi174 *:8443,9283,8765 running (4m) 30s ago 12m 452M - 19.0.0-1450-gf78a58c0 f081f847070a f962b1af5aed 2024-02-19T19:08:01.870 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (4m) 96s ago 16m 48.1M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e51f650f9be9 2024-02-19T19:08:01.870 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi174 smithi174 running (4m) 30s ago 12m 36.6M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e811581f05fb 2024-02-19T19:08:01.870 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (14m) 96s ago 14m 14.3M - 1.5.0 0da6a335fe13 b6b258074cce 2024-02-19T19:08:01.870 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi174 smithi174 *:9100 running (12m) 30s ago 12m 15.0M - 1.5.0 0da6a335fe13 d0b4fdf67b22 2024-02-19T19:08:01.871 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (2m) 96s ago 11m 51.2M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 764769439e0e 2024-02-19T19:08:01.871 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (2m) 96s ago 11m 51.8M 1216M 19.0.0-1450-gf78a58c0 f081f847070a c1c32dbdf573 2024-02-19T19:08:01.871 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (101s) 96s ago 10m 11.5M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 11e64d20e378 2024-02-19T19:08:01.871 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi174 running (66s) 30s ago 10m 44.5M 2240M 19.0.0-1450-gf78a58c0 f081f847070a cc3a55bd908f 2024-02-19T19:08:01.871 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi174 running (33s) 30s ago 9m 12.4M 2240M 19.0.0-1450-gf78a58c0 f081f847070a b3609ea94466 2024-02-19T19:08:01.871 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi174 running (9m) 30s ago 9m 55.2M 2240M 18.2.1 94eadd990e0a 5771a995ed75 2024-02-19T19:08:01.871 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (4m) 96s ago 13m 63.5M - 2.43.0 a07b618ecd1d a36922c7c9a1 2024-02-19T19:08:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.851+0000 7f31fef70700 1 -- 172.21.15.114:0/3893128476 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f31e806d8a0 msgr2=0x7f31e806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.851+0000 7f31fef70700 1 --2- 172.21.15.114:0/3893128476 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f31e806d8a0 0x7f31e806fd60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f31e4009c60 tx=0x7f31e4009260 comp rx=0 tx=0).stop 2024-02-19T19:08:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.851+0000 7f31fef70700 1 -- 172.21.15.114:0/3893128476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31f8100bc0 msgr2=0x7f31f8196390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.851+0000 7f31fef70700 1 --2- 172.21.15.114:0/3893128476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31f8100bc0 0x7f31f8196390 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f31ec000c00 tx=0x7f31ec00f690 comp rx=0 tx=0).stop 2024-02-19T19:08:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.851+0000 7f31fef70700 1 -- 172.21.15.114:0/3893128476 shutdown_connections 2024-02-19T19:08:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.851+0000 7f31fef70700 1 --2- 172.21.15.114:0/3893128476 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f31e806d8a0 0x7f31e806fd60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.851+0000 7f31fef70700 1 --2- 172.21.15.114:0/3893128476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31f8100bc0 0x7f31f8196390 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.851+0000 7f31fef70700 1 --2- 172.21.15.114:0/3893128476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31f8101dc0 0x7f31f81968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.851+0000 7f31fef70700 1 -- 172.21.15.114:0/3893128476 >> 172.21.15.114:0/3893128476 conn(0x7f31f80fc140 msgr2=0x7f31f8104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.851+0000 7f31fef70700 1 -- 172.21.15.114:0/3893128476 shutdown_connections 2024-02-19T19:08:01.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.851+0000 7f31fef70700 1 -- 172.21.15.114:0/3893128476 wait complete. 2024-02-19T19:08:01.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.953+0000 7fe0aaf6c700 1 -- 172.21.15.114:0/1958354169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0a4074d60 msgr2=0x7fe0a40731c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:01.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.953+0000 7fe0aaf6c700 1 --2- 172.21.15.114:0/1958354169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0a4074d60 0x7fe0a40731c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe098009910 tx=0x7fe098009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:01.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.953+0000 7fe0aaf6c700 1 -- 172.21.15.114:0/1958354169 shutdown_connections 2024-02-19T19:08:01.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.953+0000 7fe0aaf6c700 1 --2- 172.21.15.114:0/1958354169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0a4073790 0x7fe0a4073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.953+0000 7fe0aaf6c700 1 --2- 172.21.15.114:0/1958354169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0a4074d60 0x7fe0a40731c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.953+0000 7fe0aaf6c700 1 -- 172.21.15.114:0/1958354169 >> 172.21.15.114:0/1958354169 conn(0x7fe0a40fc140 msgr2=0x7fe0a40fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:01.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.953+0000 7fe0aaf6c700 1 -- 172.21.15.114:0/1958354169 shutdown_connections 2024-02-19T19:08:01.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.953+0000 7fe0aaf6c700 1 -- 172.21.15.114:0/1958354169 wait complete. 2024-02-19T19:08:01.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.954+0000 7fe0aaf6c700 1 Processor -- start 2024-02-19T19:08:01.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.954+0000 7fe0aaf6c700 1 -- start start 2024-02-19T19:08:01.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.954+0000 7fe0aaf6c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0a4073790 0x7fe0a4198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:01.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.954+0000 7fe0aaf6c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0a4074d60 0x7fe0a4198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:01.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.954+0000 7fe0aaf6c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a41990e0 con 0x7fe0a4074d60 2024-02-19T19:08:01.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.954+0000 7fe0aaf6c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a4199220 con 0x7fe0a4073790 2024-02-19T19:08:01.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.954+0000 7fe0a3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0a4074d60 0x7fe0a4198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:01.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.954+0000 7fe0a3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0a4074d60 0x7fe0a4198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37358/0 (socket says 172.21.15.114:37358) 2024-02-19T19:08:01.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.954+0000 7fe0a3fff700 1 -- 172.21.15.114:0/2755604482 learned_addr learned my addr 172.21.15.114:0/2755604482 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:08:01.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.954+0000 7fe0a8d08700 1 --2- 172.21.15.114:0/2755604482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0a4073790 0x7fe0a4198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:01.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.954+0000 7fe0a3fff700 1 -- 172.21.15.114:0/2755604482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0a4073790 msgr2=0x7fe0a4198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:01.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.954+0000 7fe0a3fff700 1 --2- 172.21.15.114:0/2755604482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0a4073790 0x7fe0a4198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:01.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.955+0000 7fe0a3fff700 1 -- 172.21.15.114:0/2755604482 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0980095f0 con 0x7fe0a4074d60 2024-02-19T19:08:01.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.955+0000 7fe0a3fff700 1 --2- 172.21.15.114:0/2755604482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0a4074d60 0x7fe0a4198ac0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe09000d8d0 tx=0x7fe09000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:01.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.955+0000 7fe0a1ffb700 1 -- 172.21.15.114:0/2755604482 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0900098a0 con 0x7fe0a4074d60 2024-02-19T19:08:01.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.955+0000 7fe0a1ffb700 1 -- 172.21.15.114:0/2755604482 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe090010460 con 0x7fe0a4074d60 2024-02-19T19:08:01.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.955+0000 7fe0a1ffb700 1 -- 172.21.15.114:0/2755604482 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe09000f5f0 con 0x7fe0a4074d60 2024-02-19T19:08:01.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.955+0000 7fe0aaf6c700 1 -- 172.21.15.114:0/2755604482 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a419dcd0 con 0x7fe0a4074d60 2024-02-19T19:08:01.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.955+0000 7fe0aaf6c700 1 -- 172.21.15.114:0/2755604482 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a419e160 con 0x7fe0a4074d60 2024-02-19T19:08:01.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.956+0000 7fe0aaf6c700 1 -- 172.21.15.114:0/2755604482 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a410b930 con 0x7fe0a4074d60 2024-02-19T19:08:01.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.957+0000 7fe0a1ffb700 1 -- 172.21.15.114:0/2755604482 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7fe090009a00 con 0x7fe0a4074d60 2024-02-19T19:08:01.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.957+0000 7fe0a1ffb700 1 --2- 172.21.15.114:0/2755604482 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fe09406d8a0 0x7fe09406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:01.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.957+0000 7fe0a1ffb700 1 -- 172.21.15.114:0/2755604482 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6790+0+0 (secure 0 0 0) 0x7fe09008d3e0 con 0x7fe0a4074d60 2024-02-19T19:08:01.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.957+0000 7fe0a8d08700 1 --2- 172.21.15.114:0/2755604482 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fe09406d8a0 0x7fe09406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:01.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.958+0000 7fe0a8d08700 1 --2- 172.21.15.114:0/2755604482 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fe09406d8a0 0x7fe09406fd60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe098006010 tx=0x7fe09800b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:01.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:01.962+0000 7fe0a1ffb700 1 -- 172.21.15.114:0/2755604482 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe09005a670 con 0x7fe0a4074d60 2024-02-19T19:08:02.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.239+0000 7fe0aaf6c700 1 -- 172.21.15.114:0/2755604482 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe0a410bb20 con 0x7fe0a4074d60 2024-02-19T19:08:02.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.240+0000 7fe0a1ffb700 1 -- 172.21.15.114:0/2755604482 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+700 (secure 0 0 0) 0x7fe09005a200 con 0x7fe0a4074d60 2024-02-19T19:08:02.261 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:08:02.261 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-02-19T19:08:02.261 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:08:02.261 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:08:02.261 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-02-19T19:08:02.261 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:08:02.261 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:08:02.261 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-02-19T19:08:02.262 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 5 2024-02-19T19:08:02.262 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:08:02.262 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-02-19T19:08:02.262 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:08:02.262 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:08:02.262 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-02-19T19:08:02.262 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4, 2024-02-19T19:08:02.262 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 9 2024-02-19T19:08:02.262 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-02-19T19:08:02.262 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:08:02.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.242+0000 7fe0aaf6c700 1 -- 172.21.15.114:0/2755604482 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fe09406d8a0 msgr2=0x7fe09406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:02.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.242+0000 7fe0aaf6c700 1 --2- 172.21.15.114:0/2755604482 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fe09406d8a0 0x7fe09406fd60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe098006010 tx=0x7fe09800b540 comp rx=0 tx=0).stop 2024-02-19T19:08:02.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.242+0000 7fe0aaf6c700 1 -- 172.21.15.114:0/2755604482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0a4074d60 msgr2=0x7fe0a4198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:02.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.242+0000 7fe0aaf6c700 1 --2- 172.21.15.114:0/2755604482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0a4074d60 0x7fe0a4198ac0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe09000d8d0 tx=0x7fe09000dbe0 comp rx=0 tx=0).stop 2024-02-19T19:08:02.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.242+0000 7fe0aaf6c700 1 -- 172.21.15.114:0/2755604482 shutdown_connections 2024-02-19T19:08:02.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.242+0000 7fe0aaf6c700 1 --2- 172.21.15.114:0/2755604482 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fe09406d8a0 0x7fe09406fd60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:02.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.242+0000 7fe0aaf6c700 1 --2- 172.21.15.114:0/2755604482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0a4073790 0x7fe0a4198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:02.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.242+0000 7fe0aaf6c700 1 --2- 172.21.15.114:0/2755604482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0a4074d60 0x7fe0a4198ac0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:02.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.242+0000 7fe0aaf6c700 1 -- 172.21.15.114:0/2755604482 >> 172.21.15.114:0/2755604482 conn(0x7fe0a40fc140 msgr2=0x7fe0a4106a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:02.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.242+0000 7fe0aaf6c700 1 -- 172.21.15.114:0/2755604482 shutdown_connections 2024-02-19T19:08:02.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.242+0000 7fe0aaf6c700 1 -- 172.21.15.114:0/2755604482 wait complete. 2024-02-19T19:08:02.347 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:01 smithi174 podman[107217]: 2024-02-19 19:08:01.986454167 +0000 UTC m=+0.588530363 container remove e1912521b2e67370670d61125d58afda96dc4645fd7c71e646859ce8b6e48e3d (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T19:08:02.348 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:02 smithi174 podman[107270]: 2024-02-19 19:08:02.123445029 +0000 UTC m=+0.019230800 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T19:08:02.348 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:02 smithi174 podman[107270]: 2024-02-19 19:08:02.229015846 +0000 UTC m=+0.124801618 container create c1e52f2040c2ef090c9f440d3996d67c046f5dc9cb18f80786f6cb5bc312b0a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0) 2024-02-19T19:08:02.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.346+0000 7f59501b4700 1 -- 172.21.15.114:0/3223263759 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5948100bb0 msgr2=0x7f5948100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:02.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.346+0000 7f59501b4700 1 --2- 172.21.15.114:0/3223263759 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5948100bb0 0x7f5948100fd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5944009910 tx=0x7f5944009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:02.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.346+0000 7f59501b4700 1 -- 172.21.15.114:0/3223263759 shutdown_connections 2024-02-19T19:08:02.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.346+0000 7f59501b4700 1 --2- 172.21.15.114:0/3223263759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5948101db0 0x7f5948102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:02.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.346+0000 7f59501b4700 1 --2- 172.21.15.114:0/3223263759 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5948100bb0 0x7f5948100fd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:02.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.346+0000 7f59501b4700 1 -- 172.21.15.114:0/3223263759 >> 172.21.15.114:0/3223263759 conn(0x7f59480fc110 msgr2=0x7f59480fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:02.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.346+0000 7f59501b4700 1 -- 172.21.15.114:0/3223263759 shutdown_connections 2024-02-19T19:08:02.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.346+0000 7f59501b4700 1 -- 172.21.15.114:0/3223263759 wait complete. 2024-02-19T19:08:02.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.346+0000 7f59501b4700 1 Processor -- start 2024-02-19T19:08:02.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.346+0000 7f59501b4700 1 -- start start 2024-02-19T19:08:02.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.347+0000 7f59501b4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5948100bb0 0x7f5948196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:02.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.347+0000 7f59501b4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5948101db0 0x7f59481968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:02.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.347+0000 7f59501b4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5948196ee0 con 0x7f5948100bb0 2024-02-19T19:08:02.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.347+0000 7f59501b4700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5948197020 con 0x7f5948101db0 2024-02-19T19:08:02.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.347+0000 7f594df50700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5948100bb0 0x7f5948196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:02.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.347+0000 7f594df50700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5948100bb0 0x7f5948196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37364/0 (socket says 172.21.15.114:37364) 2024-02-19T19:08:02.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.347+0000 7f594df50700 1 -- 172.21.15.114:0/2440809668 learned_addr learned my addr 172.21.15.114:0/2440809668 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:08:02.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.347+0000 7f594df50700 1 -- 172.21.15.114:0/2440809668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5948101db0 msgr2=0x7f59481968c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T19:08:02.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.347+0000 7f594df50700 1 --2- 172.21.15.114:0/2440809668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5948101db0 0x7f59481968c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:02.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.347+0000 7f594df50700 1 -- 172.21.15.114:0/2440809668 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59440095f0 con 0x7f5948100bb0 2024-02-19T19:08:02.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.347+0000 7f594df50700 1 --2- 172.21.15.114:0/2440809668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5948100bb0 0x7f5948196380 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f594400bda0 tx=0x7f594400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:02.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.347+0000 7f593effd700 1 -- 172.21.15.114:0/2440809668 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f594401d070 con 0x7f5948100bb0 2024-02-19T19:08:02.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.347+0000 7f593effd700 1 -- 172.21.15.114:0/2440809668 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f594400f460 con 0x7f5948100bb0 2024-02-19T19:08:02.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.347+0000 7f59501b4700 1 -- 172.21.15.114:0/2440809668 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594819ba70 con 0x7f5948100bb0 2024-02-19T19:08:02.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.347+0000 7f593effd700 1 -- 172.21.15.114:0/2440809668 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5944021640 con 0x7f5948100bb0 2024-02-19T19:08:02.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.347+0000 7f59501b4700 1 -- 172.21.15.114:0/2440809668 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594819bf60 con 0x7f5948100bb0 2024-02-19T19:08:02.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.348+0000 7f59501b4700 1 -- 172.21.15.114:0/2440809668 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5948109710 con 0x7f5948100bb0 2024-02-19T19:08:02.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.349+0000 7f593effd700 1 -- 172.21.15.114:0/2440809668 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f594400f5d0 con 0x7f5948100bb0 2024-02-19T19:08:02.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.349+0000 7f593effd700 1 --2- 172.21.15.114:0/2440809668 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f593406d8a0 0x7f593406fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:02.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.349+0000 7f593effd700 1 -- 172.21.15.114:0/2440809668 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f594408e9f0 con 0x7f5948100bb0 2024-02-19T19:08:02.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.349+0000 7f594d74f700 1 --2- 172.21.15.114:0/2440809668 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f593406d8a0 0x7f593406fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:02.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.350+0000 7f594d74f700 1 --2- 172.21.15.114:0/2440809668 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f593406d8a0 0x7f593406fd60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f593800ba10 tx=0x7f593800b3f0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:02.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.354+0000 7f593effd700 1 -- 172.21.15.114:0/2440809668 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5944090e60 con 0x7f5948100bb0 2024-02-19T19:08:02.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:02 smithi174 ceph-mon[87279]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:02.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:02 smithi174 ceph-mon[87279]: from='client.34374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:02.645 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:02 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/2755604482' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:02.646 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:02 smithi174 podman[107270]: 2024-02-19 19:08:02.431525833 +0000 UTC m=+0.327311611 container init c1e52f2040c2ef090c9f440d3996d67c046f5dc9cb18f80786f6cb5bc312b0a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-02-19T19:08:02.646 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:02 smithi174 podman[107270]: 2024-02-19 19:08:02.436459911 +0000 UTC m=+0.332245680 container start c1e52f2040c2ef090c9f440d3996d67c046f5dc9cb18f80786f6cb5bc312b0a8 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux ) 2024-02-19T19:08:02.646 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:02 smithi174 ceph-osd[107289]: -- 172.21.15.174:0/2870797204 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557ddf1f51e0 con 0x557ddf1c6c00 2024-02-19T19:08:02.646 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:02 smithi174 bash[107270]: c1e52f2040c2ef090c9f440d3996d67c046f5dc9cb18f80786f6cb5bc312b0a8 2024-02-19T19:08:02.646 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:02 smithi174 systemd[1]: Started Ceph osd.5 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T19:08:02.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.632+0000 7f59501b4700 1 -- 172.21.15.114:0/2440809668 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f594819c5d0 con 0x7f5948100bb0 2024-02-19T19:08:02.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.633+0000 7f593effd700 1 -- 172.21.15.114:0/2440809668 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1871 (secure 0 0 0) 0x7f594405bc00 con 0x7f5948100bb0 2024-02-19T19:08:02.654 INFO:teuthology.orchestra.run.smithi114.stdout:e18 2024-02-19T19:08:02.654 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:08:02.654 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:08:02.654 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-02-19T19:08:02.654 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:08:02.654 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:08:02.654 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-02-19T19:08:02.654 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 18 2024-02-19T19:08:02.654 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-19T19:08:02.654 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:08:02.655 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-02-19T19:00:21.203922+0000 2024-02-19T19:08:02.655 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-02-19T19:08:02.655 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-02-19T19:08:02.655 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-02-19T19:08:02.655 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-02-19T19:08:02.655 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-02-19T19:08:02.655 INFO:teuthology.orchestra.run.smithi114.stdout:max_xattr_size 65536 2024-02-19T19:08:02.655 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-02-19T19:08:02.655 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-02-19T19:08:02.655 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 47 2024-02-19T19:08:02.655 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:08:02.655 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 2 2024-02-19T19:08:02.655 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1 2024-02-19T19:08:02.655 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=14530,1=24269} 2024-02-19T19:08:02.655 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-02-19T19:08:02.655 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-02-19T19:08:02.656 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-02-19T19:08:02.656 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-02-19T19:08:02.656 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-02-19T19:08:02.656 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data enabled 2024-02-19T19:08:02.656 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-02-19T19:08:02.656 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-02-19T19:08:02.656 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 1 2024-02-19T19:08:02.656 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:08:02.656 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.vpoeop{0:14540} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/4219542841,v1:172.21.15.114:6827/4219542841] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:08:02.656 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.znhuvd{1:24269} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:08:02.656 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.stnfnf{1:24275} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.174:6826/963404733,v1:172.21.15.174:6827/963404733] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:08:02.656 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:08:02.656 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:08:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.635+0000 7f59501b4700 1 -- 172.21.15.114:0/2440809668 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f593406d8a0 msgr2=0x7f593406fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.635+0000 7f59501b4700 1 --2- 172.21.15.114:0/2440809668 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f593406d8a0 0x7f593406fd60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f593800ba10 tx=0x7f593800b3f0 comp rx=0 tx=0).stop 2024-02-19T19:08:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.635+0000 7f59501b4700 1 -- 172.21.15.114:0/2440809668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5948100bb0 msgr2=0x7f5948196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.635+0000 7f59501b4700 1 --2- 172.21.15.114:0/2440809668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5948100bb0 0x7f5948196380 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f594400bda0 tx=0x7f594400be80 comp rx=0 tx=0).stop 2024-02-19T19:08:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.635+0000 7f59501b4700 1 -- 172.21.15.114:0/2440809668 shutdown_connections 2024-02-19T19:08:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.635+0000 7f59501b4700 1 --2- 172.21.15.114:0/2440809668 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f593406d8a0 0x7f593406fd60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:02.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.635+0000 7f59501b4700 1 --2- 172.21.15.114:0/2440809668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5948100bb0 0x7f5948196380 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:02.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.635+0000 7f59501b4700 1 --2- 172.21.15.114:0/2440809668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5948101db0 0x7f59481968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:02.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.635+0000 7f59501b4700 1 -- 172.21.15.114:0/2440809668 >> 172.21.15.114:0/2440809668 conn(0x7f59480fc110 msgr2=0x7f5948104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:02.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.635+0000 7f59501b4700 1 -- 172.21.15.114:0/2440809668 shutdown_connections 2024-02-19T19:08:02.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.635+0000 7f59501b4700 1 -- 172.21.15.114:0/2440809668 wait complete. 2024-02-19T19:08:02.658 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 18 2024-02-19T19:08:02.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.765+0000 7f45a6260700 1 -- 172.21.15.114:0/2767139156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45a0101db0 msgr2=0x7f45a0102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:02.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.765+0000 7f45a6260700 1 --2- 172.21.15.114:0/2767139156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45a0101db0 0x7f45a0102210 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4594009910 tx=0x7f4594009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:02.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.765+0000 7f45a6260700 1 -- 172.21.15.114:0/2767139156 shutdown_connections 2024-02-19T19:08:02.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.765+0000 7f45a6260700 1 --2- 172.21.15.114:0/2767139156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45a0101db0 0x7f45a0102210 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:02.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.765+0000 7f45a6260700 1 --2- 172.21.15.114:0/2767139156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45a0100bb0 0x7f45a0100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:02.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.765+0000 7f45a6260700 1 -- 172.21.15.114:0/2767139156 >> 172.21.15.114:0/2767139156 conn(0x7f45a00fc110 msgr2=0x7f45a00fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:02.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.765+0000 7f45a6260700 1 -- 172.21.15.114:0/2767139156 shutdown_connections 2024-02-19T19:08:02.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.765+0000 7f45a6260700 1 -- 172.21.15.114:0/2767139156 wait complete. 2024-02-19T19:08:02.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.766+0000 7f45a6260700 1 Processor -- start 2024-02-19T19:08:02.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.766+0000 7f45a6260700 1 -- start start 2024-02-19T19:08:02.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.766+0000 7f45a6260700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45a0100bb0 0x7f45a0074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:02.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.766+0000 7f45a6260700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45a0101db0 0x7f45a0073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:02.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.766+0000 7f45a6260700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a00736c0 con 0x7f45a0100bb0 2024-02-19T19:08:02.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.766+0000 7f45a6260700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a0073800 con 0x7f45a0101db0 2024-02-19T19:08:02.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.767+0000 7f459f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45a0100bb0 0x7f45a0074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:02.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.767+0000 7f459f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45a0100bb0 0x7f45a0074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37382/0 (socket says 172.21.15.114:37382) 2024-02-19T19:08:02.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.767+0000 7f459f7fe700 1 -- 172.21.15.114:0/416166190 learned_addr learned my addr 172.21.15.114:0/416166190 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:08:02.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.767+0000 7f459effd700 1 --2- 172.21.15.114:0/416166190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45a0101db0 0x7f45a0073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:02.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.767+0000 7f459f7fe700 1 -- 172.21.15.114:0/416166190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45a0101db0 msgr2=0x7f45a0073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:02.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.767+0000 7f459f7fe700 1 --2- 172.21.15.114:0/416166190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45a0101db0 0x7f45a0073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:02.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.767+0000 7f459f7fe700 1 -- 172.21.15.114:0/416166190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45940095f0 con 0x7f45a0100bb0 2024-02-19T19:08:02.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.767+0000 7f459f7fe700 1 --2- 172.21.15.114:0/416166190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45a0100bb0 0x7f45a0074b30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f459000d900 tx=0x7f459000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:02.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:02 smithi114 ceph-mon[92983]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:02.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:02 smithi114 ceph-mon[92983]: from='client.34374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:02.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:02 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/2755604482' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:02.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.767+0000 7f459cff9700 1 -- 172.21.15.114:0/416166190 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45900041f0 con 0x7f45a0100bb0 2024-02-19T19:08:02.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.767+0000 7f45a6260700 1 -- 172.21.15.114:0/416166190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a0073ae0 con 0x7f45a0100bb0 2024-02-19T19:08:02.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.767+0000 7f459cff9700 1 -- 172.21.15.114:0/416166190 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4590004350 con 0x7f45a0100bb0 2024-02-19T19:08:02.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.767+0000 7f459cff9700 1 -- 172.21.15.114:0/416166190 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4590003d20 con 0x7f45a0100bb0 2024-02-19T19:08:02.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.767+0000 7f45a6260700 1 -- 172.21.15.114:0/416166190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a019e3c0 con 0x7f45a0100bb0 2024-02-19T19:08:02.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.769+0000 7f45a6260700 1 -- 172.21.15.114:0/416166190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458c005320 con 0x7f45a0100bb0 2024-02-19T19:08:02.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.769+0000 7f459cff9700 1 -- 172.21.15.114:0/416166190 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f4590010460 con 0x7f45a0100bb0 2024-02-19T19:08:02.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.770+0000 7f459cff9700 1 --2- 172.21.15.114:0/416166190 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f458806d8a0 0x7f458806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:02.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.770+0000 7f459cff9700 1 -- 172.21.15.114:0/416166190 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f459008d4e0 con 0x7f45a0100bb0 2024-02-19T19:08:02.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.770+0000 7f459effd700 1 --2- 172.21.15.114:0/416166190 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f458806d8a0 0x7f458806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:02.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.771+0000 7f459effd700 1 --2- 172.21.15.114:0/416166190 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f458806d8a0 0x7f458806fd60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4594006010 tx=0x7f459400b560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:02.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:02.778+0000 7f459cff9700 1 -- 172.21.15.114:0/416166190 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f459005a6f0 con 0x7f45a0100bb0 2024-02-19T19:08:03.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.039+0000 7f45a6260700 1 -- 172.21.15.114:0/416166190 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f458c000bf0 con 0x7f458806d8a0 2024-02-19T19:08:03.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.041+0000 7f459cff9700 1 -- 172.21.15.114:0/416166190 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f458c000bf0 con 0x7f458806d8a0 2024-02-19T19:08:03.062 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:08:03.062 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:08:03.062 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-02-19T19:08:03.062 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:08:03.062 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [ 2024-02-19T19:08:03.062 INFO:teuthology.orchestra.run.smithi114.stdout: "crash", 2024-02-19T19:08:03.062 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr", 2024-02-19T19:08:03.062 INFO:teuthology.orchestra.run.smithi114.stdout: "mon" 2024-02-19T19:08:03.062 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-02-19T19:08:03.062 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "11/23 daemons upgraded", 2024-02-19T19:08:03.062 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "Currently upgrading osd daemons", 2024-02-19T19:08:03.062 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-02-19T19:08:03.063 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:08:03.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.043+0000 7f45a6260700 1 -- 172.21.15.114:0/416166190 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f458806d8a0 msgr2=0x7f458806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.043+0000 7f45a6260700 1 --2- 172.21.15.114:0/416166190 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f458806d8a0 0x7f458806fd60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4594006010 tx=0x7f459400b560 comp rx=0 tx=0).stop 2024-02-19T19:08:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.043+0000 7f45a6260700 1 -- 172.21.15.114:0/416166190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45a0100bb0 msgr2=0x7f45a0074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.043+0000 7f45a6260700 1 --2- 172.21.15.114:0/416166190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45a0100bb0 0x7f45a0074b30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f459000d900 tx=0x7f459000dc10 comp rx=0 tx=0).stop 2024-02-19T19:08:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.044+0000 7f45a6260700 1 -- 172.21.15.114:0/416166190 shutdown_connections 2024-02-19T19:08:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.044+0000 7f45a6260700 1 --2- 172.21.15.114:0/416166190 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f458806d8a0 0x7f458806fd60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.044+0000 7f45a6260700 1 --2- 172.21.15.114:0/416166190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45a0100bb0 0x7f45a0074b30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.044+0000 7f45a6260700 1 --2- 172.21.15.114:0/416166190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45a0101db0 0x7f45a0073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.044+0000 7f45a6260700 1 -- 172.21.15.114:0/416166190 >> 172.21.15.114:0/416166190 conn(0x7f45a00fc110 msgr2=0x7f45a0104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.044+0000 7f45a6260700 1 -- 172.21.15.114:0/416166190 shutdown_connections 2024-02-19T19:08:03.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.044+0000 7f45a6260700 1 -- 172.21.15.114:0/416166190 wait complete. 2024-02-19T19:08:03.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.167+0000 7f37ba96d700 1 -- 172.21.15.114:0/2159133877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b40ffab0 msgr2=0x7f37b40ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:03.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.167+0000 7f37ba96d700 1 --2- 172.21.15.114:0/2159133877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b40ffab0 0x7f37b40ffed0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f37a4009910 tx=0x7f37a4009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:03.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.168+0000 7f37ba96d700 1 -- 172.21.15.114:0/2159133877 shutdown_connections 2024-02-19T19:08:03.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.168+0000 7f37ba96d700 1 --2- 172.21.15.114:0/2159133877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37b4100cb0 0x7f37b4101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:03.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.168+0000 7f37ba96d700 1 --2- 172.21.15.114:0/2159133877 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b40ffab0 0x7f37b40ffed0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:03.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.168+0000 7f37ba96d700 1 -- 172.21.15.114:0/2159133877 >> 172.21.15.114:0/2159133877 conn(0x7f37b40fb030 msgr2=0x7f37b40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:03.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.168+0000 7f37ba96d700 1 -- 172.21.15.114:0/2159133877 shutdown_connections 2024-02-19T19:08:03.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.168+0000 7f37ba96d700 1 -- 172.21.15.114:0/2159133877 wait complete. 2024-02-19T19:08:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.169+0000 7f37ba96d700 1 Processor -- start 2024-02-19T19:08:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.169+0000 7f37ba96d700 1 -- start start 2024-02-19T19:08:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.169+0000 7f37ba96d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b40ffab0 0x7f37b4195240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.169+0000 7f37ba96d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37b4100cb0 0x7f37b4195780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.169+0000 7f37ba96d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b4195da0 con 0x7f37b40ffab0 2024-02-19T19:08:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.169+0000 7f37ba96d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b4195ee0 con 0x7f37b4100cb0 2024-02-19T19:08:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.169+0000 7f37b3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b40ffab0 0x7f37b4195240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.169+0000 7f37b3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b40ffab0 0x7f37b4195240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37398/0 (socket says 172.21.15.114:37398) 2024-02-19T19:08:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.169+0000 7f37b3fff700 1 -- 172.21.15.114:0/264023917 learned_addr learned my addr 172.21.15.114:0/264023917 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:08:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.169+0000 7f37b37fe700 1 --2- 172.21.15.114:0/264023917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37b4100cb0 0x7f37b4195780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:03.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.170+0000 7f37b3fff700 1 -- 172.21.15.114:0/264023917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37b4100cb0 msgr2=0x7f37b4195780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.170+0000 7f37b3fff700 1 --2- 172.21.15.114:0/264023917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37b4100cb0 0x7f37b4195780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.170+0000 7f37b3fff700 1 -- 172.21.15.114:0/264023917 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a40095f0 con 0x7f37b40ffab0 2024-02-19T19:08:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.170+0000 7f37b37fe700 1 --2- 172.21.15.114:0/264023917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37b4100cb0 0x7f37b4195780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T19:08:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.170+0000 7f37b3fff700 1 --2- 172.21.15.114:0/264023917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b40ffab0 0x7f37b4195240 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f37a400bda0 tx=0x7f37a400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.170+0000 7f37b17fa700 1 -- 172.21.15.114:0/264023917 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37a401d070 con 0x7f37b40ffab0 2024-02-19T19:08:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.170+0000 7f37ba96d700 1 -- 172.21.15.114:0/264023917 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b4068410 con 0x7f37b40ffab0 2024-02-19T19:08:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.170+0000 7f37b17fa700 1 -- 172.21.15.114:0/264023917 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f37a400f460 con 0x7f37b40ffab0 2024-02-19T19:08:03.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.170+0000 7f37b17fa700 1 -- 172.21.15.114:0/264023917 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37a4021640 con 0x7f37b40ffab0 2024-02-19T19:08:03.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.171+0000 7f37ba96d700 1 -- 172.21.15.114:0/264023917 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b4068900 con 0x7f37b40ffab0 2024-02-19T19:08:03.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.172+0000 7f379affd700 1 -- 172.21.15.114:0/264023917 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37940052f0 con 0x7f37b40ffab0 2024-02-19T19:08:03.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.172+0000 7f37b17fa700 1 -- 172.21.15.114:0/264023917 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f37a400f5d0 con 0x7f37b40ffab0 2024-02-19T19:08:03.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.173+0000 7f37b17fa700 1 --2- 172.21.15.114:0/264023917 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f379c06d8a0 0x7f379c06fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:03.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.173+0000 7f37b17fa700 1 -- 172.21.15.114:0/264023917 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f37a408e820 con 0x7f37b40ffab0 2024-02-19T19:08:03.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.173+0000 7f37b37fe700 1 --2- 172.21.15.114:0/264023917 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f379c06d8a0 0x7f379c06fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:03.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.173+0000 7f37b37fe700 1 --2- 172.21.15.114:0/264023917 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f379c06d8a0 0x7f379c06fd60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f37a8005fd0 tx=0x7f37a8005ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:03.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.181+0000 7f37b17fa700 1 -- 172.21.15.114:0/264023917 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f37a405ba80 con 0x7f37b40ffab0 2024-02-19T19:08:03.395 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:03 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5[107285]: 2024-02-19T19:08:03.003+0000 7fdab8c147c0 -1 Falling back to public interface 2024-02-19T19:08:03.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.553+0000 7f379affd700 1 -- 172.21.15.114:0/264023917 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3794005c90 con 0x7f37b40ffab0 2024-02-19T19:08:03.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.554+0000 7f37b17fa700 1 -- 172.21.15.114:0/264023917 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1403 (secure 0 0 0) 0x7f37a405b610 con 0x7f37b40ffab0 2024-02-19T19:08:03.575 INFO:teuthology.orchestra.run.smithi114.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Reduced data availability: 3 pgs inactive, 3 pgs peering; Degraded data redundancy: 13/129 objects degraded (10.078%), 6 pgs degraded 2024-02-19T19:08:03.575 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:08:03.575 INFO:teuthology.orchestra.run.smithi114.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:08:03.575 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-19T19:08:03.575 INFO:teuthology.orchestra.run.smithi114.stdout: osd.5 (root=default,host=smithi174) is down 2024-02-19T19:08:03.575 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-02-19T19:08:03.575 INFO:teuthology.orchestra.run.smithi114.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-02-19T19:08:03.575 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs inactive, 3 pgs peering 2024-02-19T19:08:03.576 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.2 is stuck peering for 2m, current state peering, last acting [1,0] 2024-02-19T19:08:03.576 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.16 is stuck peering for 2m, current state peering, last acting [1,2] 2024-02-19T19:08:03.576 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.18 is stuck peering for 8m, current state peering, last acting [4,3] 2024-02-19T19:08:03.576 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 13/129 objects degraded (10.078%), 6 pgs degraded 2024-02-19T19:08:03.576 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.3 is active+undersized+degraded, acting [2,1] 2024-02-19T19:08:03.576 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.8 is active+undersized+degraded, acting [3,0] 2024-02-19T19:08:03.576 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.b is active+undersized+degraded, acting [3,4] 2024-02-19T19:08:03.576 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.14 is active+undersized+degraded, acting [3,4] 2024-02-19T19:08:03.576 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.17 is active+undersized+degraded, acting [2,1] 2024-02-19T19:08:03.576 INFO:teuthology.orchestra.run.smithi114.stdout: pg 2.1d is active+undersized+degraded, acting [3,0] 2024-02-19T19:08:03.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.556+0000 7f379affd700 1 -- 172.21.15.114:0/264023917 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f379c06d8a0 msgr2=0x7f379c06fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:03.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.556+0000 7f379affd700 1 --2- 172.21.15.114:0/264023917 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f379c06d8a0 0x7f379c06fd60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f37a8005fd0 tx=0x7f37a8005ee0 comp rx=0 tx=0).stop 2024-02-19T19:08:03.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.556+0000 7f379affd700 1 -- 172.21.15.114:0/264023917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b40ffab0 msgr2=0x7f37b4195240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:03.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.556+0000 7f379affd700 1 --2- 172.21.15.114:0/264023917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b40ffab0 0x7f37b4195240 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f37a400bda0 tx=0x7f37a400be80 comp rx=0 tx=0).stop 2024-02-19T19:08:03.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.556+0000 7f379affd700 1 -- 172.21.15.114:0/264023917 shutdown_connections 2024-02-19T19:08:03.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.556+0000 7f379affd700 1 --2- 172.21.15.114:0/264023917 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f379c06d8a0 0x7f379c06fd60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:03.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.557+0000 7f379affd700 1 --2- 172.21.15.114:0/264023917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b40ffab0 0x7f37b4195240 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:03.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.557+0000 7f379affd700 1 --2- 172.21.15.114:0/264023917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37b4100cb0 0x7f37b4195780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:03.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.557+0000 7f379affd700 1 -- 172.21.15.114:0/264023917 >> 172.21.15.114:0/264023917 conn(0x7f37b40fb030 msgr2=0x7f37b4103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:03.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.557+0000 7f379affd700 1 -- 172.21.15.114:0/264023917 shutdown_connections 2024-02-19T19:08:03.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:03.557+0000 7f379affd700 1 -- 172.21.15.114:0/264023917 wait complete. 2024-02-19T19:08:03.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:03 smithi174 ceph-mon[87279]: pgmap v164: 65 pgs: 6 active+undersized+degraded, 12 peering, 3 stale+active+clean, 8 active+undersized, 36 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 13/129 objects degraded (10.078%) 2024-02-19T19:08:03.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:03 smithi174 ceph-mon[87279]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:03.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:03 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:03.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:03 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:03.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:03 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:03.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:03 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/2440809668' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:08:03.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:03 smithi174 ceph-mon[87279]: from='client.? 172.21.15.114:0/264023917' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:08:04.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:03 smithi114 ceph-mon[92983]: pgmap v164: 65 pgs: 6 active+undersized+degraded, 12 peering, 3 stale+active+clean, 8 active+undersized, 36 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 13/129 objects degraded (10.078%) 2024-02-19T19:08:04.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:03 smithi114 ceph-mon[92983]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:04.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:03 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:04.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:03 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:04.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:03 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:04.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:03 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/2440809668' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T19:08:04.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:03 smithi114 ceph-mon[92983]: from='client.? 172.21.15.114:0/264023917' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-19T19:08:04.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:04 smithi174 ceph-mon[87279]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:05.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:04 smithi114 ceph-mon[92983]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:05.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:05 smithi174 ceph-mon[87279]: pgmap v165: 65 pgs: 11 active+undersized+degraded, 19 active+undersized, 35 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 21/129 objects degraded (16.279%) 2024-02-19T19:08:05.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:05 smithi174 ceph-mon[87279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-02-19T19:08:06.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:05 smithi114 ceph-mon[92983]: pgmap v165: 65 pgs: 11 active+undersized+degraded, 19 active+undersized, 35 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 21/129 objects degraded (16.279%) 2024-02-19T19:08:06.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:05 smithi114 ceph-mon[92983]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-02-19T19:08:06.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:06 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:06.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:06 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:06.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:06 smithi174 ceph-mon[87279]: Health check update: Degraded data redundancy: 21/129 objects degraded (16.279%), 11 pgs degraded (PG_DEGRADED) 2024-02-19T19:08:06.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:06 smithi174 ceph-mon[87279]: from='osd.5 [v2:172.21.15.174:6816/39094006,v1:172.21.15.174:6817/39094006]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T19:08:06.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:06 smithi174 ceph-mon[87279]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T19:08:06.896 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:06 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5[107285]: 2024-02-19T19:08:06.511+0000 7fdab8c147c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-02-19T19:08:06.896 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:06 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5[107285]: 2024-02-19T19:08:06.546+0000 7fdab8c147c0 -1 osd.5 78 log_to_monitors true 2024-02-19T19:08:07.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:06 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:07.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:06 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:07.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:06 smithi114 ceph-mon[92983]: Health check update: Degraded data redundancy: 21/129 objects degraded (16.279%), 11 pgs degraded (PG_DEGRADED) 2024-02-19T19:08:07.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:06 smithi114 ceph-mon[92983]: from='osd.5 [v2:172.21.15.174:6816/39094006,v1:172.21.15.174:6817/39094006]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T19:08:07.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:06 smithi114 ceph-mon[92983]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T19:08:08.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:07 smithi114 ceph-mon[92983]: pgmap v166: 65 pgs: 11 active+undersized+degraded, 19 active+undersized, 35 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 21/129 objects degraded (16.279%) 2024-02-19T19:08:08.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:07 smithi114 ceph-mon[92983]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-19T19:08:08.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:07 smithi114 ceph-mon[92983]: osdmap e81: 6 total, 5 up, 6 in 2024-02-19T19:08:08.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:07 smithi114 ceph-mon[92983]: from='osd.5 [v2:172.21.15.174:6816/39094006,v1:172.21.15.174:6817/39094006]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T19:08:08.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:07 smithi114 ceph-mon[92983]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T19:08:08.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:07 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:08.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:07 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:08.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:07 smithi114 ceph-mon[92983]: from='osd.5 ' entity='osd.5' 2024-02-19T19:08:08.146 INFO:journalctl@ceph.osd.5.smithi174.stdout:Feb 19 19:08:07 smithi174 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-5[107285]: 2024-02-19T19:08:07.709+0000 7fdaa6064700 -1 osd.5 78 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T19:08:08.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:07 smithi174 ceph-mon[87279]: pgmap v166: 65 pgs: 11 active+undersized+degraded, 19 active+undersized, 35 active+clean; 594 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 21/129 objects degraded (16.279%) 2024-02-19T19:08:08.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:07 smithi174 ceph-mon[87279]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-19T19:08:08.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:07 smithi174 ceph-mon[87279]: osdmap e81: 6 total, 5 up, 6 in 2024-02-19T19:08:08.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:07 smithi174 ceph-mon[87279]: from='osd.5 [v2:172.21.15.174:6816/39094006,v1:172.21.15.174:6817/39094006]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T19:08:08.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:07 smithi174 ceph-mon[87279]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-02-19T19:08:08.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:07 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:08.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:07 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:08.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:07 smithi174 ceph-mon[87279]: from='osd.5 ' entity='osd.5' 2024-02-19T19:08:09.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:08 smithi114 ceph-mon[92983]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:08:09.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:08 smithi114 ceph-mon[92983]: osd.5 [v2:172.21.15.174:6816/39094006,v1:172.21.15.174:6817/39094006] boot 2024-02-19T19:08:09.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:08 smithi114 ceph-mon[92983]: osdmap e82: 6 total, 6 up, 6 in 2024-02-19T19:08:09.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:08 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:08:09.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:08 smithi174 ceph-mon[87279]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-19T19:08:09.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:08 smithi174 ceph-mon[87279]: osd.5 [v2:172.21.15.174:6816/39094006,v1:172.21.15.174:6817/39094006] boot 2024-02-19T19:08:09.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:08 smithi174 ceph-mon[87279]: osdmap e82: 6 total, 6 up, 6 in 2024-02-19T19:08:09.146 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:08 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:08:09.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:09 smithi174 ceph-mon[87279]: pgmap v168: 65 pgs: 11 active+undersized+degraded, 19 active+undersized, 35 active+clean; 594 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 21/129 objects degraded (16.279%) 2024-02-19T19:08:09.895 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:09 smithi174 ceph-mon[87279]: osdmap e83: 6 total, 6 up, 6 in 2024-02-19T19:08:10.039 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:09 smithi114 ceph-mon[92983]: pgmap v168: 65 pgs: 11 active+undersized+degraded, 19 active+undersized, 35 active+clean; 594 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 21/129 objects degraded (16.279%) 2024-02-19T19:08:10.040 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:09 smithi114 ceph-mon[92983]: osdmap e83: 6 total, 6 up, 6 in 2024-02-19T19:08:10.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:10 smithi174 ceph-mon[87279]: pgmap v171: 65 pgs: 2 peering, 11 active+undersized+degraded, 17 active+undersized, 35 active+clean; 594 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 21/129 objects degraded (16.279%) 2024-02-19T19:08:10.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:10 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:10.896 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:10 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:08:11.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:10 smithi114 ceph-mon[92983]: pgmap v171: 65 pgs: 2 peering, 11 active+undersized+degraded, 17 active+undersized, 35 active+clean; 594 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 21/129 objects degraded (16.279%) 2024-02-19T19:08:11.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:10 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:11.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:10 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:08:12.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:12.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:12.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:12.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:08:12.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:12.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:12.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:12.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:12.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:12.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:12.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: Upgrade: Setting container_image for all osd 2024-02-19T19:08:12.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:12.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-19T19:08:12.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-19T19:08:12.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-19T19:08:12.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-19T19:08:12.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-19T19:08:12.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-19T19:08:12.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-19T19:08:12.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-19T19:08:12.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-19T19:08:12.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-19T19:08:12.398 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-19T19:08:12.398 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-19T19:08:12.398 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: Upgrade: Setting require_osd_release to 19 squid 2024-02-19T19:08:12.398 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:12 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-19T19:08:12.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:12.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:12.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:12.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:08:12.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:12.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:12.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:12.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:12.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:12.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:12.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: Upgrade: Setting container_image for all osd 2024-02-19T19:08:12.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:12.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-19T19:08:12.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-19T19:08:12.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-19T19:08:12.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-19T19:08:12.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-19T19:08:12.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-19T19:08:12.542 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-19T19:08:12.542 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-19T19:08:12.542 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-19T19:08:12.542 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-19T19:08:12.542 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-19T19:08:12.542 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-19T19:08:12.542 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: Upgrade: Setting require_osd_release to 19 squid 2024-02-19T19:08:12.542 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:12 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-19T19:08:13.327 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:13 smithi114 ceph-mon[92983]: pgmap v172: 65 pgs: 2 peering, 7 active+undersized+degraded, 14 active+undersized, 42 active+clean; 594 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 1 op/s; 12/129 objects degraded (9.302%) 2024-02-19T19:08:13.327 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:13 smithi114 ceph-mon[92983]: Health check update: Degraded data redundancy: 12/129 objects degraded (9.302%), 7 pgs degraded (PG_DEGRADED) 2024-02-19T19:08:13.328 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:13 smithi114 ceph-mon[92983]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-19T19:08:13.328 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:13 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-19T19:08:13.328 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:13 smithi114 ceph-mon[92983]: osdmap e84: 6 total, 6 up, 6 in 2024-02-19T19:08:13.328 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:13 smithi114 ceph-mon[92983]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-02-19T19:08:13.328 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:13 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:13.328 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:13 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-02-19T19:08:13.335 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:13 smithi174 ceph-mon[87279]: pgmap v172: 65 pgs: 2 peering, 7 active+undersized+degraded, 14 active+undersized, 42 active+clean; 594 KiB data, 180 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 1 op/s; 12/129 objects degraded (9.302%) 2024-02-19T19:08:13.336 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:13 smithi174 ceph-mon[87279]: Health check update: Degraded data redundancy: 12/129 objects degraded (9.302%), 7 pgs degraded (PG_DEGRADED) 2024-02-19T19:08:13.336 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:13 smithi174 ceph-mon[87279]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-19T19:08:13.336 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:13 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-19T19:08:13.336 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:13 smithi174 ceph-mon[87279]: osdmap e84: 6 total, 6 up, 6 in 2024-02-19T19:08:13.336 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:13 smithi174 ceph-mon[87279]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-02-19T19:08:13.336 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:13 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:13.336 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:13 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-02-19T19:08:14.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:14 smithi174 ceph-mon[87279]: osdmap e85: 6 total, 6 up, 6 in 2024-02-19T19:08:14.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:14 smithi174 ceph-mon[87279]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-19T19:08:14.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-02-19T19:08:14.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:14 smithi174 ceph-mon[87279]: fsmap cephfs:2 {0=cephfs.smithi114.tcwpod=up:active,1=cephfs.smithi174.znhuvd=up:active} 2024-02-19T19:08:14.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:14.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:14.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:08:14.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:14.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:14.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:14.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:14.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:14.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:14.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:14 smithi174 ceph-mon[87279]: Upgrade: Scaling down filesystem cephfs 2024-02-19T19:08:14.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:14.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:14 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-02-19T19:08:14.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:14 smithi114 ceph-mon[92983]: osdmap e85: 6 total, 6 up, 6 in 2024-02-19T19:08:14.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:14 smithi114 ceph-mon[92983]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-19T19:08:14.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-02-19T19:08:14.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:14 smithi114 ceph-mon[92983]: fsmap cephfs:2 {0=cephfs.smithi114.tcwpod=up:active,1=cephfs.smithi174.znhuvd=up:active} 2024-02-19T19:08:14.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:14.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:14.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:08:14.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:14.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:14.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:14.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:14.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:14.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:14.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:14 smithi114 ceph-mon[92983]: Upgrade: Scaling down filesystem cephfs 2024-02-19T19:08:14.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:14.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:14 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-02-19T19:08:15.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: pgmap v175: 65 pgs: 2 peering, 63 active+clean; 594 KiB data, 188 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-02-19T19:08:15.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-19T19:08:15.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-02-19T19:08:15.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: mds.? [v2:172.21.15.114:6826/2633507350,v1:172.21.15.114:6827/2633507350] up:boot 2024-02-19T19:08:15.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: mds.? [v2:172.21.15.174:6826/3615753934,v1:172.21.15.174:6827/3615753934] up:boot 2024-02-19T19:08:15.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: stopping daemon mds.cephfs.smithi174.znhuvd 2024-02-19T19:08:15.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: fsmap cephfs:2 {0=cephfs.smithi114.tcwpod=up:active,1=cephfs.smithi174.znhuvd=up:active} 2 up:standby 2024-02-19T19:08:15.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.vpoeop"}]: dispatch 2024-02-19T19:08:15.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.stnfnf"}]: dispatch 2024-02-19T19:08:15.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:15.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: fsmap cephfs:2 {0=cephfs.smithi114.tcwpod=up:active,1=cephfs.smithi174.znhuvd=up:stopping} 2 up:standby 2024-02-19T19:08:15.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:15.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:08:15.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:15.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/129 objects degraded (9.302%), 7 pgs degraded) 2024-02-19T19:08:15.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:15.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:15.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:15.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:15.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:15.397 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:15 smithi174 ceph-mon[87279]: Upgrade: Waiting for fs cephfs to scale down to reach 1 MDS 2024-02-19T19:08:15.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: pgmap v175: 65 pgs: 2 peering, 63 active+clean; 594 KiB data, 188 MiB used, 536 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-02-19T19:08:15.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-19T19:08:15.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-02-19T19:08:15.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: mds.? [v2:172.21.15.114:6826/2633507350,v1:172.21.15.114:6827/2633507350] up:boot 2024-02-19T19:08:15.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: mds.? [v2:172.21.15.174:6826/3615753934,v1:172.21.15.174:6827/3615753934] up:boot 2024-02-19T19:08:15.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: stopping daemon mds.cephfs.smithi174.znhuvd 2024-02-19T19:08:15.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: fsmap cephfs:2 {0=cephfs.smithi114.tcwpod=up:active,1=cephfs.smithi174.znhuvd=up:active} 2 up:standby 2024-02-19T19:08:15.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi114.vpoeop"}]: dispatch 2024-02-19T19:08:15.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi174.stnfnf"}]: dispatch 2024-02-19T19:08:15.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:15.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: fsmap cephfs:2 {0=cephfs.smithi114.tcwpod=up:active,1=cephfs.smithi174.znhuvd=up:stopping} 2 up:standby 2024-02-19T19:08:15.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:15.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:08:15.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:15.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/129 objects degraded (9.302%), 7 pgs degraded) 2024-02-19T19:08:15.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:15.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:15.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:15.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:15.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:15.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:15 smithi114 ceph-mon[92983]: Upgrade: Waiting for fs cephfs to scale down to reach 1 MDS 2024-02-19T19:08:17.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:17 smithi174 ceph-mon[87279]: pgmap v176: 65 pgs: 65 active+clean; 594 KiB data, 188 MiB used, 536 GiB / 536 GiB avail; 505 B/s rd, 1 op/s 2024-02-19T19:08:17.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:17 smithi114 ceph-mon[92983]: pgmap v176: 65 pgs: 65 active+clean; 594 KiB data, 188 MiB used, 536 GiB / 536 GiB avail; 505 B/s rd, 1 op/s 2024-02-19T19:08:17.668 INFO:tasks.workunit.client.1.smithi174.stderr:Note: switching to '1ce7ea8877c709fa862eac608550374bd96e166e'. 2024-02-19T19:08:17.669 INFO:tasks.workunit.client.1.smithi174.stderr: 2024-02-19T19:08:17.669 INFO:tasks.workunit.client.1.smithi174.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-19T19:08:17.669 INFO:tasks.workunit.client.1.smithi174.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-19T19:08:17.669 INFO:tasks.workunit.client.1.smithi174.stderr:state without impacting any branches by switching back to a branch. 2024-02-19T19:08:17.669 INFO:tasks.workunit.client.1.smithi174.stderr: 2024-02-19T19:08:17.669 INFO:tasks.workunit.client.1.smithi174.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-19T19:08:17.669 INFO:tasks.workunit.client.1.smithi174.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-19T19:08:17.669 INFO:tasks.workunit.client.1.smithi174.stderr: 2024-02-19T19:08:17.669 INFO:tasks.workunit.client.1.smithi174.stderr: git switch -c 2024-02-19T19:08:17.669 INFO:tasks.workunit.client.1.smithi174.stderr: 2024-02-19T19:08:17.669 INFO:tasks.workunit.client.1.smithi174.stderr:Or undo this operation with: 2024-02-19T19:08:17.669 INFO:tasks.workunit.client.1.smithi174.stderr: 2024-02-19T19:08:17.669 INFO:tasks.workunit.client.1.smithi174.stderr: git switch - 2024-02-19T19:08:17.669 INFO:tasks.workunit.client.1.smithi174.stderr: 2024-02-19T19:08:17.670 INFO:tasks.workunit.client.1.smithi174.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-19T19:08:17.670 INFO:tasks.workunit.client.1.smithi174.stderr: 2024-02-19T19:08:17.670 INFO:tasks.workunit.client.1.smithi174.stderr:HEAD is now at 1ce7ea8877c fix 2024-02-19T19:08:17.674 DEBUG:teuthology.orchestra.run.smithi174:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-02-19T19:08:17.699 INFO:tasks.workunit.client.1.smithi174.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-19T19:08:17.701 INFO:tasks.workunit.client.1.smithi174.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-19T19:08:17.701 INFO:tasks.workunit.client.1.smithi174.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-19T19:08:18.533 INFO:tasks.workunit.client.1.smithi174.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-19T19:08:18.602 INFO:tasks.workunit.client.1.smithi174.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-19T19:08:18.642 INFO:tasks.workunit.client.1.smithi174.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-19T19:08:18.644 INFO:tasks.workunit.client.1.smithi174.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-19T19:08:18.644 INFO:tasks.workunit.client.1.smithi174.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-19T19:08:18.683 INFO:tasks.workunit.client.1.smithi174.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-19T19:08:18.686 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-02-19T19:08:18.686 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-02-19T19:08:18.741 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-02-19T19:08:18.743 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-19T19:08:18.743 DEBUG:teuthology.orchestra.run.smithi174:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ce7ea8877c709fa862eac608550374bd96e166e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-02-19T19:08:18.802 INFO:tasks.workunit.client.1.smithi174.stderr:+ mkdir -p fsstress 2024-02-19T19:08:18.804 INFO:tasks.workunit.client.1.smithi174.stderr:+ pushd fsstress 2024-02-19T19:08:18.805 INFO:tasks.workunit.client.1.smithi174.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-19T19:08:18.805 INFO:tasks.workunit.client.1.smithi174.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-19T19:08:19.145 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:19 smithi174 ceph-mon[87279]: pgmap v177: 65 pgs: 65 active+clean; 594 KiB data, 188 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 1 op/s 2024-02-19T19:08:19.423 INFO:tasks.workunit.client.1.smithi174.stderr:+ tar xzf ltp-full.tgz 2024-02-19T19:08:19.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:19 smithi114 ceph-mon[92983]: pgmap v177: 65 pgs: 65 active+clean; 594 KiB data, 188 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 1 op/s 2024-02-19T19:08:21.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:21 smithi174 ceph-mon[87279]: pgmap v178: 65 pgs: 65 active+clean; 594 KiB data, 188 MiB used, 536 GiB / 536 GiB avail; 127 B/s rd, 127 B/s wr, 0 op/s 2024-02-19T19:08:21.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:21 smithi114 ceph-mon[92983]: pgmap v178: 65 pgs: 65 active+clean; 594 KiB data, 188 MiB used, 536 GiB / 536 GiB avail; 127 B/s rd, 127 B/s wr, 0 op/s 2024-02-19T19:08:23.065 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:23 smithi114 ceph-mon[92983]: pgmap v179: 65 pgs: 65 active+clean; 1.9 MiB data, 190 MiB used, 536 GiB / 536 GiB avail; 429 B/s rd, 141 KiB/s wr, 1 op/s 2024-02-19T19:08:23.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:23 smithi174 ceph-mon[87279]: pgmap v179: 65 pgs: 65 active+clean; 1.9 MiB data, 190 MiB used, 536 GiB / 536 GiB avail; 429 B/s rd, 141 KiB/s wr, 1 op/s 2024-02-19T19:08:25.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:25 smithi174 ceph-mon[87279]: pgmap v180: 65 pgs: 65 active+clean; 4.8 MiB data, 201 MiB used, 536 GiB / 536 GiB avail; 380 B/s rd, 423 KiB/s wr, 2 op/s 2024-02-19T19:08:25.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:25.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:25.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:08:25.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:25.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:25.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:25.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:25.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:25.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:25.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:25 smithi174 ceph-mon[87279]: Upgrade: Waiting for fs cephfs to scale down to reach 1 MDS 2024-02-19T19:08:25.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:25.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:08:25.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:25 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:25.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:25 smithi114 ceph-mon[92983]: pgmap v180: 65 pgs: 65 active+clean; 4.8 MiB data, 201 MiB used, 536 GiB / 536 GiB avail; 380 B/s rd, 423 KiB/s wr, 2 op/s 2024-02-19T19:08:25.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:25.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:08:25.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:08:25.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:25.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:25.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:25.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:25.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:25.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:08:25.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:25 smithi114 ceph-mon[92983]: Upgrade: Waiting for fs cephfs to scale down to reach 1 MDS 2024-02-19T19:08:25.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:25.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:08:25.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:25 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' 2024-02-19T19:08:27.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:27 smithi174 ceph-mon[87279]: pgmap v181: 65 pgs: 65 active+clean; 12 MiB data, 223 MiB used, 536 GiB / 536 GiB avail; 341 B/s rd, 997 KiB/s wr, 77 op/s 2024-02-19T19:08:27.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:27 smithi114 ceph-mon[92983]: pgmap v181: 65 pgs: 65 active+clean; 12 MiB data, 223 MiB used, 536 GiB / 536 GiB avail; 341 B/s rd, 997 KiB/s wr, 77 op/s 2024-02-19T19:08:28.031 INFO:tasks.workunit.client.0.smithi114.stderr:Note: switching to '1ce7ea8877c709fa862eac608550374bd96e166e'. 2024-02-19T19:08:28.031 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-02-19T19:08:28.031 INFO:tasks.workunit.client.0.smithi114.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-19T19:08:28.032 INFO:tasks.workunit.client.0.smithi114.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-19T19:08:28.032 INFO:tasks.workunit.client.0.smithi114.stderr:state without impacting any branches by switching back to a branch. 2024-02-19T19:08:28.032 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-02-19T19:08:28.032 INFO:tasks.workunit.client.0.smithi114.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-19T19:08:28.032 INFO:tasks.workunit.client.0.smithi114.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-19T19:08:28.032 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-02-19T19:08:28.032 INFO:tasks.workunit.client.0.smithi114.stderr: git switch -c 2024-02-19T19:08:28.032 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-02-19T19:08:28.032 INFO:tasks.workunit.client.0.smithi114.stderr:Or undo this operation with: 2024-02-19T19:08:28.032 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-02-19T19:08:28.032 INFO:tasks.workunit.client.0.smithi114.stderr: git switch - 2024-02-19T19:08:28.032 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-02-19T19:08:28.032 INFO:tasks.workunit.client.0.smithi114.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-19T19:08:28.032 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-02-19T19:08:28.032 INFO:tasks.workunit.client.0.smithi114.stderr:HEAD is now at 1ce7ea8877c fix 2024-02-19T19:08:28.036 DEBUG:teuthology.orchestra.run.smithi114:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-19T19:08:28.100 INFO:tasks.workunit.client.0.smithi114.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-19T19:08:28.102 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-19T19:08:28.103 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-19T19:08:29.396 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:29 smithi174 ceph-mon[87279]: pgmap v182: 65 pgs: 65 active+clean; 26 MiB data, 287 MiB used, 536 GiB / 536 GiB avail; 85 B/s rd, 2.2 MiB/s wr, 233 op/s 2024-02-19T19:08:29.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:29 smithi114 ceph-mon[92983]: pgmap v182: 65 pgs: 65 active+clean; 26 MiB data, 287 MiB used, 536 GiB / 536 GiB avail; 85 B/s rd, 2.2 MiB/s wr, 233 op/s 2024-02-19T19:08:31.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:31 smithi174 ceph-mon[87279]: pgmap v183: 65 pgs: 65 active+clean; 47 MiB data, 370 MiB used, 536 GiB / 536 GiB avail; 4.0 MiB/s wr, 335 op/s 2024-02-19T19:08:31.539 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:31 smithi114 ceph-mon[92983]: pgmap v183: 65 pgs: 65 active+clean; 47 MiB data, 370 MiB used, 536 GiB / 536 GiB avail; 4.0 MiB/s wr, 335 op/s 2024-02-19T19:08:32.483 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-19T19:08:32.765 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-19T19:08:32.836 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-19T19:08:32.837 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-19T19:08:32.838 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-19T19:08:32.981 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-19T19:08:32.984 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T19:08:32.984 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-19T19:08:33.040 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-02-19T19:08:33.041 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-19T19:08:33.042 DEBUG:teuthology.orchestra.run.smithi114:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ce7ea8877c709fa862eac608550374bd96e166e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-02-19T19:08:33.101 INFO:tasks.workunit.client.0.smithi114.stderr:+ mkdir -p fsstress 2024-02-19T19:08:33.102 INFO:tasks.workunit.client.0.smithi114.stderr:+ pushd fsstress 2024-02-19T19:08:33.103 INFO:tasks.workunit.client.0.smithi114.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-02-19T19:08:33.103 INFO:tasks.workunit.client.0.smithi114.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-19T19:08:33.289 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:33 smithi114 ceph-mon[92983]: pgmap v184: 65 pgs: 65 active+clean; 55 MiB data, 394 MiB used, 536 GiB / 536 GiB avail; 4.7 MiB/s wr, 344 op/s 2024-02-19T19:08:33.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:33 smithi174 ceph-mon[87279]: pgmap v184: 65 pgs: 65 active+clean; 55 MiB data, 394 MiB used, 536 GiB / 536 GiB avail; 4.7 MiB/s wr, 344 op/s 2024-02-19T19:08:33.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.683+0000 7fc7bc196700 1 -- 172.21.15.114:0/1165012053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7b40ffab0 msgr2=0x7fc7b40ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:33.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.683+0000 7fc7bc196700 1 --2- 172.21.15.114:0/1165012053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7b40ffab0 0x7fc7b40ffed0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0009910 tx=0x7fc7b0009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:33.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.683+0000 7fc7bc196700 1 -- 172.21.15.114:0/1165012053 shutdown_connections 2024-02-19T19:08:33.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.683+0000 7fc7bc196700 1 --2- 172.21.15.114:0/1165012053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7b4100cb0 0x7fc7b4101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:33.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.683+0000 7fc7bc196700 1 --2- 172.21.15.114:0/1165012053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7b40ffab0 0x7fc7b40ffed0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:33.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.683+0000 7fc7bc196700 1 -- 172.21.15.114:0/1165012053 >> 172.21.15.114:0/1165012053 conn(0x7fc7b40fb030 msgr2=0x7fc7b40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:33.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.683+0000 7fc7bc196700 1 -- 172.21.15.114:0/1165012053 shutdown_connections 2024-02-19T19:08:33.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.683+0000 7fc7bc196700 1 -- 172.21.15.114:0/1165012053 wait complete. 2024-02-19T19:08:33.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.684+0000 7fc7bc196700 1 Processor -- start 2024-02-19T19:08:33.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.684+0000 7fc7bc196700 1 -- start start 2024-02-19T19:08:33.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.684+0000 7fc7bc196700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7b40ffab0 0x7fc7b41963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:33.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.684+0000 7fc7bc196700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7b4100cb0 0x7fc7b4196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:33.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.684+0000 7fc7bc196700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4196f50 con 0x7fc7b4100cb0 2024-02-19T19:08:33.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.684+0000 7fc7bc196700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4197090 con 0x7fc7b40ffab0 2024-02-19T19:08:33.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.684+0000 7fc7b9731700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7b4100cb0 0x7fc7b4196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:33.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.684+0000 7fc7b9731700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7b4100cb0 0x7fc7b4196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53348/0 (socket says 172.21.15.114:53348) 2024-02-19T19:08:33.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.684+0000 7fc7b9731700 1 -- 172.21.15.114:0/2790266328 learned_addr learned my addr 172.21.15.114:0/2790266328 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:08:33.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.684+0000 7fc7b9f32700 1 --2- 172.21.15.114:0/2790266328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7b40ffab0 0x7fc7b41963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:33.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.685+0000 7fc7b9731700 1 -- 172.21.15.114:0/2790266328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7b40ffab0 msgr2=0x7fc7b41963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:33.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.685+0000 7fc7b9731700 1 --2- 172.21.15.114:0/2790266328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7b40ffab0 0x7fc7b41963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:33.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.685+0000 7fc7b9731700 1 -- 172.21.15.114:0/2790266328 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b00095f0 con 0x7fc7b4100cb0 2024-02-19T19:08:33.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.685+0000 7fc7b9731700 1 --2- 172.21.15.114:0/2790266328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7b4100cb0 0x7fc7b4196930 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400b4b0 tx=0x7fc7a400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:33.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.685+0000 7fc7aaffd700 1 -- 172.21.15.114:0/2790266328 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7a4011800 con 0x7fc7b4100cb0 2024-02-19T19:08:33.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.685+0000 7fc7bc196700 1 -- 172.21.15.114:0/2790266328 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b419bb40 con 0x7fc7b4100cb0 2024-02-19T19:08:33.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.685+0000 7fc7aaffd700 1 -- 172.21.15.114:0/2790266328 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc7a4011e10 con 0x7fc7b4100cb0 2024-02-19T19:08:33.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.685+0000 7fc7aaffd700 1 -- 172.21.15.114:0/2790266328 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7a400f2e0 con 0x7fc7b4100cb0 2024-02-19T19:08:33.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.685+0000 7fc7bc196700 1 -- 172.21.15.114:0/2790266328 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b419c110 con 0x7fc7b4100cb0 2024-02-19T19:08:33.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.687+0000 7fc7bc196700 1 -- 172.21.15.114:0/2790266328 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b4108690 con 0x7fc7b4100cb0 2024-02-19T19:08:33.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.687+0000 7fc7aaffd700 1 -- 172.21.15.114:0/2790266328 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7fc7a4010380 con 0x7fc7b4100cb0 2024-02-19T19:08:33.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.687+0000 7fc7aaffd700 1 --2- 172.21.15.114:0/2790266328 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc7a006d8a0 0x7fc7a006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:33.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.688+0000 7fc7aaffd700 1 -- 172.21.15.114:0/2790266328 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6790+0+0 (secure 0 0 0) 0x7fc7a408d1b0 con 0x7fc7b4100cb0 2024-02-19T19:08:33.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.688+0000 7fc7b9f32700 1 --2- 172.21.15.114:0/2790266328 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc7a006d8a0 0x7fc7a006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:33.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.688+0000 7fc7b9f32700 1 --2- 172.21.15.114:0/2790266328 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc7a006d8a0 0x7fc7a006fd60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0006010 tx=0x7fc7b000b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:33.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.696+0000 7fc7aaffd700 1 -- 172.21.15.114:0/2790266328 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc7a405a3c0 con 0x7fc7b4100cb0 2024-02-19T19:08:33.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.968+0000 7fc7bc196700 1 -- 172.21.15.114:0/2790266328 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7b4068b20 con 0x7fc7a006d8a0 2024-02-19T19:08:33.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.969+0000 7fc7aaffd700 1 -- 172.21.15.114:0/2790266328 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc7b4068b20 con 0x7fc7a006d8a0 2024-02-19T19:08:33.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.971+0000 7fc7bc196700 1 -- 172.21.15.114:0/2790266328 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc7a006d8a0 msgr2=0x7fc7a006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:33.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.971+0000 7fc7bc196700 1 --2- 172.21.15.114:0/2790266328 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc7a006d8a0 0x7fc7a006fd60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0006010 tx=0x7fc7b000b540 comp rx=0 tx=0).stop 2024-02-19T19:08:33.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.971+0000 7fc7bc196700 1 -- 172.21.15.114:0/2790266328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7b4100cb0 msgr2=0x7fc7b4196930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:33.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.971+0000 7fc7bc196700 1 --2- 172.21.15.114:0/2790266328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7b4100cb0 0x7fc7b4196930 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400b4b0 tx=0x7fc7a400b7c0 comp rx=0 tx=0).stop 2024-02-19T19:08:33.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.971+0000 7fc7bc196700 1 -- 172.21.15.114:0/2790266328 shutdown_connections 2024-02-19T19:08:33.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.971+0000 7fc7bc196700 1 --2- 172.21.15.114:0/2790266328 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fc7a006d8a0 0x7fc7a006fd60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:33.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.971+0000 7fc7bc196700 1 --2- 172.21.15.114:0/2790266328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7b40ffab0 0x7fc7b41963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:33.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.971+0000 7fc7bc196700 1 --2- 172.21.15.114:0/2790266328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc7b4100cb0 0x7fc7b4196930 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:33.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.971+0000 7fc7bc196700 1 -- 172.21.15.114:0/2790266328 >> 172.21.15.114:0/2790266328 conn(0x7fc7b40fb030 msgr2=0x7fc7b4068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:33.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.972+0000 7fc7bc196700 1 -- 172.21.15.114:0/2790266328 shutdown_connections 2024-02-19T19:08:33.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:33.972+0000 7fc7bc196700 1 -- 172.21.15.114:0/2790266328 wait complete. 2024-02-19T19:08:34.003 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-02-19T19:08:34.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.088+0000 7f16de454700 1 -- 172.21.15.114:0/2162723401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16d8103f70 msgr2=0x7f16d8106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:34.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.088+0000 7f16de454700 1 --2- 172.21.15.114:0/2162723401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16d8103f70 0x7f16d8106360 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f16cc009910 tx=0x7f16cc009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:34.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.088+0000 7f16de454700 1 -- 172.21.15.114:0/2162723401 shutdown_connections 2024-02-19T19:08:34.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.088+0000 7f16de454700 1 --2- 172.21.15.114:0/2162723401 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16d8103f70 0x7f16d8106360 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.088+0000 7f16de454700 1 --2- 172.21.15.114:0/2162723401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16d8101640 0x7f16d8103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.088+0000 7f16de454700 1 -- 172.21.15.114:0/2162723401 >> 172.21.15.114:0/2162723401 conn(0x7f16d80fb030 msgr2=0x7f16d80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:34.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.088+0000 7f16de454700 1 -- 172.21.15.114:0/2162723401 shutdown_connections 2024-02-19T19:08:34.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.088+0000 7f16de454700 1 -- 172.21.15.114:0/2162723401 wait complete. 2024-02-19T19:08:34.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.088+0000 7f16de454700 1 Processor -- start 2024-02-19T19:08:34.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.088+0000 7f16de454700 1 -- start start 2024-02-19T19:08:34.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.089+0000 7f16de454700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16d8101640 0x7f16d8196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:34.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.089+0000 7f16de454700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16d8103f70 0x7f16d8196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:34.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.089+0000 7f16de454700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d8196e00 con 0x7f16d8101640 2024-02-19T19:08:34.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.089+0000 7f16de454700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d8196f40 con 0x7f16d8103f70 2024-02-19T19:08:34.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.089+0000 7f16d7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16d8101640 0x7f16d8196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:34.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.089+0000 7f16d7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16d8101640 0x7f16d8196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53364/0 (socket says 172.21.15.114:53364) 2024-02-19T19:08:34.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.089+0000 7f16d7fff700 1 -- 172.21.15.114:0/3693956207 learned_addr learned my addr 172.21.15.114:0/3693956207 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:08:34.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.089+0000 7f16d77fe700 1 --2- 172.21.15.114:0/3693956207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16d8103f70 0x7f16d8196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:34.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.089+0000 7f16d7fff700 1 -- 172.21.15.114:0/3693956207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16d8103f70 msgr2=0x7f16d8196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:34.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.089+0000 7f16d7fff700 1 --2- 172.21.15.114:0/3693956207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16d8103f70 0x7f16d8196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.089+0000 7f16d7fff700 1 -- 172.21.15.114:0/3693956207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16cc0095f0 con 0x7f16d8101640 2024-02-19T19:08:34.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.089+0000 7f16d77fe700 1 --2- 172.21.15.114:0/3693956207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16d8103f70 0x7f16d8196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:08:34.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.089+0000 7f16d7fff700 1 --2- 172.21.15.114:0/3693956207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16d8101640 0x7f16d8196330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f16c800c930 tx=0x7f16c800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:34.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.090+0000 7f16d57fa700 1 -- 172.21.15.114:0/3693956207 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16c8007ad0 con 0x7f16d8101640 2024-02-19T19:08:34.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.090+0000 7f16d57fa700 1 -- 172.21.15.114:0/3693956207 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f16c8007c30 con 0x7f16d8101640 2024-02-19T19:08:34.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.090+0000 7f16de454700 1 -- 172.21.15.114:0/3693956207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d819ba00 con 0x7f16d8101640 2024-02-19T19:08:34.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.090+0000 7f16de454700 1 -- 172.21.15.114:0/3693956207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d819bf20 con 0x7f16d8101640 2024-02-19T19:08:34.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.090+0000 7f16d57fa700 1 -- 172.21.15.114:0/3693956207 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16c8018630 con 0x7f16d8101640 2024-02-19T19:08:34.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.091+0000 7f16de454700 1 -- 172.21.15.114:0/3693956207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d8190520 con 0x7f16d8101640 2024-02-19T19:08:34.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.091+0000 7f16d57fa700 1 -- 172.21.15.114:0/3693956207 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f16c800f450 con 0x7f16d8101640 2024-02-19T19:08:34.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.092+0000 7f16d57fa700 1 --2- 172.21.15.114:0/3693956207 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f16c006d8a0 0x7f16c006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:34.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.092+0000 7f16d57fa700 1 -- 172.21.15.114:0/3693956207 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f16c808d710 con 0x7f16d8101640 2024-02-19T19:08:34.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.092+0000 7f16d77fe700 1 --2- 172.21.15.114:0/3693956207 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f16c006d8a0 0x7f16c006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:34.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.092+0000 7f16d77fe700 1 --2- 172.21.15.114:0/3693956207 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f16c006d8a0 0x7f16c006fd60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f16cc005b40 tx=0x7f16cc005ab0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:34.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.097+0000 7f16d57fa700 1 -- 172.21.15.114:0/3693956207 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f16c805a9a0 con 0x7f16d8101640 2024-02-19T19:08:34.315 INFO:tasks.workunit.client.0.smithi114.stderr:+ tar xzf ltp-full.tgz 2024-02-19T19:08:34.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.377+0000 7f16de454700 1 -- 172.21.15.114:0/3693956207 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f16d80611d0 con 0x7f16c006d8a0 2024-02-19T19:08:34.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.379+0000 7f16d57fa700 1 -- 172.21.15.114:0/3693956207 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f16d80611d0 con 0x7f16c006d8a0 2024-02-19T19:08:34.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.381+0000 7f16de454700 1 -- 172.21.15.114:0/3693956207 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f16c006d8a0 msgr2=0x7f16c006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:34.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.381+0000 7f16de454700 1 --2- 172.21.15.114:0/3693956207 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f16c006d8a0 0x7f16c006fd60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f16cc005b40 tx=0x7f16cc005ab0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.381+0000 7f16de454700 1 -- 172.21.15.114:0/3693956207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16d8101640 msgr2=0x7f16d8196330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:34.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.381+0000 7f16de454700 1 --2- 172.21.15.114:0/3693956207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16d8101640 0x7f16d8196330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f16c800c930 tx=0x7f16c800ccf0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.381+0000 7f16de454700 1 -- 172.21.15.114:0/3693956207 shutdown_connections 2024-02-19T19:08:34.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.381+0000 7f16de454700 1 --2- 172.21.15.114:0/3693956207 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f16c006d8a0 0x7f16c006fd60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.381+0000 7f16de454700 1 --2- 172.21.15.114:0/3693956207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f16d8101640 0x7f16d8196330 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.381+0000 7f16de454700 1 --2- 172.21.15.114:0/3693956207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16d8103f70 0x7f16d8196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.381+0000 7f16de454700 1 -- 172.21.15.114:0/3693956207 >> 172.21.15.114:0/3693956207 conn(0x7f16d80fb030 msgr2=0x7f16d80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:34.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.382+0000 7f16de454700 1 -- 172.21.15.114:0/3693956207 shutdown_connections 2024-02-19T19:08:34.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.382+0000 7f16de454700 1 -- 172.21.15.114:0/3693956207 wait complete. 2024-02-19T19:08:34.415 INFO:tasks.workunit.client.1.smithi174.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-02-19T19:08:34.415 INFO:tasks.workunit.client.1.smithi174.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-19T19:08:34.415 INFO:tasks.workunit.client.1.smithi174.stderr:+ make 2024-02-19T19:08:34.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.495+0000 7f7d7d529700 1 -- 172.21.15.114:0/4141409381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d780ffab0 msgr2=0x7f7d780ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:34.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.495+0000 7f7d7d529700 1 --2- 172.21.15.114:0/4141409381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d780ffab0 0x7f7d780ffed0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7d68009910 tx=0x7f7d68009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:34.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.495+0000 7f7d7d529700 1 -- 172.21.15.114:0/4141409381 shutdown_connections 2024-02-19T19:08:34.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.495+0000 7f7d7d529700 1 --2- 172.21.15.114:0/4141409381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d78100cb0 0x7f7d78101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.495+0000 7f7d7d529700 1 --2- 172.21.15.114:0/4141409381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d780ffab0 0x7f7d780ffed0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.495+0000 7f7d7d529700 1 -- 172.21.15.114:0/4141409381 >> 172.21.15.114:0/4141409381 conn(0x7f7d780fb030 msgr2=0x7f7d780fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:34.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.495+0000 7f7d7d529700 1 -- 172.21.15.114:0/4141409381 shutdown_connections 2024-02-19T19:08:34.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.495+0000 7f7d7d529700 1 -- 172.21.15.114:0/4141409381 wait complete. 2024-02-19T19:08:34.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.495+0000 7f7d7d529700 1 Processor -- start 2024-02-19T19:08:34.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.496+0000 7f7d7d529700 1 -- start start 2024-02-19T19:08:34.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.496+0000 7f7d7d529700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d780ffab0 0x7f7d78068f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:34.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.496+0000 7f7d7d529700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d78100cb0 0x7f7d7819b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:34.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.496+0000 7f7d7d529700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7819b670 con 0x7f7d78100cb0 2024-02-19T19:08:34.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.496+0000 7f7d7d529700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7819b7b0 con 0x7f7d780ffab0 2024-02-19T19:08:34.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.496+0000 7f7d767fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d78100cb0 0x7f7d7819b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:34.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.496+0000 7f7d767fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d78100cb0 0x7f7d7819b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53374/0 (socket says 172.21.15.114:53374) 2024-02-19T19:08:34.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.496+0000 7f7d767fc700 1 -- 172.21.15.114:0/2969542255 learned_addr learned my addr 172.21.15.114:0/2969542255 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:08:34.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.496+0000 7f7d76ffd700 1 --2- 172.21.15.114:0/2969542255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d780ffab0 0x7f7d78068f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:34.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.496+0000 7f7d767fc700 1 -- 172.21.15.114:0/2969542255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d780ffab0 msgr2=0x7f7d78068f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:34.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.496+0000 7f7d767fc700 1 --2- 172.21.15.114:0/2969542255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d780ffab0 0x7f7d78068f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.496+0000 7f7d767fc700 1 -- 172.21.15.114:0/2969542255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d680095f0 con 0x7f7d78100cb0 2024-02-19T19:08:34.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.496+0000 7f7d767fc700 1 --2- 172.21.15.114:0/2969542255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d78100cb0 0x7f7d7819b0a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00c8a0 tx=0x7f7d6c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:34.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.497+0000 7f7d5ffff700 1 -- 172.21.15.114:0/2969542255 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d6c0078e0 con 0x7f7d78100cb0 2024-02-19T19:08:34.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.497+0000 7f7d5ffff700 1 -- 172.21.15.114:0/2969542255 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7d6c00f450 con 0x7f7d78100cb0 2024-02-19T19:08:34.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.497+0000 7f7d5ffff700 1 -- 172.21.15.114:0/2969542255 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d6c00e5e0 con 0x7f7d78100cb0 2024-02-19T19:08:34.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.497+0000 7f7d7d529700 1 -- 172.21.15.114:0/2969542255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d7819ba90 con 0x7f7d78100cb0 2024-02-19T19:08:34.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.497+0000 7f7d7d529700 1 -- 172.21.15.114:0/2969542255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d7819bfe0 con 0x7f7d78100cb0 2024-02-19T19:08:34.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.498+0000 7f7d7d529700 1 -- 172.21.15.114:0/2969542255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d78108610 con 0x7f7d78100cb0 2024-02-19T19:08:34.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.498+0000 7f7d5ffff700 1 -- 172.21.15.114:0/2969542255 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f7d6c00f5c0 con 0x7f7d78100cb0 2024-02-19T19:08:34.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.499+0000 7f7d5ffff700 1 --2- 172.21.15.114:0/2969542255 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f7d6006d8a0 0x7f7d6006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:34.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.499+0000 7f7d5ffff700 1 -- 172.21.15.114:0/2969542255 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f7d6c08d4d0 con 0x7f7d78100cb0 2024-02-19T19:08:34.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.499+0000 7f7d76ffd700 1 --2- 172.21.15.114:0/2969542255 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f7d6006d8a0 0x7f7d6006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:34.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.499+0000 7f7d76ffd700 1 --2- 172.21.15.114:0/2969542255 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f7d6006d8a0 0x7f7d6006fd60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7d68006010 tx=0x7f7d6800b540 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:34.529 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.508+0000 7f7d5ffff700 1 -- 172.21.15.114:0/2969542255 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7d6c05a760 con 0x7f7d78100cb0 2024-02-19T19:08:34.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.775+0000 7f7d7d529700 1 -- 172.21.15.114:0/2969542255 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d78105600 con 0x7f7d6006d8a0 2024-02-19T19:08:34.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.788+0000 7f7d5ffff700 1 -- 172.21.15.114:0/2969542255 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f7d78105600 con 0x7f7d6006d8a0 2024-02-19T19:08:34.809 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:08:34.809 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.smithi114 smithi114 *:9093,9094 running (12m) 2m ago 15m 26.5M - 0.25.0 c8568f914cd2 30444b676ca4 2024-02-19T19:08:34.809 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi114 smithi114 running (15m) 2m ago 15m 8472k - 18.2.1 94eadd990e0a 4918cd87e312 2024-02-19T19:08:34.809 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-exporter.smithi174 smithi174 running (10m) 29s ago 13m 8488k - 18.2.1 94eadd990e0a 516c34431b93 2024-02-19T19:08:34.809 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi114 smithi114 running (3m) 2m ago 15m 6660k - 19.0.0-1450-gf78a58c0 f081f847070a f10b46e9d636 2024-02-19T19:08:34.810 INFO:teuthology.orchestra.run.smithi114.stdout:crash.smithi174 smithi174 running (3m) 29s ago 13m 6660k - 19.0.0-1450-gf78a58c0 f081f847070a fb3b17c81bd8 2024-02-19T19:08:34.810 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.smithi114 smithi114 *:3000 running (12m) 2m ago 14m 85.5M - 9.4.7 954c08fa6188 478bd533e827 2024-02-19T19:08:34.810 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.tcwpod smithi114 running (8m) 2m ago 8m 27.5M - 18.2.1 94eadd990e0a 420175777cf4 2024-02-19T19:08:34.810 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi114.vpoeop smithi114 running (8m) 2m ago 8m 24.6M - 18.2.1 94eadd990e0a f81a79e53ead 2024-02-19T19:08:34.810 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.stnfnf smithi174 running (8m) 29s ago 8m 26.4M - 18.2.1 94eadd990e0a 3df35954a492 2024-02-19T19:08:34.810 INFO:teuthology.orchestra.run.smithi114.stdout:mds.cephfs.smithi174.znhuvd smithi174 running (8m) 29s ago 8m 25.5M - 18.2.1 94eadd990e0a 2d2267494ae1 2024-02-19T19:08:34.810 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi114.pcvtrk smithi114 *:8443,9283,8765 running (5m) 2m ago 16m 503M - 19.0.0-1450-gf78a58c0 f081f847070a 05b5c06122b1 2024-02-19T19:08:34.810 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.smithi174.lydvnj smithi174 *:8443,9283,8765 running (5m) 29s ago 13m 452M - 19.0.0-1450-gf78a58c0 f081f847070a f962b1af5aed 2024-02-19T19:08:34.810 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi114 smithi114 running (5m) 2m ago 17m 48.1M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e51f650f9be9 2024-02-19T19:08:34.810 INFO:teuthology.orchestra.run.smithi114.stdout:mon.smithi174 smithi174 running (4m) 29s ago 12m 38.0M 2048M 19.0.0-1450-gf78a58c0 f081f847070a e811581f05fb 2024-02-19T19:08:34.810 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi114 smithi114 *:9100 running (15m) 2m ago 15m 14.3M - 1.5.0 0da6a335fe13 b6b258074cce 2024-02-19T19:08:34.810 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.smithi174 smithi174 *:9100 running (13m) 29s ago 13m 15.3M - 1.5.0 0da6a335fe13 d0b4fdf67b22 2024-02-19T19:08:34.810 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (3m) 2m ago 12m 51.2M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 764769439e0e 2024-02-19T19:08:34.811 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (2m) 2m ago 11m 51.8M 1216M 19.0.0-1450-gf78a58c0 f081f847070a c1c32dbdf573 2024-02-19T19:08:34.811 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (2m) 2m ago 11m 11.5M 1216M 19.0.0-1450-gf78a58c0 f081f847070a 11e64d20e378 2024-02-19T19:08:34.811 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi174 running (99s) 29s ago 10m 44.6M 2240M 19.0.0-1450-gf78a58c0 f081f847070a cc3a55bd908f 2024-02-19T19:08:34.811 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi174 running (66s) 29s ago 10m 50.5M 2240M 19.0.0-1450-gf78a58c0 f081f847070a b3609ea94466 2024-02-19T19:08:34.811 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi174 running (32s) 29s ago 9m 15.0M 2240M 19.0.0-1450-gf78a58c0 f081f847070a c1e52f2040c2 2024-02-19T19:08:34.811 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.smithi114 smithi114 *:9095 running (5m) 2m ago 13m 63.5M - 2.43.0 a07b618ecd1d a36922c7c9a1 2024-02-19T19:08:34.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.791+0000 7f7d7d529700 1 -- 172.21.15.114:0/2969542255 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f7d6006d8a0 msgr2=0x7f7d6006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:34.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.791+0000 7f7d7d529700 1 --2- 172.21.15.114:0/2969542255 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f7d6006d8a0 0x7f7d6006fd60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7d68006010 tx=0x7f7d6800b540 comp rx=0 tx=0).stop 2024-02-19T19:08:34.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.791+0000 7f7d7d529700 1 -- 172.21.15.114:0/2969542255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d78100cb0 msgr2=0x7f7d7819b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:34.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.791+0000 7f7d7d529700 1 --2- 172.21.15.114:0/2969542255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d78100cb0 0x7f7d7819b0a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00c8a0 tx=0x7f7d6c00cbb0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.791+0000 7f7d7d529700 1 -- 172.21.15.114:0/2969542255 shutdown_connections 2024-02-19T19:08:34.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.791+0000 7f7d7d529700 1 --2- 172.21.15.114:0/2969542255 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f7d6006d8a0 0x7f7d6006fd60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.791+0000 7f7d7d529700 1 --2- 172.21.15.114:0/2969542255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d780ffab0 0x7f7d78068f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.791+0000 7f7d7d529700 1 --2- 172.21.15.114:0/2969542255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d78100cb0 0x7f7d7819b0a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.791+0000 7f7d7d529700 1 -- 172.21.15.114:0/2969542255 >> 172.21.15.114:0/2969542255 conn(0x7f7d780fb030 msgr2=0x7f7d78103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:34.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.791+0000 7f7d7d529700 1 -- 172.21.15.114:0/2969542255 shutdown_connections 2024-02-19T19:08:34.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.791+0000 7f7d7d529700 1 -- 172.21.15.114:0/2969542255 wait complete. 2024-02-19T19:08:34.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.936+0000 7f9e24492700 1 -- 172.21.15.114:0/2766370078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e1c0ff430 msgr2=0x7f9e1c0ff850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:34.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.936+0000 7f9e24492700 1 --2- 172.21.15.114:0/2766370078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e1c0ff430 0x7f9e1c0ff850 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009910 tx=0x7f9e18009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:34.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.936+0000 7f9e24492700 1 -- 172.21.15.114:0/2766370078 shutdown_connections 2024-02-19T19:08:34.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.936+0000 7f9e24492700 1 --2- 172.21.15.114:0/2766370078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e1c0ffd90 0x7f9e1c100210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.936+0000 7f9e24492700 1 --2- 172.21.15.114:0/2766370078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e1c0ff430 0x7f9e1c0ff850 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.936+0000 7f9e24492700 1 -- 172.21.15.114:0/2766370078 >> 172.21.15.114:0/2766370078 conn(0x7f9e1c0fb030 msgr2=0x7f9e1c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:34.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.936+0000 7f9e24492700 1 -- 172.21.15.114:0/2766370078 shutdown_connections 2024-02-19T19:08:34.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.936+0000 7f9e24492700 1 -- 172.21.15.114:0/2766370078 wait complete. 2024-02-19T19:08:34.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.937+0000 7f9e24492700 1 Processor -- start 2024-02-19T19:08:34.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.937+0000 7f9e24492700 1 -- start start 2024-02-19T19:08:34.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.937+0000 7f9e24492700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e1c0ff430 0x7f9e1c104fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:34.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.937+0000 7f9e24492700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e1c0ffd90 0x7f9e1c1054f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:34.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.937+0000 7f9e24492700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c101aa0 con 0x7f9e1c0ffd90 2024-02-19T19:08:34.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.937+0000 7f9e24492700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c101c10 con 0x7f9e1c0ff430 2024-02-19T19:08:34.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.937+0000 7f9e21a2d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e1c0ffd90 0x7f9e1c1054f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:34.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.937+0000 7f9e2222e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e1c0ff430 0x7f9e1c104fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:34.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.937+0000 7f9e21a2d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e1c0ffd90 0x7f9e1c1054f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53386/0 (socket says 172.21.15.114:53386) 2024-02-19T19:08:34.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.937+0000 7f9e21a2d700 1 -- 172.21.15.114:0/3407366258 learned_addr learned my addr 172.21.15.114:0/3407366258 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:08:34.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.938+0000 7f9e21a2d700 1 -- 172.21.15.114:0/3407366258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e1c0ff430 msgr2=0x7f9e1c104fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:34.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.938+0000 7f9e21a2d700 1 --2- 172.21.15.114:0/3407366258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e1c0ff430 0x7f9e1c104fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:34.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.938+0000 7f9e21a2d700 1 -- 172.21.15.114:0/3407366258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e180095f0 con 0x7f9e1c0ffd90 2024-02-19T19:08:34.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.938+0000 7f9e2222e700 1 --2- 172.21.15.114:0/3407366258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e1c0ff430 0x7f9e1c104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:08:34.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.938+0000 7f9e21a2d700 1 --2- 172.21.15.114:0/3407366258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e1c0ffd90 0x7f9e1c1054f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00e920 tx=0x7f9e0c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:34.960 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.938+0000 7f9e137fe700 1 -- 172.21.15.114:0/3407366258 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e0c00c9d0 con 0x7f9e1c0ffd90 2024-02-19T19:08:34.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.938+0000 7f9e24492700 1 -- 172.21.15.114:0/3407366258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1c101ef0 con 0x7f9e1c0ffd90 2024-02-19T19:08:34.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.938+0000 7f9e137fe700 1 -- 172.21.15.114:0/3407366258 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9e0c010430 con 0x7f9e1c0ffd90 2024-02-19T19:08:34.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.938+0000 7f9e24492700 1 -- 172.21.15.114:0/3407366258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1c102380 con 0x7f9e1c0ffd90 2024-02-19T19:08:34.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.938+0000 7f9e137fe700 1 -- 172.21.15.114:0/3407366258 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e0c018650 con 0x7f9e1c0ffd90 2024-02-19T19:08:34.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.939+0000 7f9e24492700 1 -- 172.21.15.114:0/3407366258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1c10b900 con 0x7f9e1c0ffd90 2024-02-19T19:08:34.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.940+0000 7f9e137fe700 1 -- 172.21.15.114:0/3407366258 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7f9e0c0105a0 con 0x7f9e1c0ffd90 2024-02-19T19:08:34.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.941+0000 7f9e137fe700 1 --2- 172.21.15.114:0/3407366258 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9e0806d8f0 0x7f9e0806fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:34.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.941+0000 7f9e137fe700 1 -- 172.21.15.114:0/3407366258 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6790+0+0 (secure 0 0 0) 0x7f9e0c014070 con 0x7f9e1c0ffd90 2024-02-19T19:08:34.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.941+0000 7f9e2222e700 1 --2- 172.21.15.114:0/3407366258 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9e0806d8f0 0x7f9e0806fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:34.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.941+0000 7f9e2222e700 1 --2- 172.21.15.114:0/3407366258 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9e0806d8f0 0x7f9e0806fdb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009fd0 tx=0x7f9e18005fd0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:34.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:34.949+0000 7f9e137fe700 1 -- 172.21.15.114:0/3407366258 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9e0c052d20 con 0x7f9e1c0ffd90 2024-02-19T19:08:35.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.292+0000 7f9e24492700 1 -- 172.21.15.114:0/3407366258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9e1c10bac0 con 0x7f9e1c0ffd90 2024-02-19T19:08:35.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.292+0000 7f9e137fe700 1 -- 172.21.15.114:0/3407366258 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+701 (secure 0 0 0) 0x7f9e0c05aad0 con 0x7f9e1c0ffd90 2024-02-19T19:08:35.313 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:08:35.314 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-02-19T19:08:35.314 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:08:35.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:08:35.314 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-02-19T19:08:35.314 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:08:35.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:08:35.314 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-02-19T19:08:35.314 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 6 2024-02-19T19:08:35.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:08:35.314 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-02-19T19:08:35.314 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-02-19T19:08:35.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-02-19T19:08:35.314 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-02-19T19:08:35.315 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4, 2024-02-19T19:08:35.315 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 10 2024-02-19T19:08:35.315 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-02-19T19:08:35.315 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:08:35.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.294+0000 7f9e24492700 1 -- 172.21.15.114:0/3407366258 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9e0806d8f0 msgr2=0x7f9e0806fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:35.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.294+0000 7f9e24492700 1 --2- 172.21.15.114:0/3407366258 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9e0806d8f0 0x7f9e0806fdb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009fd0 tx=0x7f9e18005fd0 comp rx=0 tx=0).stop 2024-02-19T19:08:35.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.294+0000 7f9e24492700 1 -- 172.21.15.114:0/3407366258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e1c0ffd90 msgr2=0x7f9e1c1054f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:35.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.294+0000 7f9e24492700 1 --2- 172.21.15.114:0/3407366258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e1c0ffd90 0x7f9e1c1054f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00e920 tx=0x7f9e0c00ec30 comp rx=0 tx=0).stop 2024-02-19T19:08:35.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.295+0000 7f9e24492700 1 -- 172.21.15.114:0/3407366258 shutdown_connections 2024-02-19T19:08:35.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.295+0000 7f9e24492700 1 --2- 172.21.15.114:0/3407366258 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7f9e0806d8f0 0x7f9e0806fdb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:35.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.295+0000 7f9e24492700 1 --2- 172.21.15.114:0/3407366258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e1c0ff430 0x7f9e1c104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:35.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.295+0000 7f9e24492700 1 --2- 172.21.15.114:0/3407366258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e1c0ffd90 0x7f9e1c1054f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:35.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.295+0000 7f9e24492700 1 -- 172.21.15.114:0/3407366258 >> 172.21.15.114:0/3407366258 conn(0x7f9e1c0fb030 msgr2=0x7f9e1c107a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:35.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.295+0000 7f9e24492700 1 -- 172.21.15.114:0/3407366258 shutdown_connections 2024-02-19T19:08:35.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.295+0000 7f9e24492700 1 -- 172.21.15.114:0/3407366258 wait complete. 2024-02-19T19:08:35.380 INFO:tasks.workunit.client.1.smithi174.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-02-19T19:08:35.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:35 smithi174 ceph-mon[87279]: pgmap v185: 65 pgs: 65 active+clean; 55 MiB data, 401 MiB used, 536 GiB / 536 GiB avail; 4.6 MiB/s wr, 344 op/s 2024-02-19T19:08:35.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:35 smithi174 ceph-mon[87279]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:35.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:35 smithi174 ceph-mon[87279]: from='client.34414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:35.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:35 smithi174 ceph-mon[87279]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:35.395 INFO:journalctl@ceph.mon.smithi174.smithi174.stdout:Feb 19 19:08:35 smithi174 ceph-mon[87279]: from='client.34418 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:35.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.415+0000 7eff73d72700 1 -- 172.21.15.114:0/3381206209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff6c1016a0 msgr2=0x7eff6c103a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:35.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.415+0000 7eff73d72700 1 --2- 172.21.15.114:0/3381206209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff6c1016a0 0x7eff6c103a90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7eff68009910 tx=0x7eff68009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:35.436 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:35 smithi114 ceph-mon[92983]: pgmap v185: 65 pgs: 65 active+clean; 55 MiB data, 401 MiB used, 536 GiB / 536 GiB avail; 4.6 MiB/s wr, 344 op/s 2024-02-19T19:08:35.436 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:35 smithi114 ceph-mon[92983]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:35.437 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:35 smithi114 ceph-mon[92983]: from='client.34414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:35.437 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:35 smithi114 ceph-mon[92983]: from='mgr.34104 172.21.15.114:0/4118886442' entity='mgr.smithi114.pcvtrk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:08:35.437 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:08:35 smithi114 ceph-mon[92983]: from='client.34418 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:08:35.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.415+0000 7eff73d72700 1 -- 172.21.15.114:0/3381206209 shutdown_connections 2024-02-19T19:08:35.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.415+0000 7eff73d72700 1 --2- 172.21.15.114:0/3381206209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff6c103fd0 0x7eff6c1063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:35.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.415+0000 7eff73d72700 1 --2- 172.21.15.114:0/3381206209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff6c1016a0 0x7eff6c103a90 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:35.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.415+0000 7eff73d72700 1 -- 172.21.15.114:0/3381206209 >> 172.21.15.114:0/3381206209 conn(0x7eff6c0fb030 msgr2=0x7eff6c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:35.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.415+0000 7eff73d72700 1 -- 172.21.15.114:0/3381206209 shutdown_connections 2024-02-19T19:08:35.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.415+0000 7eff73d72700 1 -- 172.21.15.114:0/3381206209 wait complete. 2024-02-19T19:08:35.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.416+0000 7eff73d72700 1 Processor -- start 2024-02-19T19:08:35.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.416+0000 7eff73d72700 1 -- start start 2024-02-19T19:08:35.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.416+0000 7eff73d72700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff6c1016a0 0x7eff6c198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:35.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.416+0000 7eff73d72700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff6c103fd0 0x7eff6c198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:35.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.416+0000 7eff73d72700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c199060 con 0x7eff6c103fd0 2024-02-19T19:08:35.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.416+0000 7eff73d72700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c1991a0 con 0x7eff6c1016a0 2024-02-19T19:08:35.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.416+0000 7eff7130d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff6c103fd0 0x7eff6c198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:35.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.416+0000 7eff7130d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff6c103fd0 0x7eff6c198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53400/0 (socket says 172.21.15.114:53400) 2024-02-19T19:08:35.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.416+0000 7eff7130d700 1 -- 172.21.15.114:0/865948384 learned_addr learned my addr 172.21.15.114:0/865948384 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:08:35.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.417+0000 7eff71b0e700 1 --2- 172.21.15.114:0/865948384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff6c1016a0 0x7eff6c198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:35.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.417+0000 7eff7130d700 1 -- 172.21.15.114:0/865948384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff6c1016a0 msgr2=0x7eff6c198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:35.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.417+0000 7eff7130d700 1 --2- 172.21.15.114:0/865948384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff6c1016a0 0x7eff6c198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:35.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.417+0000 7eff7130d700 1 -- 172.21.15.114:0/865948384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff680095f0 con 0x7eff6c103fd0 2024-02-19T19:08:35.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.417+0000 7eff7130d700 1 --2- 172.21.15.114:0/865948384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff6c103fd0 0x7eff6c198a40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00d8d0 tx=0x7eff5c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:35.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.417+0000 7eff62ffd700 1 -- 172.21.15.114:0/865948384 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff5c0098a0 con 0x7eff6c103fd0 2024-02-19T19:08:35.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.417+0000 7eff73d72700 1 -- 172.21.15.114:0/865948384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff6c19dc50 con 0x7eff6c103fd0 2024-02-19T19:08:35.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.417+0000 7eff62ffd700 1 -- 172.21.15.114:0/865948384 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7eff5c010460 con 0x7eff6c103fd0 2024-02-19T19:08:35.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.417+0000 7eff62ffd700 1 -- 172.21.15.114:0/865948384 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff5c01b620 con 0x7eff6c103fd0 2024-02-19T19:08:35.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.418+0000 7eff73d72700 1 -- 172.21.15.114:0/865948384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6c19e1a0 con 0x7eff6c103fd0 2024-02-19T19:08:35.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.419+0000 7eff73d72700 1 -- 172.21.15.114:0/865948384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff6c0fcc20 con 0x7eff6c103fd0 2024-02-19T19:08:35.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.420+0000 7eff62ffd700 1 -- 172.21.15.114:0/865948384 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7eff5c009a00 con 0x7eff6c103fd0 2024-02-19T19:08:35.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.420+0000 7eff62ffd700 1 --2- 172.21.15.114:0/865948384 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7eff5806d8a0 0x7eff5806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:35.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.420+0000 7eff62ffd700 1 -- 172.21.15.114:0/865948384 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6790+0+0 (secure 0 0 0) 0x7eff5c08d850 con 0x7eff6c103fd0 2024-02-19T19:08:35.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.421+0000 7eff71b0e700 1 --2- 172.21.15.114:0/865948384 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7eff5806d8a0 0x7eff5806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:35.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.421+0000 7eff71b0e700 1 --2- 172.21.15.114:0/865948384 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7eff5806d8a0 0x7eff5806fd60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7eff68006010 tx=0x7eff6800b560 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:35.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.429+0000 7eff62ffd700 1 -- 172.21.15.114:0/865948384 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7eff5c05aa60 con 0x7eff6c103fd0 2024-02-19T19:08:35.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.740+0000 7eff73d72700 1 -- 172.21.15.114:0/865948384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7eff6c061a40 con 0x7eff6c103fd0 2024-02-19T19:08:35.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.741+0000 7eff62ffd700 1 -- 172.21.15.114:0/865948384 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1860 (secure 0 0 0) 0x7eff5c05a5f0 con 0x7eff6c103fd0 2024-02-19T19:08:35.762 INFO:teuthology.orchestra.run.smithi114.stdout:e21 2024-02-19T19:08:35.762 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T19:08:35.763 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:08:35.763 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-02-19T19:08:35.763 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:08:35.763 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-02-19T19:08:35.763 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-02-19T19:08:35.763 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 21 2024-02-19T19:08:35.763 INFO:teuthology.orchestra.run.smithi114.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-19T19:08:35.763 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-02-19T19:00:00.979838+0000 2024-02-19T19:08:35.763 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-02-19T19:08:14.275100+0000 2024-02-19T19:08:35.763 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-02-19T19:08:35.763 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-02-19T19:08:35.763 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 60 2024-02-19T19:08:35.764 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-02-19T19:08:35.764 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-02-19T19:08:35.764 INFO:teuthology.orchestra.run.smithi114.stdout:max_xattr_size 65536 2024-02-19T19:08:35.764 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-02-19T19:08:35.764 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-02-19T19:08:35.764 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 0 2024-02-19T19:08:35.764 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T19:08:35.764 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 1 2024-02-19T19:08:35.764 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1 2024-02-19T19:08:35.764 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=14530,1=24269} 2024-02-19T19:08:35.764 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-02-19T19:08:35.764 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-02-19T19:08:35.764 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-02-19T19:08:35.764 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-02-19T19:08:35.765 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-02-19T19:08:35.765 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data enabled 2024-02-19T19:08:35.765 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-02-19T19:08:35.765 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-02-19T19:08:35.765 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 1 2024-02-19T19:08:35.765 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.tcwpod{0:14530} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.114:6828/450622269,v1:172.21.15.114:6829/450622269] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:08:35.765 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.znhuvd{1:24269} state up:stopping seq 4 join_fscid=1 addr [v2:172.21.15.174:6824/3869337142,v1:172.21.15.174:6825/3869337142] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:08:35.765 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:08:35.765 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:08:35.765 INFO:teuthology.orchestra.run.smithi114.stdout:Standby daemons: 2024-02-19T19:08:35.765 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-02-19T19:08:35.765 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi114.vpoeop{-1:34402} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.114:6826/2633507350,v1:172.21.15.114:6827/2633507350] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:08:35.765 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.cephfs.smithi174.stnfnf{-1:44289} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.174:6826/3615753934,v1:172.21.15.174:6827/3615753934] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T19:08:35.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.744+0000 7eff73d72700 1 -- 172.21.15.114:0/865948384 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7eff5806d8a0 msgr2=0x7eff5806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:35.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.744+0000 7eff73d72700 1 --2- 172.21.15.114:0/865948384 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7eff5806d8a0 0x7eff5806fd60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7eff68006010 tx=0x7eff6800b560 comp rx=0 tx=0).stop 2024-02-19T19:08:35.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.744+0000 7eff73d72700 1 -- 172.21.15.114:0/865948384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff6c103fd0 msgr2=0x7eff6c198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:35.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.744+0000 7eff73d72700 1 --2- 172.21.15.114:0/865948384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff6c103fd0 0x7eff6c198a40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00d8d0 tx=0x7eff5c00dbe0 comp rx=0 tx=0).stop 2024-02-19T19:08:35.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.744+0000 7eff73d72700 1 -- 172.21.15.114:0/865948384 shutdown_connections 2024-02-19T19:08:35.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.744+0000 7eff73d72700 1 --2- 172.21.15.114:0/865948384 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7eff5806d8a0 0x7eff5806fd60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:35.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.744+0000 7eff73d72700 1 --2- 172.21.15.114:0/865948384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff6c1016a0 0x7eff6c198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:35.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.744+0000 7eff73d72700 1 --2- 172.21.15.114:0/865948384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff6c103fd0 0x7eff6c198a40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:35.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.744+0000 7eff73d72700 1 -- 172.21.15.114:0/865948384 >> 172.21.15.114:0/865948384 conn(0x7eff6c0fb030 msgr2=0x7eff6c0ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:35.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.744+0000 7eff73d72700 1 -- 172.21.15.114:0/865948384 shutdown_connections 2024-02-19T19:08:35.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.744+0000 7eff73d72700 1 -- 172.21.15.114:0/865948384 wait complete. 2024-02-19T19:08:35.767 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-02-19T19:08:35.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.865+0000 7fa753342700 1 -- 172.21.15.114:0/404202495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa74c103fd0 msgr2=0x7fa74c1063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:35.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.865+0000 7fa753342700 1 --2- 172.21.15.114:0/404202495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa74c103fd0 0x7fa74c1063c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009910 tx=0x7fa73c009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:35.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.865+0000 7fa753342700 1 -- 172.21.15.114:0/404202495 shutdown_connections 2024-02-19T19:08:35.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.865+0000 7fa753342700 1 --2- 172.21.15.114:0/404202495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa74c103fd0 0x7fa74c1063c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:35.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.865+0000 7fa753342700 1 --2- 172.21.15.114:0/404202495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa74c1016a0 0x7fa74c103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:35.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.865+0000 7fa753342700 1 -- 172.21.15.114:0/404202495 >> 172.21.15.114:0/404202495 conn(0x7fa74c0fb030 msgr2=0x7fa74c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:35.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.865+0000 7fa753342700 1 -- 172.21.15.114:0/404202495 shutdown_connections 2024-02-19T19:08:35.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.865+0000 7fa753342700 1 -- 172.21.15.114:0/404202495 wait complete. 2024-02-19T19:08:35.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.866+0000 7fa753342700 1 Processor -- start 2024-02-19T19:08:35.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.866+0000 7fa753342700 1 -- start start 2024-02-19T19:08:35.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.866+0000 7fa753342700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa74c1016a0 0x7fa74c196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:35.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.866+0000 7fa753342700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa74c103fd0 0x7fa74c196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:35.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.866+0000 7fa753342700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c196e90 con 0x7fa74c103fd0 2024-02-19T19:08:35.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.866+0000 7fa753342700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c196fd0 con 0x7fa74c1016a0 2024-02-19T19:08:35.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.866+0000 7fa7508dd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa74c103fd0 0x7fa74c196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:35.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.866+0000 7fa7508dd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa74c103fd0 0x7fa74c196870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47110/0 (socket says 172.21.15.114:47110) 2024-02-19T19:08:35.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.866+0000 7fa7508dd700 1 -- 172.21.15.114:0/2003343202 learned_addr learned my addr 172.21.15.114:0/2003343202 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:08:35.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.866+0000 7fa7510de700 1 --2- 172.21.15.114:0/2003343202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa74c1016a0 0x7fa74c196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:35.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.867+0000 7fa7508dd700 1 -- 172.21.15.114:0/2003343202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa74c1016a0 msgr2=0x7fa74c196330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:35.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.867+0000 7fa7508dd700 1 --2- 172.21.15.114:0/2003343202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa74c1016a0 0x7fa74c196330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:35.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.867+0000 7fa7508dd700 1 -- 172.21.15.114:0/2003343202 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73c0095f0 con 0x7fa74c103fd0 2024-02-19T19:08:35.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.867+0000 7fa7510de700 1 --2- 172.21.15.114:0/2003343202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa74c1016a0 0x7fa74c196330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:08:35.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.867+0000 7fa7508dd700 1 --2- 172.21.15.114:0/2003343202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa74c103fd0 0x7fa74c196870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0092e0 tx=0x7fa73c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:35.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.867+0000 7fa7427fc700 1 -- 172.21.15.114:0/2003343202 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa73c01d070 con 0x7fa74c103fd0 2024-02-19T19:08:35.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.867+0000 7fa753342700 1 -- 172.21.15.114:0/2003343202 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74c19ba20 con 0x7fa74c103fd0 2024-02-19T19:08:35.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.867+0000 7fa7427fc700 1 -- 172.21.15.114:0/2003343202 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa73c005dc0 con 0x7fa74c103fd0 2024-02-19T19:08:35.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.867+0000 7fa7427fc700 1 -- 172.21.15.114:0/2003343202 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa73c017630 con 0x7fa74c103fd0 2024-02-19T19:08:35.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.868+0000 7fa753342700 1 -- 172.21.15.114:0/2003343202 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74c19bf10 con 0x7fa74c103fd0 2024-02-19T19:08:35.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.869+0000 7fa753342700 1 -- 172.21.15.114:0/2003343202 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa74c190520 con 0x7fa74c103fd0 2024-02-19T19:08:35.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.869+0000 7fa7427fc700 1 -- 172.21.15.114:0/2003343202 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7fa73c00faf0 con 0x7fa74c103fd0 2024-02-19T19:08:35.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.870+0000 7fa7427fc700 1 --2- 172.21.15.114:0/2003343202 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa73806d8a0 0x7fa73806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:35.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.870+0000 7fa7427fc700 1 -- 172.21.15.114:0/2003343202 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6790+0+0 (secure 0 0 0) 0x7fa73c08ea60 con 0x7fa74c103fd0 2024-02-19T19:08:35.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.870+0000 7fa7510de700 1 --2- 172.21.15.114:0/2003343202 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa73806d8a0 0x7fa73806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:35.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.870+0000 7fa7510de700 1 --2- 172.21.15.114:0/2003343202 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa73806d8a0 0x7fa73806fd60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa748011490 tx=0x7fa748009260 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:35.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:35.875+0000 7fa7427fc700 1 -- 172.21.15.114:0/2003343202 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa73c05bc70 con 0x7fa74c103fd0 2024-02-19T19:08:36.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.160+0000 7fa753342700 1 -- 172.21.15.114:0/2003343202 --> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa74c0611d0 con 0x7fa73806d8a0 2024-02-19T19:08:36.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.161+0000 7fa7427fc700 1 -- 172.21.15.114:0/2003343202 <== mgr.34104 v2:172.21.15.114:6800/3071028825 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa74c0611d0 con 0x7fa73806d8a0 2024-02-19T19:08:36.183 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-02-19T19:08:36.183 INFO:teuthology.orchestra.run.smithi114.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:08:36.183 INFO:teuthology.orchestra.run.smithi114.stdout: "in_progress": true, 2024-02-19T19:08:36.183 INFO:teuthology.orchestra.run.smithi114.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-19T19:08:36.183 INFO:teuthology.orchestra.run.smithi114.stdout: "services_complete": [ 2024-02-19T19:08:36.183 INFO:teuthology.orchestra.run.smithi114.stdout: "crash", 2024-02-19T19:08:36.183 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr", 2024-02-19T19:08:36.183 INFO:teuthology.orchestra.run.smithi114.stdout: "mon", 2024-02-19T19:08:36.184 INFO:teuthology.orchestra.run.smithi114.stdout: "osd" 2024-02-19T19:08:36.184 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-02-19T19:08:36.184 INFO:teuthology.orchestra.run.smithi114.stdout: "progress": "12/23 daemons upgraded", 2024-02-19T19:08:36.184 INFO:teuthology.orchestra.run.smithi114.stdout: "message": "Currently upgrading osd daemons", 2024-02-19T19:08:36.184 INFO:teuthology.orchestra.run.smithi114.stdout: "is_paused": false 2024-02-19T19:08:36.184 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-02-19T19:08:36.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.163+0000 7fa753342700 1 -- 172.21.15.114:0/2003343202 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa73806d8a0 msgr2=0x7fa73806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:36.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.163+0000 7fa753342700 1 --2- 172.21.15.114:0/2003343202 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa73806d8a0 0x7fa73806fd60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa748011490 tx=0x7fa748009260 comp rx=0 tx=0).stop 2024-02-19T19:08:36.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.163+0000 7fa753342700 1 -- 172.21.15.114:0/2003343202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa74c103fd0 msgr2=0x7fa74c196870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:36.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.163+0000 7fa753342700 1 --2- 172.21.15.114:0/2003343202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa74c103fd0 0x7fa74c196870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0092e0 tx=0x7fa73c00f690 comp rx=0 tx=0).stop 2024-02-19T19:08:36.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.163+0000 7fa753342700 1 -- 172.21.15.114:0/2003343202 shutdown_connections 2024-02-19T19:08:36.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.163+0000 7fa753342700 1 --2- 172.21.15.114:0/2003343202 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa73806d8a0 0x7fa73806fd60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:36.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.163+0000 7fa753342700 1 --2- 172.21.15.114:0/2003343202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa74c1016a0 0x7fa74c196330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:36.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.163+0000 7fa753342700 1 --2- 172.21.15.114:0/2003343202 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa74c103fd0 0x7fa74c196870 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:36.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.164+0000 7fa753342700 1 -- 172.21.15.114:0/2003343202 >> 172.21.15.114:0/2003343202 conn(0x7fa74c0fb030 msgr2=0x7fa74c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:36.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.164+0000 7fa753342700 1 -- 172.21.15.114:0/2003343202 shutdown_connections 2024-02-19T19:08:36.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.164+0000 7fa753342700 1 -- 172.21.15.114:0/2003343202 wait complete. 2024-02-19T19:08:36.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.265+0000 7fa465244700 1 -- 172.21.15.114:0/2575051121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa460101650 msgr2=0x7fa460103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:36.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.265+0000 7fa465244700 1 --2- 172.21.15.114:0/2575051121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa460101650 0x7fa460103a40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa450009910 tx=0x7fa450009c20 comp rx=0 tx=0).stop 2024-02-19T19:08:36.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.265+0000 7fa465244700 1 -- 172.21.15.114:0/2575051121 shutdown_connections 2024-02-19T19:08:36.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.265+0000 7fa465244700 1 --2- 172.21.15.114:0/2575051121 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa460103f80 0x7fa460106370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:36.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.265+0000 7fa465244700 1 --2- 172.21.15.114:0/2575051121 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa460101650 0x7fa460103a40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:36.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.265+0000 7fa465244700 1 -- 172.21.15.114:0/2575051121 >> 172.21.15.114:0/2575051121 conn(0x7fa4600fb060 msgr2=0x7fa4600fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:36.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.265+0000 7fa465244700 1 -- 172.21.15.114:0/2575051121 shutdown_connections 2024-02-19T19:08:36.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.265+0000 7fa465244700 1 -- 172.21.15.114:0/2575051121 wait complete. 2024-02-19T19:08:36.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.266+0000 7fa465244700 1 Processor -- start 2024-02-19T19:08:36.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.266+0000 7fa465244700 1 -- start start 2024-02-19T19:08:36.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.266+0000 7fa465244700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa460101650 0x7fa4601985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:36.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.266+0000 7fa465244700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa460103f80 0x7fa460198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:36.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.266+0000 7fa465244700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa460199140 con 0x7fa460101650 2024-02-19T19:08:36.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.266+0000 7fa465244700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa460199280 con 0x7fa460103f80 2024-02-19T19:08:36.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.266+0000 7fa45ed9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa460101650 0x7fa4601985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:36.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.266+0000 7fa45ed9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa460101650 0x7fa4601985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47134/0 (socket says 172.21.15.114:47134) 2024-02-19T19:08:36.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.266+0000 7fa45ed9d700 1 -- 172.21.15.114:0/2997529126 learned_addr learned my addr 172.21.15.114:0/2997529126 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-02-19T19:08:36.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.267+0000 7fa45ed9d700 1 -- 172.21.15.114:0/2997529126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa460103f80 msgr2=0x7fa460198b20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T19:08:36.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.267+0000 7fa45ed9d700 1 --2- 172.21.15.114:0/2997529126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa460103f80 0x7fa460198b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:36.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.267+0000 7fa45ed9d700 1 -- 172.21.15.114:0/2997529126 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4500095f0 con 0x7fa460101650 2024-02-19T19:08:36.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.267+0000 7fa45ed9d700 1 --2- 172.21.15.114:0/2997529126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa460101650 0x7fa4601985e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa450000c00 tx=0x7fa45000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:36.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.267+0000 7fa447fff700 1 -- 172.21.15.114:0/2997529126 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa45001d070 con 0x7fa460101650 2024-02-19T19:08:36.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.267+0000 7fa465244700 1 -- 172.21.15.114:0/2997529126 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa46019dcd0 con 0x7fa460101650 2024-02-19T19:08:36.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.267+0000 7fa447fff700 1 -- 172.21.15.114:0/2997529126 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa450022470 con 0x7fa460101650 2024-02-19T19:08:36.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.267+0000 7fa447fff700 1 -- 172.21.15.114:0/2997529126 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa45000f670 con 0x7fa460101650 2024-02-19T19:08:36.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.267+0000 7fa465244700 1 -- 172.21.15.114:0/2997529126 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa46019e1c0 con 0x7fa460101650 2024-02-19T19:08:36.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.268+0000 7fa465244700 1 -- 172.21.15.114:0/2997529126 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4600fcc30 con 0x7fa460101650 2024-02-19T19:08:36.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.269+0000 7fa447fff700 1 -- 172.21.15.114:0/2997529126 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 37) v1 ==== 91495+0+0 (secure 0 0 0) 0x7fa450004b30 con 0x7fa460101650 2024-02-19T19:08:36.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.269+0000 7fa447fff700 1 --2- 172.21.15.114:0/2997529126 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa44806d8a0 0x7fa44806fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:08:36.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.269+0000 7fa447fff700 1 -- 172.21.15.114:0/2997529126 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6790+0+0 (secure 0 0 0) 0x7fa45008f000 con 0x7fa460101650 2024-02-19T19:08:36.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.269+0000 7fa45e59c700 1 --2- 172.21.15.114:0/2997529126 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa44806d8a0 0x7fa44806fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:08:36.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.270+0000 7fa45e59c700 1 --2- 172.21.15.114:0/2997529126 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa44806d8a0 0x7fa44806fd60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa454009710 tx=0x7fa454006c60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:08:36.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.274+0000 7fa447fff700 1 -- 172.21.15.114:0/2997529126 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa45005c3e0 con 0x7fa460101650 2024-02-19T19:08:36.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.555+0000 7fa465244700 1 -- 172.21.15.114:0/2997529126 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa46002cc70 con 0x7fa460101650 2024-02-19T19:08:36.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.556+0000 7fa447fff700 1 -- 172.21.15.114:0/2997529126 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fa4500270e0 con 0x7fa460101650 2024-02-19T19:08:36.577 INFO:teuthology.orchestra.run.smithi114.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:08:36.577 INFO:teuthology.orchestra.run.smithi114.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:08:36.577 INFO:teuthology.orchestra.run.smithi114.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:08:36.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.558+0000 7fa465244700 1 -- 172.21.15.114:0/2997529126 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa44806d8a0 msgr2=0x7fa44806fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:36.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.558+0000 7fa465244700 1 --2- 172.21.15.114:0/2997529126 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa44806d8a0 0x7fa44806fd60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa454009710 tx=0x7fa454006c60 comp rx=0 tx=0).stop 2024-02-19T19:08:36.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.558+0000 7fa465244700 1 -- 172.21.15.114:0/2997529126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa460101650 msgr2=0x7fa4601985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:08:36.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.558+0000 7fa465244700 1 --2- 172.21.15.114:0/2997529126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa460101650 0x7fa4601985e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa450000c00 tx=0x7fa45000be20 comp rx=0 tx=0).stop 2024-02-19T19:08:36.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.558+0000 7fa465244700 1 -- 172.21.15.114:0/2997529126 shutdown_connections 2024-02-19T19:08:36.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.558+0000 7fa465244700 1 --2- 172.21.15.114:0/2997529126 >> [v2:172.21.15.114:6800/3071028825,v1:172.21.15.114:6801/3071028825] conn(0x7fa44806d8a0 0x7fa44806fd60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:36.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.558+0000 7fa465244700 1 --2- 172.21.15.114:0/2997529126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa460101650 0x7fa4601985e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:36.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.558+0000 7fa465244700 1 --2- 172.21.15.114:0/2997529126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa460103f80 0x7fa460198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:08:36.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.558+0000 7fa465244700 1 -- 172.21.15.114:0/2997529126 >> 172.21.15.114:0/2997529126 conn(0x7fa4600fb060 msgr2=0x7fa4600ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:08:36.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.558+0000 7fa465244700 1 -- 172.21.15.114:0/2997529126 shutdown_connections 2024-02-19T19:08:36.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:08:36.558+0000 7fa465244700 1 -- 172.21.15.114:0/2997529126 wait complete. 2024-02-19T19:08:57.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:08:57.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:08:57.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:08:57.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:08:57.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:08:57.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:08:58.243 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:08:57.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:08:58.243 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:08:57.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:08:58.243 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:08:57.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:08:58.243 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:08:57.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:08:58.243 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:08:57.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:08:58.243 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:08:57.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:08:58.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:08:58.353+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:08:58.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:08:58.353+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:08:58.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:08:58.353+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:08:59.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:08:59.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:08:59.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:08:59.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:08:59.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:08:59.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:08:59.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:08:58.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:08:59.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:08:58.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:08:59.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:08:58.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:08:59.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:08:59.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:08:59.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:08:59.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:08:59.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:08:59.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:00.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:08:59.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:00.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:08:59.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:00.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:08:59.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:00.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:00.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:00.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:00.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:00.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:00.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:00.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:00.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:00.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:00.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:00.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:00.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:01.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:01.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:01.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:01.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:01.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:01.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:01.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:00.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:01.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:00.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:01.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:00.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:01.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:01.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:01.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:01.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:01.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:01.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:02.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:01.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:02.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:01.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:02.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:01.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:02.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:02.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:02.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:02.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:02.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:02.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:02.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:02.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:02.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:02.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:02.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:02.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:03.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:02.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:03.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:02.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:03.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:02.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:03.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:02.840+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:03.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:02.840+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:03.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:02.840+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:03.551 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:03.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:03.551 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:03.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:03.551 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:03.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:04.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:03.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:04.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:03.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:04.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:03.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:04.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:04.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:04.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:04.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:04.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:04.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:04.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:04.422+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:04.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:04.422+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:04.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:04.422+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:05.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:04.901+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:05.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:04.901+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:05.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:04.901+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:05.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:05.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:05.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:05.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:05.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:05.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:05.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:05.421+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:05.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:05.421+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:05.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:05.421+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:06.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:05.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:06.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:05.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:06.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:05.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:06.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:06.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:06.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:06.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:06.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:06.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:06.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:06.413+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:06.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:06.413+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:06.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:06.413+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:07.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:06.940+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:07.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:06.940+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:07.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:06.940+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:07.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:07.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:07.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:07.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:07.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:07.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:07.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:07.385+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:07.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:07.385+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:07.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:07.385+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:08.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:07.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:08.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:07.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:08.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:07.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:08.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:07.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:08.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:07.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:08.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:07.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:08.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:08.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:08.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:08.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:08.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:08.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:09.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:08.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:09.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:08.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:09.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:08.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:09.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:09.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:09.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:09.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:09.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:09.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:09.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:09.376+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:09.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:09.376+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:09.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:09.376+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:10.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:10.041+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:10.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:10.041+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:10.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:10.041+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:10.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:09.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:10.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:09.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:10.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:09.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:10.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:10.376+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:10.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:10.376+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:10.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:10.376+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:11.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:11.068+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:11.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:11.068+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:11.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:11.069+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:11.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:10.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:11.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:10.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:11.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:10.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:11.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:11.337+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:11.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:11.337+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:11.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:11.337+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:12.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:11.924+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:12.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:11.924+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:12.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:11.924+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:12.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:12.086+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:12.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:12.086+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:12.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:12.086+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:12.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:12.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:12.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:12.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:12.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:12.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:13.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:13.042+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:13.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:13.042+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:13.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:13.042+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:13.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:12.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:13.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:12.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:13.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:12.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:13.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:13.363+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:13.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:13.363+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:13.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:13.363+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:14.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:14.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:14.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:14.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:14.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:14.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:14.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:13.840+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:14.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:13.840+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:14.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:13.840+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:14.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:14.324+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:14.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:14.324+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:14.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:14.324+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:15.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:15.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:15.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:15.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:15.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:15.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:15.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:14.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:15.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:14.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:15.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:14.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:15.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:15.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:15.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:15.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:15.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:15.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:16.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:15.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:16.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:15.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:16.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:15.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:16.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:15.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:16.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:15.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:16.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:15.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:16.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:16.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:16.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:16.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:16.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:16.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:17.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:16.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:17.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:16.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:17.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:16.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:17.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:16.843+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:17.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:16.843+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:17.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:16.843+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:17.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:17.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:17.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:17.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:17.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:17.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:18.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:17.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:18.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:17.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:18.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:17.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:18.280 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:17.877+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:18.280 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:17.877+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:18.280 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:17.877+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:18.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:18.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:18.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:18.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:18.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:18.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:19.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:18.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:19.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:18.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:19.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:18.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:19.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:18.890+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:19.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:18.890+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:19.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:18.890+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:19.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:19.242+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:19.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:19.242+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:19.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:19.242+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:20.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:19.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:20.278 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:19.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:20.278 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:19.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:20.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:19.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:20.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:19.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:20.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:19.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:20.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:20.256+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:20.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:20.256+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:20.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:20.256+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:21.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:20.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:21.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:20.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:21.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:20.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:21.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:20.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:21.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:20.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:21.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:20.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:21.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:21.254+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:21.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:21.254+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:21.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:21.254+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:22.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:21.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:22.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:21.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:22.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:21.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:22.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:21.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:22.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:21.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:22.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:21.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:22.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:22.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:22.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:22.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:22.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:22.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:23.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:22.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:23.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:22.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:23.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:22.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:23.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:22.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:23.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:22.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:23.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:22.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:23.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:23.249+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:23.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:23.249+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:23.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:23.249+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:24.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:23.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:24.282 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:23.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:24.282 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:23.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:24.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:23.927+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:24.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:23.927+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:24.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:23.927+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:24.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:24.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:24.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:24.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:24.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:24.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:25.286 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:24.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:25.286 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:24.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:25.286 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:24.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:25.286 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:24.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:25.286 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:24.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:25.286 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:24.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:25.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:25.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:25.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:25.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:25.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:25.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:26.252 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:25.824+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:26.252 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:25.824+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:26.252 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:25.824+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:26.252 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:25.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:26.252 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:25.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:26.252 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:25.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:26.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:26.230+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:26.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:26.230+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:26.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:26.230+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:27.246 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:26.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:27.247 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:26.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:27.247 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:26.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:27.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:26.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:27.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:26.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:27.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:26.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:27.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:27.225+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:27.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:27.225+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:27.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:27.225+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:28.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:27.857+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:28.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:27.857+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:28.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:27.857+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:28.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:27.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:28.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:27.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:28.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:27.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:28.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:28.196+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:28.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:28.196+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:28.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:28.196+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:29.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:28.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:29.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:28.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:29.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:28.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:29.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:28.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:29.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:28.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:29.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:28.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:29.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:29.165+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:29.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:29.165+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:29.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:29.165+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:30.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:29.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:30.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:29.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:30.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:29.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:30.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:30.121+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:30.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:30.121+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:30.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:30.121+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:30.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:29.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:30.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:29.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:30.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:29.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:31.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:30.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:31.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:30.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:31.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:30.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:31.290 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:31.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:31.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:31.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:31.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:31.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:31.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:30.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:31.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:30.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:31.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:30.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:32.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:31.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:32.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:31.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:32.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:31.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:32.290 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:32.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:32.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:32.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:32.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:32.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:32.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:31.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:32.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:31.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:32.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:31.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:33.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:32.940+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:33.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:32.940+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:33.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:32.940+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:33.290 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:33.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:33.290 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:33.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:33.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:33.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:33.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:32.917+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:33.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:32.917+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:33.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:32.917+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:34.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:33.927+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:34.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:33.927+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:34.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:33.927+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:34.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:34.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:34.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:34.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:34.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:34.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:34.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:33.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:34.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:33.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:34.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:33.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:35.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:34.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:35.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:34.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:35.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:34.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:35.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:35.087+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:35.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:35.087+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:35.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:35.087+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:35.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:34.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:35.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:34.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:35.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:34.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:36.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:35.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:36.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:35.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:36.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:35.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:36.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:35.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:36.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:35.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:36.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:35.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:36.444 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:36.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:36.445 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:36.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:36.445 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:36.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:37.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:36.894+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:37.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:36.894+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:37.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:36.894+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:37.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:36.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:37.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:36.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:37.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:36.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:37.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:37.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:37.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:37.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:37.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:37.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:37.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:37.790+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:37.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:37.790+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:37.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:37.790+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:38.219 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:37.917+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:38.220 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:37.917+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:38.220 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:37.917+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:38.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:38.198+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:38.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:38.198+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:38.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:38.198+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:38.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:38.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:38.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:38.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:38.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:38.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:39.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:38.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:39.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:38.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:39.245 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:38.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:39.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:39.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:39.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:39.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:39.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:39.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:40.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:39.768+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:40.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:39.768+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:40.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:39.768+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:40.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:39.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:40.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:39.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:40.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:39.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:40.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:40.247+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:40.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:40.247+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:40.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:40.247+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:41.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:40.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:41.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:40.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:41.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:40.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:41.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:40.745+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:41.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:40.745+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:41.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:40.745+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:41.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:41.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:41.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:41.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:41.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:41.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:42.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:41.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:42.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:41.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:42.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:41.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:42.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:41.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:42.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:41.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:42.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:41.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:42.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:42.226+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:42.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:42.226+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:42.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:42.226+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:43.250 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:42.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:43.250 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:42.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:43.250 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:42.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:43.251 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:42.853+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:43.251 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:42.853+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:43.251 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:42.853+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:43.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:43.229+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:43.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:43.229+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:43.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:43.229+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:43.816 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:43.738+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:43.816 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:43.738+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:43.816 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:43.738+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:44.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:43.871+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:44.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:43.871+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:44.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:43.871+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:44.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:44.255+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:44.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:44.255+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:44.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:44.255+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:45.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:44.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:45.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:44.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:45.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:44.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:45.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:44.872+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:45.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:44.872+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:45.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:44.872+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:45.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:45.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:45.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:45.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:45.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:45.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:46.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:45.838+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:46.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:45.838+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:46.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:45.838+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:46.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:45.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:46.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:45.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:46.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:45.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:46.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:46.339+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:46.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:46.339+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:46.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:46.339+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:47.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:46.880+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:47.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:46.880+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:47.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:46.880+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:47.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:46.794+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:47.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:46.794+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:47.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:46.794+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:47.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:47.344+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:47.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:47.344+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:47.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:47.344+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:48.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:47.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:48.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:47.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:48.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:47.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:48.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:47.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:48.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:47.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:48.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:47.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:48.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:48.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:48.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:48.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:48.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:48.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:49.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:48.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:49.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:48.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:49.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:48.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:49.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:48.789+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:49.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:48.789+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:49.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:48.789+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:49.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:49.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:49.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:49.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:49.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:49.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:50.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:49.749+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:50.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:49.749+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:50.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:49.749+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:50.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:49.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:50.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:49.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:50.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:49.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:50.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:50.324+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:50.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:50.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:50.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:50.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:51.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:50.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:51.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:50.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:51.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:50.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:51.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:50.853+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:51.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:50.853+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:51.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:50.853+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:51.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:51.291+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:51.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:51.291+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:51.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:51.291+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:52.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:51.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:52.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:51.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:52.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:51.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:52.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:51.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:52.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:51.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:52.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:51.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:52.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:52.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:52.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:52.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:52.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:52.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:53.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:52.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:53.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:52.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:53.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:52.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:53.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:52.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:53.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:52.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:53.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:52.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:53.602 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:53.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:53.602 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:53.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:53.602 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:53.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:54.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:53.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:54.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:53.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:54.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:53.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:54.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:53.849+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:54.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:53.849+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:54.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:53.849+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:54.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:54.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:54.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:54.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:54.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:54.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:55.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:54.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:55.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:54.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:55.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:54.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:55.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:54.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:55.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:54.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:55.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:54.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:55.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:55.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:55.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:55.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:55.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:55.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:56.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:55.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:56.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:55.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:56.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:55.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:56.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:55.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:56.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:55.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:56.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:55.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:56.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:56.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:56.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:56.294+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:56.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:56.294+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:57.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:56.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:57.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:56.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:57.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:56.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:57.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:56.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:57.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:56.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:57.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:56.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:57.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:57.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:57.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:57.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:57.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:57.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:58.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:57.844+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:58.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:57.844+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:58.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:57.844+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:58.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:57.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:58.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:57.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:58.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:57.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:58.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:58.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:58.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:58.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:58.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:58.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:59.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:58.806+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:59.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:58.806+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:59.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:58.806+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:09:59.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:58.999+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:59.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:58.999+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:59.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:09:58.999+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:09:59.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:59.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:59.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:59.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:09:59.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:09:59.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:00.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:59.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:00.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:59.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:00.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:09:59.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:00.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:00.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:00.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:00.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:00.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:00.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:00.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:00.311+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:00.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:00.311+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:00.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:00.311+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:01.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:00.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:01.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:00.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:01.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:00.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:01.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:01.023+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:01.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:01.023+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:01.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:01.023+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:01.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:01.286+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:01.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:01.286+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:01.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:01.286+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:02.248 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:02.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:02.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:02.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:02.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:02.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:02.249 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:01.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:02.249 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:01.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:02.249 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:01.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:02.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:02.266+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:02.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:02.266+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:02.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:02.266+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:03.278 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:02.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:03.278 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:02.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:03.278 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:02.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:03.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:02.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:03.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:02.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:03.279 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:02.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:03.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:03.256+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:03.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:03.256+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:03.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:03.256+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:04.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:03.996+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:04.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:03.996+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:04.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:03.996+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:04.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:03.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:04.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:03.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:04.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:03.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:04.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:04.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:04.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:04.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:04.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:04.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:05.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:05.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:05.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:05.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:05.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:05.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:05.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:04.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:05.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:04.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:05.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:04.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:05.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:05.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:05.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:05.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:05.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:05.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:06.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:06.045+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:06.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:06.045+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:06.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:06.045+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:06.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:05.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:06.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:05.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:06.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:05.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:06.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:06.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:06.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:06.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:06.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:06.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:07.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:07.092+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:07.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:07.092+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:07.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:07.092+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:07.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:06.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:07.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:06.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:07.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:06.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:07.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:07.311+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:07.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:07.311+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:07.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:07.311+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:08.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:08.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:08.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:08.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:08.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:08.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:08.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:07.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:08.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:07.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:08.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:07.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:08.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:08.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:08.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:08.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:08.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:08.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:09.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:09.074+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:09.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:09.074+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:09.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:09.074+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:09.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:08.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:09.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:08.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:09.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:08.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:09.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:09.328+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:09.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:09.328+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:09.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:09.328+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:10.131 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:09.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:10.131 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:09.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:10.131 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:09.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:10.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:10.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:10.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:10.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:10.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:10.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:10.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:10.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:10.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:10.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:10.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:10.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:11.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:10.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:11.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:10.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:11.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:10.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:11.446 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:11.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:11.446 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:11.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:11.447 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:11.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:11.447 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:11.291+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:11.447 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:11.291+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:11.447 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:11.291+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:12.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:11.811+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:12.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:11.811+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:12.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:11.811+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:12.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:12.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:12.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:12.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:12.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:12.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:12.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:12.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:12.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:12.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:12.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:12.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:13.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:12.770+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:13.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:12.770+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:13.204 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:12.770+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:13.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:13.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:13.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:13.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:13.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:13.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:13.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:13.324+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:13.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:13.324+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:13.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:13.324+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:14.226 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:13.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:14.226 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:13.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:14.226 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:13.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:14.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:14.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:14.536 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:14.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:14.536 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:14.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:14.536 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:14.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:14.536 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:14.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:14.536 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:14.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:15.223 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:14.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:15.224 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:14.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:15.224 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:14.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:15.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:15.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:15.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:15.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:15.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:15.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:15.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:15.333+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:15.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:15.333+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:15.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:15.333+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:16.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:15.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:16.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:15.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:16.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:15.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:16.447 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:16.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:16.448 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:16.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:16.448 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:16.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:16.448 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:16.301+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:16.448 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:16.301+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:16.448 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:16.301+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:17.148 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:16.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:17.148 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:16.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:17.148 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:16.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:17.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:17.126+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:17.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:17.126+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:17.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:17.126+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:17.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:17.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:17.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:17.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:17.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:17.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:18.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:17.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:18.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:17.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:18.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:17.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:18.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:18.287+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:18.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:18.287+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:18.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:18.287+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:18.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:18.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:18.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:18.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:18.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:18.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:19.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:18.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:19.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:18.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:19.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:18.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:19.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:19.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:19.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:19.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:19.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:19.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:19.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:19.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:19.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:19.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:19.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:19.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:20.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:19.733+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:20.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:19.733+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:20.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:19.733+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:20.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:20.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:20.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:20.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:20.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:20.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:20.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:20.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:20.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:20.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:20.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:20.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:21.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:20.777+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:21.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:20.777+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:21.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:20.777+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:21.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:21.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:21.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:21.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:21.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:21.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:21.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:21.252+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:21.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:21.252+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:21.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:21.252+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:22.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:21.783+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:22.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:21.783+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:22.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:21.783+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:22.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:22.087+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:22.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:22.087+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:22.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:22.087+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:22.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:22.255+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:22.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:22.255+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:22.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:22.255+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:23.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:22.828+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:23.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:22.828+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:23.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:22.828+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:23.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:23.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:23.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:23.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:23.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:23.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:23.556 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:23.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:23.556 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:23.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:23.556 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:23.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:24.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:23.865+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:24.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:23.865+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:24.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:23.865+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:24.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:24.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:24.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:24.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:24.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:24.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:24.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:24.237+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:24.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:24.237+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:24.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:24.237+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:25.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:25.064+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:25.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:25.064+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:25.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:25.064+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:25.232 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:24.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:25.232 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:24.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:25.232 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:24.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:25.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:25.211+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:25.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:25.211+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:25.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:25.211+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:26.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:25.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:26.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:25.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:26.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:25.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:26.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:26.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:26.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:26.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:26.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:26.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:26.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:26.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:26.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:26.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:26.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:26.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:27.095 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:26.792+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:27.095 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:26.792+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:27.095 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:26.792+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:27.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:27.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:27.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:27.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:27.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:27.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:27.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:27.261+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:27.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:27.261+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:27.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:27.261+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:28.132 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:27.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:28.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:27.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:28.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:27.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:28.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:28.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:28.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:28.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:28.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:28.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:28.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:28.110+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:28.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:28.110+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:28.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:28.110+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:29.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:28.784+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:29.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:28.784+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:29.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:28.784+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:29.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:29.104+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:29.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:29.104+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:29.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:29.104+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:29.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:29.353+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:29.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:29.353+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:29.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:29.353+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:30.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:29.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:30.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:29.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:30.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:29.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:30.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:30.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:30.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:30.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:30.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:30.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:30.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:30.361+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:30.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:30.361+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:30.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:30.361+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:31.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:30.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:31.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:30.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:31.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:30.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:31.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:31.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:31.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:31.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:31.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:31.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:31.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:31.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:31.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:31.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:31.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:31.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:32.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:31.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:32.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:31.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:32.204 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:31.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:32.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:32.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:32.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:32.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:32.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:32.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:32.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:32.338+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:32.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:32.338+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:32.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:32.338+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:32.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:32.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:32.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:33.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:33.194+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:33.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:33.194+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:33.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:33.194+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:33.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:33.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:33.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:33.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:33.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:33.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:34.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:33.811+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:34.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:33.811+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:34.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:33.811+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:34.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:34.152+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:34.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:34.152+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:34.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:34.152+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:34.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:34.308+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:34.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:34.308+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:34.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:34.308+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:35.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:34.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:35.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:34.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:35.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:34.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:35.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:35.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:35.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:35.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:35.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:35.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:35.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:35.353+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:35.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:35.353+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:35.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:35.353+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:36.221 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:35.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:36.221 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:35.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:36.221 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:35.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:36.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:36.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:36.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:36.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:36.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:36.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:36.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:36.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:36.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:36.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:36.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:36.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:37.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:36.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:37.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:36.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:37.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:36.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:37.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:37.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:37.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:37.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:37.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:37.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:37.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:37.246+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:37.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:37.246+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:37.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:37.246+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:38.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:37.864+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:38.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:37.864+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:38.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:37.864+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:38.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:38.246+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:38.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:38.246+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:38.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:38.246+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:38.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:38.330+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:38.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:38.330+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:38.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:38.330+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:39.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:38.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:39.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:38.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:39.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:38.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:39.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:39.287+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:39.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:39.287+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:39.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:39.287+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:39.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:39.246+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:39.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:39.246+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:39.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:39.246+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:40.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:39.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:40.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:39.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:40.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:39.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:40.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:40.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:40.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:40.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:40.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:40.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:40.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:40.334+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:40.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:40.334+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:40.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:40.334+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:41.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:40.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:41.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:40.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:41.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:40.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:41.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:10:41.306+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:10:41.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:41.377+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:41.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:41.377+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:41.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:41.377+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:41.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:41.287+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:41.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:41.287+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:41.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:41.287+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:42.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:41.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:42.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:41.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:42.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:41.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:42.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:42.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:42.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:42.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:42.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:42.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:42.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:42.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:42.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:42.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:42.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:42.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:43.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:42.784+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:43.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:42.784+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:43.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:42.784+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:43.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:43.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:43.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:43.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:43.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:43.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:43.603 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:43.403+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:43.603 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:43.403+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:43.603 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:43.403+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:44.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:43.792+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:44.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:43.792+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:44.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:43.792+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:44.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:44.410+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:44.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:44.410+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:44.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:44.410+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:44.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:44.352+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:44.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:44.352+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:44.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:44.352+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:45.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:44.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:45.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:44.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:45.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:44.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:45.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:45.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:45.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:45.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:45.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:45.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:45.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:45.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:45.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:45.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:45.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:45.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:46.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:45.830+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:46.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:45.830+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:46.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:45.830+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:46.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:10:46.306+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 20 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:10:46.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:46.399+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:46.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:46.399+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:46.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:46.399+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:46.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:46.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:46.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:46.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:46.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:46.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:47.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:46.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:47.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:46.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:47.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:46.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:47.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:47.435+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:47.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:47.435+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:47.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:47.435+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:47.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:47.333+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:47.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:47.333+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:47.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:47.333+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:48.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:47.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:48.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:47.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:48.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:47.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:48.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:48.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:48.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:48.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:48.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:48.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:48.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:48.466+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:48.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:48.466+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:48.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:48.466+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:49.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:48.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:49.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:48.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:49.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:48.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:49.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:49.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:49.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:49.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:49.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:49.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:49.785 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:49.478+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:49.785 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:49.478+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:49.785 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:49.478+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:50.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:49.762+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:50.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:49.762+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:50.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:49.762+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:50.748 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:50.298+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:50.748 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:50.298+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:50.748 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:50.298+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:50.749 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:50.512+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:50.749 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:50.512+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:50.749 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:50.512+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:51.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:50.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:51.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:50.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:51.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:50.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:51.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:10:51.307+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 33 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:10:51.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:51.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:51.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:51.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:51.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:51.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:52.040 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:51.532+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:52.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:51.532+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:52.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:51.532+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:52.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:51.712+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:52.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:51.712+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:52.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:51.712+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:52.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:52.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:52.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:52.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:52.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:52.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:52.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:52.493+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:52.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:52.493+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:52.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:52.493+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:53.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:52.681+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:53.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:52.681+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:53.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:52.681+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:53.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:53.303+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:53.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:53.303+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:53.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:53.303+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:53.603 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:53.497+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:53.603 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:53.497+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:53.603 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:53.497+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:54.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:53.635+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:54.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:53.635+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:54.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:53.635+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:54.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:54.308+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:54.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:54.308+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:54.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:54.308+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:54.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:54.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:54.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:54.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:54.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:54.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:55.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:54.652+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:55.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:54.652+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:55.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:54.652+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:55.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:55.451+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:55.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:55.451+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:55.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:55.451+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:55.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:55.304+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:55.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:55.304+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:55.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:55.304+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:56.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:55.695+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:56.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:55.695+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:56.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:55.695+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:56.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:10:56.307+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:10:56.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:56.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:56.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:56.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:56.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:56.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:56.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:56.495+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:56.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:56.495+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:56.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:56.495+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:57.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:56.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:57.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:56.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:57.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:56.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:57.705 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:57.525+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:57.705 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:57.525+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:57.705 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:57.525+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:57.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:57.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:57.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:57.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:57.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:57.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:58.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:57.683+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:58.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:57.683+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:58.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:57.683+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:58.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:58.496+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:58.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:58.496+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:58.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:58.496+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:58.659 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:58.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:58.659 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:58.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:58.659 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:58.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:59.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:58.636+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:59.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:58.636+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:59.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:58.636+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:10:59.654 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:59.470+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:59.654 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:59.470+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:59.654 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:10:59.470+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:10:59.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:59.339+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:59.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:59.339+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:10:59.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:10:59.339+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:00.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:59.631+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:00.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:59.631+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:00.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:10:59.631+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:00.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:00.386+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:00.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:00.386+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:00.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:00.386+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:00.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:00.431+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:00.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:00.431+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:00.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:00.431+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:00.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:00.591+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:00.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:00.591+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:00.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:00.591+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:01.623 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:11:01.307+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 88 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:11:01.624 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:01.470+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:01.624 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:01.470+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:01.624 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:01.470+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:01.624 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:01.424+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:01.624 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:01.424+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:01.624 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:01.424+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:02.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:01.601+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:02.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:01.601+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:02.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:01.601+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:02.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:02.374+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:02.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:02.374+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:02.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:02.374+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:02.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:02.431+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:02.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:02.431+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:02.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:02.431+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:02.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:02.574+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:02.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:02.574+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:02.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:02.574+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:03.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:03.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:03.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:03.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:03.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:03.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:03.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:03.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:03.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:03.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:03.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:03.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:03.689 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:03.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:03.689 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:03.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:03.689 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:03.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:04.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:04.403+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:04.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:04.403+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:04.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:04.403+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:04.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:04.401+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:04.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:04.401+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:04.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:04.401+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:04.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:04.560+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:04.792 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:04.560+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:04.792 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:04.560+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:05.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:05.382+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:05.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:05.382+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:05.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:05.382+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:05.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:05.392+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:05.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:05.392+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:05.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:05.392+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:05.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:05.557+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:05.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:05.557+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:05.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:05.557+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:06.761 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:11:06.308+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 118 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:11:06.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:06.421+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:06.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:06.421+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:06.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:06.421+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:06.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:06.341+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:06.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:06.341+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:06.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:06.341+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:06.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:06.547+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:06.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:06.547+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:06.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:06.547+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:07.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:07.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:07.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:07.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:07.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:07.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:07.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:07.377+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:07.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:07.377+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:07.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:07.377+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:07.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:07.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:07.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:07.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:07.792 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:07.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:08.612 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:08.424+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:08.671 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:08.424+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:08.671 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:08.424+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:08.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:08.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:08.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:08.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:08.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:08.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:09.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:08.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:09.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:08.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:09.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:08.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:09.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:09.394+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:09.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:09.394+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:09.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:09.394+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:09.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:09.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:09.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:09.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:09.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:09.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:10.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:09.567+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:10.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:09.567+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:10.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:09.567+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:10.617 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:10.397+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:10.617 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:10.397+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:10.617 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:10.397+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:10.618 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:10.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:10.618 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:10.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:10.618 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:10.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:11.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:10.595+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:11.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:10.595+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:11.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:10.595+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:11.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:11:11.308+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 145 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:11:11.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:11.268+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:11.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:11.268+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:11.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:11.268+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:11.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:11.374+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:11.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:11.374+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:11.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:11.374+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:12.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:11.610+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:12.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:11.610+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:12.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:11.610+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:12.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:12.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:12.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:12.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:12.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:12.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:12.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:12.405+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:12.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:12.405+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:12.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:12.405+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:13.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:12.622+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:13.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:12.622+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:13.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:12.622+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:13.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:13.229+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:13.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:13.229+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:13.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:13.229+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:13.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:13.375+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:13.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:13.375+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:13.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:13.375+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:14.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:13.583+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:14.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:13.583+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:14.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:13.583+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:14.572 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:14.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:14.572 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:14.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:14.572 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:14.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:14.572 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:14.391+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:14.572 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:14.391+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:14.572 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:14.391+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:15.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:14.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:15.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:14.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:15.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:14.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:15.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:15.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:15.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:15.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:15.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:15.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:15.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:15.564+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:15.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:15.564+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:15.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:15.564+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:15.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:15.317+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:15.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:15.317+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:15.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:15.317+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:16.584 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:11:16.309+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 179 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:11:16.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:16.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:16.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:16.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:16.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:16.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:16.585 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:16.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:16.585 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:16.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:16.585 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:16.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:17.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:16.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:17.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:16.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:17.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:16.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:17.610 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:17.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:17.610 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:17.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:17.610 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:17.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:17.611 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:17.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:17.611 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:17.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:17.611 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:17.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:18.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:17.588+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:18.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:17.588+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:18.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:17.588+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:18.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:18.388+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:18.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:18.388+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:18.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:18.388+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:18.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:18.267+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:18.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:18.267+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:18.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:18.267+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:19.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:18.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:19.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:18.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:19.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:18.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:19.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:19.240+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:19.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:19.240+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:19.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:19.240+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:19.751 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:19.570+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:19.751 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:19.570+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:19.751 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:19.570+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:19.751 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:19.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:19.751 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:19.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:19.751 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:19.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:20.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:20.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:20.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:20.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:20.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:20.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:20.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:20.389+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:20.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:20.389+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:20.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:20.389+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:21.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:20.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:21.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:20.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:21.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:20.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:21.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:11:21.309+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 206 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:11:21.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:21.383+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:21.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:21.383+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:21.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:21.383+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:21.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:21.228+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:21.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:21.228+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:21.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:21.228+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:22.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:21.618+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:22.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:21.618+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:22.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:21.618+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:22.390 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:22.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:22.391 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:22.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:22.391 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:22.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:22.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:22.368+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:22.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:22.368+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:22.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:22.368+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:22.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:22.611+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:22.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:22.611+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:22.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:22.611+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:23.603 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:23.338+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:23.603 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:23.338+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:23.603 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:23.338+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:23.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:23.302+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:23.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:23.302+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:23.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:23.302+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:24.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:23.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:24.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:23.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:24.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:23.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:24.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:24.343+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:24.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:24.343+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:24.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:24.343+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:24.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:24.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:24.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:24.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:24.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:24.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:25.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:24.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:25.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:24.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:25.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:24.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:25.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:25.360+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:25.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:25.360+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:25.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:25.360+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:25.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:25.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:25.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:25.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:25.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:25.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:26.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:25.688+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:26.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:25.688+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:26.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:25.688+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:26.710 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:11:26.310+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 239 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:11:26.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:26.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:26.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:26.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:26.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:26.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:26.711 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:26.307+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:26.711 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:26.307+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:26.711 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:26.307+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:27.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:26.687+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:27.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:26.687+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:27.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:26.687+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:27.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:27.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:27.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:27.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:27.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:27.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:27.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:27.272+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:27.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:27.273+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:27.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:27.273+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:28.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:27.692+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:28.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:27.692+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:28.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:27.692+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:28.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:28.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:28.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:28.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:28.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:28.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:28.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:28.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:28.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:28.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:28.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:28.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:29.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:28.734+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:29.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:28.734+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:29.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:28.734+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:29.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:29.254+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:29.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:29.254+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:29.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:29.254+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:29.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:29.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:29.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:29.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:29.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:29.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:30.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:29.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:30.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:29.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:30.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:29.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:30.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:30.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:30.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:30.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:30.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:30.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:30.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:30.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:30.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:30.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:30.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:30.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:31.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:30.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:31.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:30.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:31.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:30.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:31.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:11:31.310+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 266 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:11:31.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:31.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:31.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:31.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:31.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:31.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:31.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:31.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:31.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:31.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:31.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:31.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:32.221 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:31.773+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:32.221 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:31.773+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:32.221 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:31.773+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:32.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:32.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:32.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:32.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:32.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:32.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:32.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:32.302+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:32.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:32.302+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:32.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:32.302+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:33.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:32.768+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:33.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:32.768+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:33.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:32.768+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:33.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:33.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:33.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:33.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:33.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:33.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:33.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:33.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:33.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:33.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:33.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:33.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:34.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:33.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:34.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:33.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:34.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:33.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:34.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:34.266+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:34.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:34.266+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:34.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:34.266+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:34.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:34.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:34.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:34.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:34.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:34.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:35.239 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:34.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:35.239 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:34.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:35.239 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:34.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:35.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:35.217+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:35.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:35.217+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:35.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:35.217+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:35.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:35.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:35.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:35.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:35.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:35.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:36.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:35.862+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:36.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:35.862+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:36.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:35.862+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:36.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:11:36.311+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 300 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:11:36.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:36.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:36.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:36.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:36.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:36.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:36.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:36.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:36.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:36.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:36.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:36.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:37.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:36.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:37.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:36.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:37.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:36.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:37.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:37.192+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:37.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:37.192+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:37.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:37.192+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:37.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:37.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:37.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:37.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:37.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:37.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:38.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:37.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:38.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:37.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:38.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:37.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:38.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:38.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:38.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:38.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:38.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:38.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:38.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:38.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:38.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:38.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:38.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:38.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:38.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:38.236+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.13:255 2.13 2:ceffbd45:::200.0000000a:head [write 527852~2382 [fadvise_dontneed] in=2382b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:39.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:38.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:39.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:38.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:39.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:38.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:39.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:38.908+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:39.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:39.183+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:39.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:39.183+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:39.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:39.183+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:39.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:39.184+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:39.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:39.191+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:39.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:39.191+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:39.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:39.191+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:39.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:39.192+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.13:255 2.13 2:ceffbd45:::200.0000000a:head [write 527852~2382 [fadvise_dontneed] in=2382b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:40.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:39.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:40.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:39.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:40.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:39.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:40.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:39.874+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:40.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:40.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:40.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:40.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:40.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:40.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:40.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:40.154+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.13:255 2.13 2:ceffbd45:::200.0000000a:head [write 527852~2382 [fadvise_dontneed] in=2382b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:40.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:40.232+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:40.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:40.232+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:40.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:40.232+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:40.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:40.233+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:41.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:40.865+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:41.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:40.865+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:41.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:40.865+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:41.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:40.866+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:41.456 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:11:41.311+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 333 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:11:41.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:41.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:41.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:41.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:41.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:41.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:41.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:41.165+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.13:255 2.13 2:ceffbd45:::200.0000000a:head [write 527852~2382 [fadvise_dontneed] in=2382b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:41.457 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:41.215+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:41.457 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:41.215+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:41.457 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:41.215+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:41.457 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:41.215+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:42.138 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:41.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:42.139 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:41.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:42.139 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:41.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:42.139 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:41.819+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:42.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:42.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:42.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:42.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:42.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:42.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:42.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:42.238+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:42.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:42.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:42.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:42.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:42.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:42.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:42.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:42.116+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.13:255 2.13 2:ceffbd45:::200.0000000a:head [write 527852~2382 [fadvise_dontneed] in=2382b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:43.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:42.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:43.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:42.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:43.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:42.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:43.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:42.832+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:43.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:43.088+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:43.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:43.088+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:43.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:43.088+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:43.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:43.089+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 115 slow ops, oldest is osd_op(mds.0.13:255 2.13 2:ceffbd45:::200.0000000a:head [write 527852~2382 [fadvise_dontneed] in=2382b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:43.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:43.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:43.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:43.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:43.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:43.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:43.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:43.259+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:44.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:43.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:44.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:43.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:44.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:43.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:44.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:43.797+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:44.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:44.268+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:44.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:44.268+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:44.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:44.268+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:44.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:44.268+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:44.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:44.118+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:44.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:44.118+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:44.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:44.118+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:44.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:44.119+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 115 slow ops, oldest is osd_op(mds.0.13:255 2.13 2:ceffbd45:::200.0000000a:head [write 527852~2382 [fadvise_dontneed] in=2382b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:45.149 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:44.834+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:45.149 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:44.834+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:45.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:44.834+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:45.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:44.834+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:45.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:45.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:45.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:45.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:45.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:45.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:45.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:45.257+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:45.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:45.127+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:45.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:45.127+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:45.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:45.127+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:45.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:45.127+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 115 slow ops, oldest is osd_op(mds.0.13:255 2.13 2:ceffbd45:::200.0000000a:head [write 527852~2382 [fadvise_dontneed] in=2382b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:46.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:46.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:46.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:46.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:46.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:46.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:46.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:46.118+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 115 slow ops, oldest is osd_op(mds.0.13:255 2.13 2:ceffbd45:::200.0000000a:head [write 527852~2382 [fadvise_dontneed] in=2382b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:46.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:45.877+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:46.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:45.877+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:46.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:45.877+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:46.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:45.878+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:46.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:11:46.312+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 368 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:11:46.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:46.238+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:46.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:46.238+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:46.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:46.238+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:46.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:46.239+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:47.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:47.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:47.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:47.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:47.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:47.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:47.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:47.084+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 115 slow ops, oldest is osd_op(mds.0.13:255 2.13 2:ceffbd45:::200.0000000a:head [write 527852~2382 [fadvise_dontneed] in=2382b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:47.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:46.837+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:47.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:46.837+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:47.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:46.837+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:47.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:46.838+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:47.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:47.268+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:47.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:47.268+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:47.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:47.268+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:47.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:47.269+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:48.282 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:48.061+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:48.282 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:48.061+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:48.282 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:48.061+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:48.282 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:48.063+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:48.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:47.828+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:48.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:47.828+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:48.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:47.828+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:48.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:47.828+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:48.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:48.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:48.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:48.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:48.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:48.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:48.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:48.260+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:49.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:48.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:49.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:48.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:49.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:48.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:49.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:48.846+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:49.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:49.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:49.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:49.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:49.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:49.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:49.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:49.097+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:49.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:49.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:49.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:49.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:49.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:49.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:49.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:49.253+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:50.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:49.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:50.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:49.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:50.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:49.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:50.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:49.831+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:50.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:50.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:50.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:50.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:50.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:50.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:50.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:50.090+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:50.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:50.268+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:50.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:50.268+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:50.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:50.268+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:50.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:50.269+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:51.079 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:50.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:51.079 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:50.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:51.079 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:50.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:51.079 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:50.787+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:51.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:51.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:51.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:51.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:51.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:51.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:51.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:51.227+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:51.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:51.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:51.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:51.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:51.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:51.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:51.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:51.057+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:51.779 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:11:51.312+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 395 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:11:52.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:51.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:52.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:51.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:52.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:51.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:52.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:51.757+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:52.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:52.190+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:52.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:52.190+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:52.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:52.190+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:52.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:52.191+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:52.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:52.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:52.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:52.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:52.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:52.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:52.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:52.095+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:53.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:52.710+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:53.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:52.710+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:53.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:52.710+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:53.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:52.710+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:53.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:53.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:53.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:53.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:53.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:53.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:53.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:53.117+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:53.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:53.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:53.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:53.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:53.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:53.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:53.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:53.207+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:54.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:53.700+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:54.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:53.700+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:54.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:53.700+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:54.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:53.701+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:54.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:54.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:54.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:54.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:54.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:54.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:54.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:54.200+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:54.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:54.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:54.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:54.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:54.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:54.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:54.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:54.142+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:55.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:54.716+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:55.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:54.716+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:55.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:54.716+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:55.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:54.716+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:55.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:55.248+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:55.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:55.248+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:55.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:55.248+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:55.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:55.249+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:55.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:55.127+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:55.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:55.127+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:55.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:55.127+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:55.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:55.128+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:56.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:55.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:56.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:55.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:56.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:55.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:56.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:55.731+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:56.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:56.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:56.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:56.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:56.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:56.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:56.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:56.134+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:56.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:56.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:56.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:56.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:56.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:56.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:56.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:56.227+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:56.771 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:11:56.313+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 425 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:11:57.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:56.748+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:57.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:56.748+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:57.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:56.748+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:57.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:56.748+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:57.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:57.180+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:57.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:57.180+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:57.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:57.180+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:57.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:57.180+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:57.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:57.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:57.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:57.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:57.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:57.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:57.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:57.256+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:58.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:57.780+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:58.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:57.780+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:58.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:57.780+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:58.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:57.780+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:58.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:58.195+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:58.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:58.195+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:58.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:58.195+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:58.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:58.196+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:11:58.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:58.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:58.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:58.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:58.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:58.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:58.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:58.241+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:59.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:58.737+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:59.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:58.737+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:59.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:58.737+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:11:59.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:58.737+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:59.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:59.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:59.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:59.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:59.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:59.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:11:59.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:11:59.277+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:11:59.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:59.190+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:59.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:59.190+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:59.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:59.190+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:11:59.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:11:59.190+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:00.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:59.754+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:00.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:59.754+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:00.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:59.754+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:00.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:11:59.754+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:00.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:00.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:00.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:00.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:00.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:00.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:00.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:00.234+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:00.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:00.247+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:00.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:00.247+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:00.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:00.247+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:00.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:00.247+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:01.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:00.741+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:01.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:00.741+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:01.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:00.741+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:01.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:00.741+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:01.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:12:01.313+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 452 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:12:01.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:01.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:01.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:01.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:01.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:01.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:01.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:01.200+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:01.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:01.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:01.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:01.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:01.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:01.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:01.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:01.204+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:02.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:01.764+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:02.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:01.764+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:02.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:01.764+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:02.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:01.764+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:02.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:02.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:02.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:02.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:02.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:02.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:02.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:02.173+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:02.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:02.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:02.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:02.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:02.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:02.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:02.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:02.205+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:03.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:02.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:03.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:02.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:03.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:02.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:03.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:02.777+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:03.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:03.201+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:03.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:03.201+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:03.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:03.201+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:03.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:03.202+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:03.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:03.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:03.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:03.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:03.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:03.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:03.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:03.173+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:04.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:03.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:04.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:03.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:04.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:03.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:04.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:03.807+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:04.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:04.231+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:04.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:04.231+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:04.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:04.231+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:04.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:04.232+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:04.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:04.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:04.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:04.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:04.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:04.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:04.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:04.134+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:05.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:04.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:05.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:04.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:05.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:04.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:05.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:04.857+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:05.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:05.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:05.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:05.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:05.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:05.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:05.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:05.178+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:05.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:05.201+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:05.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:05.201+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:05.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:05.201+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:05.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:05.202+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:06.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:05.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:06.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:05.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:06.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:05.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:06.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:05.808+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:06.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:12:06.314+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 482 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:12:06.458 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:06.194+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:06.458 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:06.194+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:06.458 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:06.194+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:06.458 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:06.195+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:06.459 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:06.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:06.459 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:06.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:06.459 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:06.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:06.459 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:06.161+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:07.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:06.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:07.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:06.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:07.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:06.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:07.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:06.778+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:07.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:07.147+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:07.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:07.147+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:07.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:07.147+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:07.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:07.148+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:07.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:07.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:07.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:07.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:07.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:07.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:07.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:07.179+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:08.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:07.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:08.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:07.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:08.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:07.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:08.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:07.778+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:08.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:08.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:08.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:08.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:08.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:08.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:08.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:08.103+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:08.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:08.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:08.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:08.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:08.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:08.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:08.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:08.208+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:09.091 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:08.804+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:09.091 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:08.804+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:09.091 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:08.804+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:09.091 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:08.804+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:09.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:09.069+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:09.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:09.069+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:09.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:09.069+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:09.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:09.069+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:09.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:09.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:09.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:09.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:09.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:09.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:09.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:09.211+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:10.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:09.764+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:10.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:09.764+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:10.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:09.764+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:10.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:09.764+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:10.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:10.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:10.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:10.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:10.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:10.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:10.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:10.111+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:10.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:10.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:10.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:10.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:10.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:10.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:10.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:10.194+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:11.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:10.754+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:11.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:10.754+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:11.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:10.754+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:11.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:10.755+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:11.458 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:12:11.314+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 509 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:12:11.458 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:11.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:11.458 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:11.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:11.459 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:11.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:11.459 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:11.203+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:11.459 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:11.118+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:11.459 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:11.118+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:11.459 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:11.118+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:11.459 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:11.118+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:12.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:11.705+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:12.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:11.705+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:12.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:11.705+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:12.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:11.706+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:12.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:12.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:12.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:12.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:12.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:12.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:12.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:12.139+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:12.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:12.234+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:12.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:12.234+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:12.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:12.234+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:12.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:12.235+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:13.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:12.719+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:13.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:12.719+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:13.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:12.719+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:13.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:12.719+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:13.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:13.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:13.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:13.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:13.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:13.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:13.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:13.091+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:13.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:13.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:13.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:13.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:13.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:13.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:13.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:13.208+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:14.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:13.766+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:14.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:13.767+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:14.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:13.767+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:14.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:13.767+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:14.344 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:14.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:14.345 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:14.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:14.345 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:14.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:14.345 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:14.044+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:14.345 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:14.254+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:14.345 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:14.254+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:14.345 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:14.254+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:14.345 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:14.255+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:15.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:14.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:15.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:14.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:15.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:14.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:15.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:14.817+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:15.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:15.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:15.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:15.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:15.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:15.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:15.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:15.003+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:15.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:15.240+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:15.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:15.240+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:15.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:15.240+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:15.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:15.241+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:16.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:16.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:16.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:16.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:16.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:16.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:16.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:16.030+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:16.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:15.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:16.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:15.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:16.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:15.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:16.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:15.847+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:16.791 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:12:16.314+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 539 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:12:16.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:16.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:16.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:16.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:16.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:16.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:16.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:16.271+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:17.247 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:17.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:17.247 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:17.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:17.247 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:17.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:17.247 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:17.062+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:17.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:16.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:17.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:16.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:17.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:16.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:17.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:16.857+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:17.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:17.224+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:17.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:17.224+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:17.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:17.224+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:17.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:17.225+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:18.247 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:18.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:18.248 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:18.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:18.248 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:18.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:18.248 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:18.049+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:18.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:17.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:18.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:17.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:18.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:17.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:18.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:17.872+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:18.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:18.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:18.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:18.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:18.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:18.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:18.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:18.226+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:19.282 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:19.072+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:19.282 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:19.072+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:19.282 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:19.072+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:19.282 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:19.073+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:19.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:18.834+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:19.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:18.834+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:19.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:18.834+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:19.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:18.834+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:19.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:19.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:19.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:19.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:19.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:19.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:19.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:19.260+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:20.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:20.033+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:20.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:20.033+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:20.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:20.033+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:20.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:20.033+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:20.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:19.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:20.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:19.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:20.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:19.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:20.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:19.871+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:20.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:20.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:20.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:20.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:20.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:20.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:20.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:20.276+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:21.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:21.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:21.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:21.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:21.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:21.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:21.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:21.052+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:21.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:20.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:21.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:20.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:21.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:20.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:21.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:20.871+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:21.791 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:12:21.315+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 566 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:12:21.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:21.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:21.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:21.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:21.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:21.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:21.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:21.291+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:22.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:21.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:22.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:21.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:22.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:21.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:22.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:21.860+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:22.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:22.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:22.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:22.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:22.292 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:22.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:22.292 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:22.029+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:22.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:22.332+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:22.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:22.332+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:22.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:22.332+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:22.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:22.333+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:23.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:22.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:23.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:22.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:23.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:22.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:23.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:22.871+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:23.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:23.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:23.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:23.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:23.292 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:23.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:23.292 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:23.025+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:23.561 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:23.286+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:23.561 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:23.286+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:23.561 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:23.286+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:23.561 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:23.286+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:24.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:24.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:24.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:24.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:24.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:24.055+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:24.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:24.055+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:24.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:23.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:24.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:23.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:24.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:23.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:24.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:23.886+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:24.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:24.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:24.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:24.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:24.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:24.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:24.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:24.282+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:25.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:25.093+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:25.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:25.093+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:25.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:25.093+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:25.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:25.094+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:25.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:24.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:25.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:24.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:25.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:24.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:25.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:24.872+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:25.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:25.274+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:25.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:25.274+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:25.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:25.274+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:25.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:25.274+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:26.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:25.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:26.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:25.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:26.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:25.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:26.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:25.851+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:26.462 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:12:26.315+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 596 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:12:26.462 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:26.111+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:26.462 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:26.111+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:26.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:26.111+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:26.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:26.112+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:26.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:26.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:26.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:26.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:26.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:26.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:26.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:26.314+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:27.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:27.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:27.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:27.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:27.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:27.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:27.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:27.066+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:27.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:26.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:27.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:26.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:27.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:26.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:27.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:26.836+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:27.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:27.321+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:27.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:27.321+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:27.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:27.321+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:27.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:27.321+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:28.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:28.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:28.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:28.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:28.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:28.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:28.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:28.093+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:28.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:27.862+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:28.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:27.862+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:28.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:27.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:28.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:27.863+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:28.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:28.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:28.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:28.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:28.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:28.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:28.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:28.312+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:29.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:29.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:29.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:29.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:29.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:29.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:29.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:29.135+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:29.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:28.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:29.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:28.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:29.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:28.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:29.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:28.913+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:29.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:29.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:29.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:29.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:29.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:29.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:29.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:29.293+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:30.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:30.115+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:30.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:30.115+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:30.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:30.115+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:30.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:30.116+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:30.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:29.923+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:30.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:29.923+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:30.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:29.923+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:30.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:29.924+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:30.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:30.322+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:30.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:30.322+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:30.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:30.322+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:30.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:30.322+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:31.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:31.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:31.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:31.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:31.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:31.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:31.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:31.164+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:31.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:30.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:31.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:30.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:31.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:30.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:31.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:30.953+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:31.791 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:12:31.316+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 623 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:12:31.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:31.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:31.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:31.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:31.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:31.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:31.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:31.356+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:32.212 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:31.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:32.212 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:31.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:32.212 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:31.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:32.212 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:31.928+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:32.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:32.342+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:32.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:32.342+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:32.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:32.342+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:32.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:32.343+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:32.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:32.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:32.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:32.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:32.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:32.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:32.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:32.190+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:33.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:32.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:33.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:32.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:33.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:32.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:33.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:32.938+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:33.415 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:33.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:33.415 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:33.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:33.415 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:33.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:33.415 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:33.142+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:33.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:33.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:33.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:33.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:33.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:33.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:33.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:33.393+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:34.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:34.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:34.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:34.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:34.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:34.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:34.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:34.158+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:34.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:33.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:34.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:33.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:34.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:33.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:34.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:33.908+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:34.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:34.419+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:34.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:34.419+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:34.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:34.419+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:34.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:34.420+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:35.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:34.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:35.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:34.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:35.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:34.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:35.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:34.901+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:35.471 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:35.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:35.471 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:35.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:35.471 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:35.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:35.471 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:35.170+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:35.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:35.448+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:35.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:35.448+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:35.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:35.448+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:35.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:35.449+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:36.243 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:35.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:36.243 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:35.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:36.243 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:35.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:36.243 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:35.859+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:36.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:12:36.316+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 653 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:12:36.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:36.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:36.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:36.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:36.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:36.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:36.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:36.221+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:36.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:36.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:36.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:36.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:36.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:36.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:36.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:36.418+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:36.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:36.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:36.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:36.824+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:37.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:37.395+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:37.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:37.395+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:37.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:37.395+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:37.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:37.396+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:37.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:37.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:37.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:37.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:37.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:37.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:37.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:37.249+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:38.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:37.815+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:38.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:37.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:38.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:37.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:38.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:37.816+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:38.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:38.386+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:38.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:38.386+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:38.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:38.386+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:38.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:38.387+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:38.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:38.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:38.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:38.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:38.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:38.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:38.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:38.236+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:39.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:38.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:39.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:38.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:39.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:38.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:39.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:38.831+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:39.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:39.401+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:39.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:39.401+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:39.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:39.401+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:39.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:39.402+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:39.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:39.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:39.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:39.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:39.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:39.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:39.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:39.251+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:40.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:39.875+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:40.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:39.875+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:40.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:39.875+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:40.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:39.875+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:40.616 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:40.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:40.617 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:40.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:40.617 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:40.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:40.617 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:40.374+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:40.617 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:40.276+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:40.617 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:40.276+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:40.617 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:40.276+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:40.617 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:40.276+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:41.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:40.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:41.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:40.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:41.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:40.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:41.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:40.902+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:41.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:12:41.317+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 680 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:12:41.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:41.239+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:41.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:41.239+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:41.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:41.239+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:41.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:41.240+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:41.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:41.406+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:41.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:41.406+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:41.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:41.406+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:41.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:41.407+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:42.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:41.864+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:42.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:41.864+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:42.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:41.864+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:42.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:41.865+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:42.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:42.436+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:42.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:42.436+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:42.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:42.436+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:42.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:42.437+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:42.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:42.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:42.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:42.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:42.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:42.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:42.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:42.284+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:43.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:42.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:43.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:42.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:43.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:42.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:43.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:42.895+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:43.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:43.480+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:43.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:43.480+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:43.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:43.480+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:43.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:43.480+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:43.605 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:43.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:43.605 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:43.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:43.605 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:43.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:43.605 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:43.299+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:44.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:43.876+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:44.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:43.876+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:44.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:43.876+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:44.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:43.876+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:44.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:44.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:44.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:44.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:44.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:44.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:44.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:44.514+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:44.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:44.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:44.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:44.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:44.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:44.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:44.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:44.299+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:45.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:44.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:45.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:44.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:45.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:44.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:45.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:44.854+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:45.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:45.495+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:45.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:45.495+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:45.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:45.495+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:45.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:45.495+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:45.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:45.294+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:45.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:45.295+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:45.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:45.295+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:45.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:45.295+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:46.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:45.862+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:46.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:45.862+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:46.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:45.862+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:46.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:45.862+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:46.791 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:12:46.317+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 710 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:12:46.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:46.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:46.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:46.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:46.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:46.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:46.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:46.496+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:46.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:46.313+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:46.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:46.313+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:46.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:46.313+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:46.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:46.314+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:47.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:46.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:47.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:46.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:47.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:46.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:47.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:46.885+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:47.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:47.476+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:47.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:47.476+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:47.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:47.476+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:47.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:47.477+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:47.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:47.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:47.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:47.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:47.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:47.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:47.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:47.276+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:48.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:47.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:48.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:47.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:48.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:47.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:48.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:47.846+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:48.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:48.239+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:48.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:48.239+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:48.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:48.239+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:48.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:48.240+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:48.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:48.465+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:48.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:48.465+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:48.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:48.465+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:48.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:48.466+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:49.227 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:48.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:49.227 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:48.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:49.227 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:48.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:49.227 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:48.895+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:49.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:49.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:49.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:49.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:49.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:49.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:49.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:49.204+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:49.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:49.438+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:49.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:49.438+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:49.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:49.438+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:49.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:49.439+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:50.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:49.852+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:50.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:49.852+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:50.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:49.852+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:50.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:49.852+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:50.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:50.180+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:50.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:50.180+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:50.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:50.180+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:50.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:50.180+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:50.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:50.426+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:50.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:50.426+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:50.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:50.426+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:50.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:50.427+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:51.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:50.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:51.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:50.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:51.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:50.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:51.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:50.846+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:51.463 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:12:51.318+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 737 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:12:51.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:51.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:51.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:51.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:51.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:51.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:51.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:51.167+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:51.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:51.459+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:51.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:51.459+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:51.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:51.459+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:51.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:51.460+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:52.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:51.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:52.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:51.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:52.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:51.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:52.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:51.832+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:52.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:52.144+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:52.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:52.144+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:52.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:52.144+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:52.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:52.145+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:52.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:52.416+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:52.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:52.416+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:52.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:52.416+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:52.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:52.417+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:53.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:52.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:53.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:52.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:53.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:52.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:53.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:52.867+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:53.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:53.124+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:53.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:53.124+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:53.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:53.124+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:53.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:53.125+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:53.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:53.442+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:53.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:53.442+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:53.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:53.442+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:53.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:53.443+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:54.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:53.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:54.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:53.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:54.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:53.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:54.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:53.900+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:54.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:54.397+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:54.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:54.397+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:54.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:54.397+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:54.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:54.398+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:54.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:54.153+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:54.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:54.153+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:54.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:54.153+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:54.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:54.153+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:55.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:54.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:55.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:54.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:55.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:54.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:55.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:54.868+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:55.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:55.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:55.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:55.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:55.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:55.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:55.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:55.139+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:55.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:55.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:55.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:55.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:55.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:55.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:55.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:55.441+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:56.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:55.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:56.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:55.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:56.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:55.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:56.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:55.883+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:56.341 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:56.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:56.341 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:56.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:56.341 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:56.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:56.342 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:56.165+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:56.711 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:12:56.318+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 767 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:12:56.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:56.399+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:56.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:56.399+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:56.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:56.399+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:56.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:56.400+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:57.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:56.834+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:57.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:56.834+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:57.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:56.834+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:57.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:56.835+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:57.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:57.438+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:57.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:57.438+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:57.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:57.438+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:57.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:57.439+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:57.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:57.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:57.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:57.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:57.543 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:57.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:57.543 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:57.190+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:58.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:57.869+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:58.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:57.869+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:58.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:57.869+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:58.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:57.869+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:58.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:58.237+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:58.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:58.237+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:58.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:58.237+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:58.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:58.238+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:58.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:58.404+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:58.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:58.404+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:58.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:58.404+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:58.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:58.404+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:59.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:58.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:59.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:58.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:59.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:58.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:12:59.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:58.854+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:12:59.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:59.222+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:59.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:59.222+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:59.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:59.222+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:12:59.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:12:59.223+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:12:59.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:59.395+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:59.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:59.395+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:59.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:59.395+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:12:59.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:12:59.396+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:00.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:59.877+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:00.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:59.877+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:00.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:59.877+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:00.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:12:59.877+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:00.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:00.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:00.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:00.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:00.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:00.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:00.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:00.397+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:00.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:00.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:00.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:00.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:00.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:00.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:00.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:00.188+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:01.190 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:00.892+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:01.190 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:00.892+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:01.190 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:00.892+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:01.190 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:00.893+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:01.465 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:13:01.319+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 795 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:13:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:01.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:01.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:01.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:01.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:01.354+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:01.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:01.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:01.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:01.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:01.167+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:02.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:01.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:02.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:01.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:02.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:01.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:02.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:01.868+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:02.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:02.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:02.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:02.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:02.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:02.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:02.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:02.338+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:02.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:02.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:02.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:02.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:02.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:02.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:02.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:02.169+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:03.226 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:02.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:03.226 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:02.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:03.226 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:02.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:03.226 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:02.822+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:03.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:03.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:03.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:03.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:03.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:03.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:03.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:03.298+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:03.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:03.203+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:03.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:03.203+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:03.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:03.203+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:03.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:03.204+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:04.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:03.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:04.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:03.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:04.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:03.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:04.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:03.797+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:04.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:04.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:04.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:04.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:04.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:04.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:04.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:04.249+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:04.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:04.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:04.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:04.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:04.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:04.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:04.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:04.282+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:05.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:04.767+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:05.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:04.767+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:05.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:04.767+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:05.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:04.767+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:05.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:05.248+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:05.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:05.248+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:05.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:05.248+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:05.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:05.250+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:05.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:05.262+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:05.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:05.262+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:05.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:05.262+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:05.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:05.263+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:06.250 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:05.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:06.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:05.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:06.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:05.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:06.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:05.815+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:06.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:13:06.319+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 823 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:13:06.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:06.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:06.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:06.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:06.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:06.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:06.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:06.229+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:06.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:06.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:06.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:06.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:06.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:06.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:06.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:06.234+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:07.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:06.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:06.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:06.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:06.831+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:07.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:07.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:07.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:07.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:07.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:07.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:07.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:07.257+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:07.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:07.247+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:07.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:07.247+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:07.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:07.247+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:07.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:07.247+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:08.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:07.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:08.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:07.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:08.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:07.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:08.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:07.880+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:08.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:08.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:08.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:08.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:08.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:08.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:08.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:08.298+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:08.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:08.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:08.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:08.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:08.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:08.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:08.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:08.286+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:09.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:08.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:09.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:08.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:09.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:08.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:09.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:08.895+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:09.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:09.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:09.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:09.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:09.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:09.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:09.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:09.292+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:09.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:09.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:09.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:09.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:09.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:09.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:09.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:09.309+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:10.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:09.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:10.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:09.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:10.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:09.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:10.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:09.938+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:10.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:10.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:10.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:10.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:10.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:10.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:10.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:10.259+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:10.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:10.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:10.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:10.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:10.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:10.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:10.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:10.336+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:11.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:10.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:11.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:10.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:11.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:10.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:11.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:10.962+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:11.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:13:11.320+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 852 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:13:11.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:11.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:11.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:11.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:11.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:11.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:11.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:11.240+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:11.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:11.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:11.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:11.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:11.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:11.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:11.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:11.315+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:12.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:12.230+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:12.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:12.230+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:12.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:12.230+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:12.292 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:12.231+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:12.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:11.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:12.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:11.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:12.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:11.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:12.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:11.995+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:12.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:12.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:12.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:12.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:12.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:12.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:12.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:12.299+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:13.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:13.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:13.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:13.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:13.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:13.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:13.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:13.216+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:13.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:12.971+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:13.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:12.971+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:13.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:12.971+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:13.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:12.971+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:13.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:13.333+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:13.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:13.333+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:13.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:13.333+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:13.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:13.334+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:14.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:13.940+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:14.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:13.940+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:14.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:13.940+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:14.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:13.941+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:14.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:14.360+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:14.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:14.360+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:14.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:14.360+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:14.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:14.361+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:14.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:14.192+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:14.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:14.192+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:14.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:14.192+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:14.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:14.193+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:15.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:14.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:15.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:14.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:15.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:14.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:15.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:14.926+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:15.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:15.354+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:15.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:15.354+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:15.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:15.354+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:15.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:15.355+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:15.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:15.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:15.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:15.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:15.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:15.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:15.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:15.238+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:16.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:15.967+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:16.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:15.967+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:16.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:15.967+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:16.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:15.967+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:16.791 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:13:16.320+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 884 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:13:16.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:16.376+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:16.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:16.376+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:16.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:16.376+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:16.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:16.377+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:16.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:16.283+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:16.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:16.283+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:16.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:16.283+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:16.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:16.284+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:17.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:16.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:17.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:16.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:17.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:16.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:17.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:16.942+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:17.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:17.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:17.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:17.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:17.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:17.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:17.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:17.393+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:17.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:17.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:17.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:17.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:17.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:17.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:17.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:17.324+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:18.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:17.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:18.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:17.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:18.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:17.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:18.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:17.960+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:18.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:18.395+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:18.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:18.395+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:18.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:18.395+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:18.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:18.396+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:18.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:18.288+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:18.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:18.288+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:18.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:18.288+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:18.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:18.289+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:19.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:18.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:19.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:18.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:19.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:18.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:19.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:18.928+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:19.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:19.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:19.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:19.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:19.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:19.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:19.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:19.380+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:19.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:19.243+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:19.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:19.243+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:19.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:19.243+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:19.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:19.244+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:20.223 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:19.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:20.224 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:19.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:20.224 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:19.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:20.224 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:19.913+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:20.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:20.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:20.543 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:20.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:20.543 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:20.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:20.543 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:20.202+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:20.544 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:20.405+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:20.544 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:20.405+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:20.544 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:20.405+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:20.544 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:20.406+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:20.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:21.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:20.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:21.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:20.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:21.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:20.956+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:21.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:13:21.321+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 913 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:13:21.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:21.409+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:21.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:21.409+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:21.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:21.409+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:21.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:21.409+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:21.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:21.247+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:21.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:21.248+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:21.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:21.248+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:21.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:21.249+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:22.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:21.967+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:22.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:21.967+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:22.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:21.967+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:22.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:21.968+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:22.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:22.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:22.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:22.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:22.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:22.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:22.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:22.238+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:22.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:22.439+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:22.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:22.439+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:22.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:22.439+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:22.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:22.440+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:23.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:23.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:23.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:23.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:23.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:23.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:23.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:23.014+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:23.572 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:23.424+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:23.573 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:23.424+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:23.573 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:23.424+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:23.573 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:23.425+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:23.573 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:23.279+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:23.574 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:23.279+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:23.574 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:23.279+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:23.574 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:23.280+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:24.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:23.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:24.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:23.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:24.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:23.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:24.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:23.977+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:24.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:24.258+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:24.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:24.258+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:24.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:24.258+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:24.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:24.259+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:24.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:24.388+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:24.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:24.388+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:24.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:24.388+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:24.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:24.389+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:25.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:24.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:25.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:24.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:25.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:24.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:25.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:24.937+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:25.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:25.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:25.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:25.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:25.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:25.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:25.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:25.277+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:25.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:25.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:25.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:25.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:25.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:25.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:25.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:25.342+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:26.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:25.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:26.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:25.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:26.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:25.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:26.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:25.930+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:26.791 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:13:26.321+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 941 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:13:26.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:26.286+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:26.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:26.286+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:26.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:26.286+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:26.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:26.287+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:26.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:26.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:26.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:26.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:26.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:26.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:26.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:26.351+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:27.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:26.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:27.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:26.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:27.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:26.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:27.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:26.904+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:27.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:27.283+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:27.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:27.283+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:27.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:27.283+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:27.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:27.284+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:27.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:27.391+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:27.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:27.391+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:27.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:27.391+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:27.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:27.392+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:28.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:27.878+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:28.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:27.878+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:28.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:27.878+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:28.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:27.879+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:28.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:28.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:28.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:28.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:28.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:28.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:28.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:28.346+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:28.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:28.258+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:28.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:28.258+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:28.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:28.258+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:28.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:28.259+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:29.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:28.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:29.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:28.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:29.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:28.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:29.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:28.888+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:29.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:29.377+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:29.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:29.377+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:29.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:29.377+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:29.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:29.377+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:29.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:29.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:29.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:29.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:29.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:29.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:29.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:29.271+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:30.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:29.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:30.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:29.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:30.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:29.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:30.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:29.931+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:30.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:30.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:30.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:30.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:30.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:30.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:30.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:30.282+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:30.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:30.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:30.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:30.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:30.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:30.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:30.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:30.349+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:31.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:30.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:31.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:30.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:31.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:30.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:31.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:30.933+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:31.792 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:13:31.321+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 969 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:13:31.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:31.338+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:31.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:31.338+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:31.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:31.338+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:31.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:31.339+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:31.793 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:31.308+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:31.793 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:31.308+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:31.793 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:31.308+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:31.793 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:31.308+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:32.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:31.970+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:32.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:31.970+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:32.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:31.970+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:32.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:31.971+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:32.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:32.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:32.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:32.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:32.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:32.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:32.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:32.273+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:32.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:32.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:32.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:32.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:32.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:32.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:32.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:32.383+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:33.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:32.951+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:33.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:32.951+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:33.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:32.951+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:33.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:32.951+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:33.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:33.251+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:33.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:33.251+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:33.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:33.251+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:33.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:33.252+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:33.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:33.352+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:33.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:33.352+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:33.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:33.352+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:33.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:33.353+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:34.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:33.927+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:34.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:33.927+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:34.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:33.927+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:34.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:33.928+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:34.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:34.277+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:34.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:34.278+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:34.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:34.278+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:34.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:34.278+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:34.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:34.389+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:34.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:34.389+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:34.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:34.389+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:34.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:34.389+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:35.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:34.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:35.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:34.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:35.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:34.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:35.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:34.955+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:35.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:35.343+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:35.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:35.343+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:35.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:35.343+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:35.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:35.344+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:35.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:35.303+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:35.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:35.303+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:35.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:35.303+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:35.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:35.303+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:36.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:35.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:36.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:35.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:36.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:35.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:36.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:35.949+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:36.791 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:13:36.322+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 998 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:13:36.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:36.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:36.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:36.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:36.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:36.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:36.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:36.348+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:36.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:36.338+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:36.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:36.338+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:36.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:36.338+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:36.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:36.339+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:37.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:36.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:37.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:36.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:37.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:36.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:37.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:36.980+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:37.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:37.368+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:37.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:37.368+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:37.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:37.368+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:37.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:37.369+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:37.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:37.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:37.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:37.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:37.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:37.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:37.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:37.331+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:38.341 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:38.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:38.341 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:38.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:38.341 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:38.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:38.341 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:38.030+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:38.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:38.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:38.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:38.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:38.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:38.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:38.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:38.374+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:38.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:38.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:38.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:38.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:38.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:38.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:38.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:38.319+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:39.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:39.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:39.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:39.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:39.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:39.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:39.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:39.012+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:39.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:39.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:39.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:39.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:39.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:39.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:39.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:39.368+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:39.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:39.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:39.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:39.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:39.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:39.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:39.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:39.283+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:40.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:40.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:40.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:40.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:40.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:40.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:40.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:40.002+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:40.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:40.361+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:40.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:40.361+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:40.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:40.361+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:40.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:40.361+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:40.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:40.280+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:40.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:40.280+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:40.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:40.280+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:40.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:40.280+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:41.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:40.983+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:41.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:40.983+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:41.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:40.983+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:41.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:40.983+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:41.541 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:13:41.323+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 1038 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:13:41.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:41.253+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:41.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:41.253+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:41.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:41.253+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:41.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:41.254+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:41.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:41.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:41.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:41.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:41.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:41.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:41.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:41.378+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:42.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:41.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:42.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:41.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:42.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:41.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:42.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:41.972+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:42.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:42.387+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:42.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:42.387+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:42.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:42.387+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:42.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:42.387+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:42.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:42.266+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:42.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:42.266+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:42.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:42.266+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:42.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:42.267+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:43.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:42.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:43.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:42.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:43.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:42.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:43.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:42.958+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:43.573 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:43.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:43.574 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:43.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:43.574 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:43.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:43.574 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:43.339+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:43.574 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:43.289+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:43.574 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:43.289+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:43.574 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:43.289+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:43.574 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:43.289+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:44.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:43.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:44.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:43.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:44.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:43.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:44.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:43.926+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:44.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:44.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:44.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:44.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:44.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:44.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:44.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:44.269+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:44.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:44.336+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:44.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:44.336+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:44.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:44.336+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:44.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:44.337+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:45.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:44.939+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:45.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:44.939+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:45.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:44.939+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:45.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:44.939+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:45.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:45.303+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:45.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:45.303+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:45.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:45.303+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:45.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:45.304+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:45.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:45.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:45.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:45.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:45.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:45.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:45.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:45.290+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:46.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:45.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:46.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:45.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:46.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:45.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:46.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:45.920+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:46.791 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:13:46.323+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 1083 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:13:46.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:46.303+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:46.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:46.303+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:46.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:46.303+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:46.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:46.304+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:46.793 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:46.308+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:46.793 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:46.308+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:46.793 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:46.308+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:46.793 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:46.309+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:47.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:46.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:47.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:46.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:47.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:46.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:47.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:46.893+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:47.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:47.342+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:47.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:47.342+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:47.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:47.342+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:47.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:47.342+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:47.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:47.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:47.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:47.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:47.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:47.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:47.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:47.316+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:48.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:47.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:48.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:47.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:48.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:47.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:48.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:47.910+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:48.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:48.360+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:48.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:48.360+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:48.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:48.360+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:48.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:48.361+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:48.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:48.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:48.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:48.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:48.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:48.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:48.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:48.294+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:49.288 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:48.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:49.288 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:48.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:49.288 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:48.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:49.288 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:48.872+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:49.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:49.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:49.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:49.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:49.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:49.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:49.542 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:49.265+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:49.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:49.384+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:49.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:49.384+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:49.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:49.384+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:49.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:49.384+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:50.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:49.830+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:50.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:49.830+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:50.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:49.830+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:50.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:49.831+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:50.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:50.397+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:50.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:50.397+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:50.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:50.397+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:50.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:50.398+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:50.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:50.307+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:50.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:50.307+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:50.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:50.307+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:50.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:50.308+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:51.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:50.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:51.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:50.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:51.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:50.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:51.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:50.788+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:51.791 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:13:51.324+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 1128 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:13:51.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:51.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:51.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:51.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:51.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:51.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:51.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:51.332+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:51.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:51.434+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:51.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:51.434+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:51.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:51.434+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:51.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:51.435+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:52.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:51.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:52.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:51.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:52.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:51.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:52.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:51.824+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:52.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:52.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:52.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:52.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:52.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:52.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:52.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:52.338+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:52.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:52.443+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:52.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:52.443+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:52.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:52.443+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:52.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:52.444+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:53.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:52.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:53.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:52.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:53.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:52.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:53.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:52.847+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:53.603 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:53.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:53.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:53.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:53.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:53.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:53.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:53.329+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:53.604 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:53.468+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:53.604 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:53.468+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:53.604 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:53.468+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:53.604 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:53.469+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:54.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:53.876+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:54.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:53.876+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:54.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:53.876+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:54.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:53.877+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:54.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:54.304+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:54.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:54.304+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:54.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:54.304+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:54.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:54.305+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:54.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:54.420+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:54.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:54.420+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:54.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:54.420+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:54.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:54.421+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:55.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:54.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:55.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:54.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:55.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:54.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:55.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:54.866+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:55.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:55.346+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:55.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:55.346+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:55.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:55.346+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:55.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:55.346+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:55.721 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:55.389+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:55.721 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:55.389+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:55.722 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:55.389+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:55.722 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:55.389+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:56.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:55.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:56.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:55.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:56.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:55.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:56.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:55.884+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:56.791 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:13:56.325+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 1174 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:13:56.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:56.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:56.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:56.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:56.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:56.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:56.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:56.432+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:56.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:56.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:56.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:56.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:56.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:56.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:56.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:56.348+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:57.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:56.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:57.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:56.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:57.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:56.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:57.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:56.840+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:57.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:57.391+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:57.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:57.391+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:57.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:57.391+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:57.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:57.392+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:57.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:57.477+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:57.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:57.477+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:57.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:57.477+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:57.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:57.477+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:58.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:57.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:58.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:57.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:58.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:57.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:58.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:57.856+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:58.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:58.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:58.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:58.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:58.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:58.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:58.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:58.373+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:58.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:58.469+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:58.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:58.469+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:58.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:58.469+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:58.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:58.469+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:13:59.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:58.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:59.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:58.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:59.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:58.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:13:59.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:58.851+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:59.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:59.412+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:59.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:59.412+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:59.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:59.412+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:13:59.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:13:59.413+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:13:59.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:59.482+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:59.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:59.482+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:59.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:59.482+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:13:59.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:13:59.483+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:00.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:59.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:00.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:59.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:00.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:59.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:00.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:13:59.810+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:00.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:00.446+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:00.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:00.446+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:00.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:00.446+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:00.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:00.446+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:00.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:00.434+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:00.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:00.434+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:00.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:00.434+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:00.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:00.435+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:01.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:00.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:01.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:00.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:01.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:00.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:01.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:00.809+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:01.791 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:14:01.325+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 1219 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:14:01.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:01.485+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:01.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:01.485+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:01.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:01.485+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:01.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:01.486+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:01.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:01.443+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:01.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:01.443+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:01.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:01.443+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:01.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:01.444+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:02.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:01.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:02.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:01.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:02.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:01.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:02.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:01.824+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:02.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:02.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:02.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:02.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:02.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:02.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:02.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:02.532+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:02.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:02.438+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:02.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:02.438+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:02.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:02.438+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:02.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:02.439+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:03.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:02.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:03.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:02.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:03.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:02.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:03.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:02.831+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:03.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:03.519+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:03.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:03.519+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:03.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:03.519+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:03.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:03.520+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:03.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:03.417+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:03.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:03.417+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:03.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:03.417+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:03.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:03.418+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:04.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:03.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:04.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:03.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:04.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:03.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:04.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:03.880+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:04.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:04.457+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:04.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:04.457+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:04.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:04.457+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:04.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:04.458+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:04.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:04.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:04.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:04.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:04.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:04.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:04.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:04.479+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:05.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:04.853+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:05.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:04.853+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:05.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:04.853+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:05.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:04.853+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:05.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:05.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:05.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:05.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:05.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:05.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:05.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:05.456+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:05.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:05.453+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:05.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:05.453+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:05.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:05.453+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:05.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:05.453+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:06.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:05.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:06.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:05.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:06.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:05.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:06.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:05.849+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:06.686 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:14:06.326+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 1263 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:14:06.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:06.493+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:06.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:06.493+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:06.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:06.493+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:06.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:06.493+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:06.687 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:06.498+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:06.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:06.498+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:06.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:06.498+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:06.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:06.499+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:06.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:14:06.663+0000 7f100ffab700 0 monclient(hunting): authenticate timed out after 300 2024-02-19T19:14:06.688 INFO:teuthology.orchestra.run.smithi114.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-19T19:14:07.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:06.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:07.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:06.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:07.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:06.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:07.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:06.819+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:07.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph orch ps' 2024-02-19T19:14:07.769 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/mon.smithi114/config 2024-02-19T19:14:07.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:07.511+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:07.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:07.511+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:07.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:07.511+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:07.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:07.512+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:07.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:07.532+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:07.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:07.532+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:07.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:07.532+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:07.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:07.533+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:08.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:07.806+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:08.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:07.806+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:08.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:07.806+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:08.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:07.806+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:08.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:08.545+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:08.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:08.545+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:08.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:08.545+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:08.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:08.546+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:08.943 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:08.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:08.944 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:08.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:08.944 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:08.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:08.944 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:08.773+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:08.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:08.525+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:08.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:08.525+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:08.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:08.525+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:08.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:08.526+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:09.792 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:09.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:09.792 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:09.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:09.792 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:09.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:09.792 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:09.757+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:09.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:09.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:09.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:09.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:09.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:09.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:09.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:09.507+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:09.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:09.566+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:09.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:09.566+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:09.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:09.566+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:09.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:09.567+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:10.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:10.495+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:10.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:10.495+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:10.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:10.495+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:10.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:10.496+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:10.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:10.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:10.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:10.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:10.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:10.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:10.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:10.582+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:11.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:10.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:11.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:10.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:11.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:10.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:11.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:10.802+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:11.612 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:14:11.326+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 1351 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:14:11.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:11.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:11.613 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:11.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:11.613 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:11.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:11.613 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:11.491+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:12.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:11.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:12.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:11.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:12.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:11.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:12.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:11.757+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:12.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:11.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:12.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:11.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:12.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:11.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:12.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:11.612+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:12.791 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:12.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:12.792 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:12.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:12.792 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:12.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:12.792 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:12.723+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:12.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:12.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:12.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:12.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:12.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:12.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:12.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:12.479+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:12.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:12.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:12.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:12.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:12.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:12.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:12.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:12.608+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:13.795 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:13.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:13.796 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:13.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:13.796 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:13.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:13.796 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:13.727+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:13.796 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:13.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:13.796 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:13.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:13.797 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:13.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:13.797 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:13.511+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:13.797 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:13.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:13.797 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:13.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:13.797 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:13.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:13.797 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:13.647+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:14.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:14.612+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:14.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:14.612+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:14.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:14.612+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:14.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:14.613+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:14.793 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:14.489+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:14.793 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:14.489+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:14.793 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:14.489+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:14.793 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:14.490+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:15.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:14.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:15.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:14.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:15.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:14.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:15.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:14.764+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:15.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:15.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:15.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:15.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:15.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:15.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:15.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:15.636+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:15.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:15.469+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:15.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:15.469+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:15.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:15.469+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:15.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:15.469+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:16.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:15.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:16.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:15.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:16.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:15.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:16.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:15.736+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:16.665 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:14:16.327+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 1447 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:14:16.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:16.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:16.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:16.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:16.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:16.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:16.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:16.468+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:17.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:16.752+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:17.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:16.752+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:17.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:16.752+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:17.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:16.752+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:17.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:16.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:17.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:16.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:17.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:16.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:17.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:16.686+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:17.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:17.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:17.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:17.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:17.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:17.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:17.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:17.669+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:17.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:17.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:17.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:17.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:17.493+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:18.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:17.745+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:18.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:17.745+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:18.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:17.745+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:18.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:17.746+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:19.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:18.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:19.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:18.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:19.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:18.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:19.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:18.537+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:19.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:18.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:19.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:18.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:19.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:18.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:19.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:18.645+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:19.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:18.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:19.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:18.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:19.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:18.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:19.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:18.732+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:20.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:19.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:20.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:19.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:20.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:19.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:20.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:19.538+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:20.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:19.702+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:20.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:19.702+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:20.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:19.702+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:20.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:19.702+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:20.043 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:19.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:20.043 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:19.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:20.043 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:19.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:20.043 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:19.620+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:21.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:20.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:21.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:20.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:21.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:20.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:21.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:20.570+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:21.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:20.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:21.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:20.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:21.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:20.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:21.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:20.636+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:21.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:20.698+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:21.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:20.699+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:21.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:20.699+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:21.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:20.699+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:21.601 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:14:21.327+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 1543 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:14:21.601 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:21.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:21.601 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:21.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:21.601 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:21.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:21.601 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:21.577+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:22.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:21.669+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:22.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:21.669+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:22.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:21.669+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:22.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:21.669+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:22.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:21.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:22.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:21.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:22.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:21.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:22.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:21.680+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:23.044 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:22.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:23.044 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:22.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:23.044 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:22.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:23.044 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:22.570+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:23.045 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:22.715+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:23.045 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:22.715+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:23.045 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:22.715+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:23.045 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:22.716+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:23.045 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:22.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:23.045 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:22.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:23.045 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:22.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:23.045 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:22.632+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:23.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:23.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:23.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:23.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:23.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:23.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:23.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:23.543+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:24.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:23.734+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:24.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:23.734+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:24.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:23.734+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:24.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:23.734+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:24.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:23.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:24.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:23.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:24.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:23.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:24.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:23.677+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:25.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:24.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:25.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:24.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:25.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:24.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:25.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:24.677+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:25.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:24.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:25.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:24.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:25.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:24.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:25.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:24.587+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:25.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:24.704+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:25.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:24.704+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:25.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:24.704+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:25.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:24.704+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:26.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:25.677+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:26.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:25.677+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:26.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:25.677+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:26.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:25.678+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:26.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:25.673+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:26.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:25.673+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:26.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:25.673+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:26.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:25.674+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:26.043 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:25.561+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:26.043 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:25.561+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:26.043 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:25.561+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:26.043 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:25.562+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:26.686 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:14:26.328+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 1639 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:14:26.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:26.521+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:26.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:26.521+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:26.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:26.521+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:26.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:26.523+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:26.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:26.663+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:26.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:26.663+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:26.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:26.663+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:26.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:26.664+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:26.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:26.669+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:26.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:26.669+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:26.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:26.669+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:26.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:26.669+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:27.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:27.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:27.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:27.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:27.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:27.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:27.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:27.528+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:27.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:27.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:27.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:27.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:27.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:27.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:27.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:27.636+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:27.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:27.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:27.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:27.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:27.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:27.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:27.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:27.682+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:28.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:28.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:28.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:28.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:28.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:28.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:28.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:28.518+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:28.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:28.601+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:28.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:28.601+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:28.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:28.601+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:28.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:28.602+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:28.793 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:28.702+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:28.793 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:28.702+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:28.793 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:28.702+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:28.793 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:28.703+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:29.791 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:29.493+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:29.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:29.493+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:29.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:29.493+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:29.792 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:29.493+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:29.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:29.618+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:29.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:29.618+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:29.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:29.618+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:29.792 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:29.619+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:29.793 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:29.741+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:29.793 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:29.741+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:29.793 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:29.741+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:29.793 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:29.741+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:30.810 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:30.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:30.810 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:30.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:30.810 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:30.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:30.810 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:30.519+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:30.810 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:30.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:30.811 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:30.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:30.811 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:30.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:30.811 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:30.644+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:31.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:30.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:31.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:30.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:31.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:30.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:31.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:30.787+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:31.683 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:14:31.328+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 1734 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:14:31.683 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:31.516+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:31.683 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:31.516+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:31.683 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:31.516+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:31.684 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:31.517+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:32.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:31.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:32.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:31.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:32.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:31.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:32.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:31.661+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:32.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:31.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:32.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:31.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:32.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:31.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:32.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:31.789+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:32.839 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:32.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:32.839 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:32.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:32.839 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:32.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:32.839 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:32.536+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:32.839 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:32.655+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:32.839 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:32.655+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:32.840 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:32.655+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:32.840 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:32.656+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:33.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:32.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:33.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:32.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:33.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:32.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:33.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:32.816+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:33.837 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:33.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:33.838 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:33.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:33.838 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:33.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:33.838 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:33.693+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:33.838 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:33.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:33.838 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:33.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:33.838 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:33.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:33.838 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:33.571+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:34.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:33.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:34.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:33.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:34.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:33.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:34.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:33.815+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:34.883 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:34.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:34.883 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:34.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:34.883 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:34.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:34.883 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:34.690+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:34.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:34.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:34.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:34.551+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:34.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:34.551+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:34.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:34.551+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:35.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:34.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:35.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:34.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:35.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:34.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:35.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:34.860+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:35.904 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:35.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:35.905 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:35.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:35.905 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:35.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:35.905 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:35.588+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:35.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:35.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:35.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:35.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:35.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:35.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:35.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:35.719+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:36.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:35.881+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:36.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:35.881+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:36.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:35.881+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:36.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:35.882+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:36.622 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:14:36.329+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 1831 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:14:36.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:36.599+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:36.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:36.599+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:36.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:36.599+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:36.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:36.600+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:36.893 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:36.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:36.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:36.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:36.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:36.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:36.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:36.680+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:37.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:36.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:37.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:36.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:37.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:36.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:37.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:36.926+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:37.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:37.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:37.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:37.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:37.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:37.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:37.726 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:37.594+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:37.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:37.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:37.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:37.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:37.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:37.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:37.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:37.896+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:37.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:37.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:37.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:37.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:37.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:37.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:37.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:37.703+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:38.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:38.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:38.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:38.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:38.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:38.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:38.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:38.561+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:38.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:38.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:38.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:38.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:38.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:38.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:38.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:38.727+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:39.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:38.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:39.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:38.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:39.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:38.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:39.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:38.894+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:39.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:39.756+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:39.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:39.756+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:39.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:39.756+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:39.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:39.756+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:39.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:39.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:39.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:39.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:39.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:39.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:39.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:39.582+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:40.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:39.877+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:40.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:39.877+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:40.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:39.877+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:40.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:39.877+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:40.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:40.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:40.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:40.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:40.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:40.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:40.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:40.711+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:40.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:40.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:40.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:40.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:40.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:40.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:40.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:40.546+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:41.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:40.892+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:41.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:40.892+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:41.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:40.892+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:41.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:40.892+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:41.616 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:14:41.330+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 1927 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:14:41.896 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:41.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:41.896 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:41.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:41.896 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:41.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:41.896 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:41.594+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:41.896 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:41.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:41.896 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:41.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:41.896 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:41.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:41.897 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:41.760+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:42.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:41.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:42.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:41.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:42.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:41.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:42.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:41.873+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:43.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:42.638+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:43.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:42.638+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:43.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:42.638+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:43.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:42.638+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:43.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:42.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:43.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:42.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:43.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:42.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:43.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:42.787+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:43.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:42.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:43.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:42.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:43.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:42.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:43.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:42.884+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:43.868 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:43.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:43.868 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:43.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:43.869 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:43.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:43.869 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:43.632+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:43.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:43.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:43.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:43.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:43.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:43.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:43.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:43.748+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:44.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:43.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:44.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:43.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:44.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:43.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:44.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:43.846+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:45.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:44.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:45.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:44.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:45.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:44.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:45.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:44.667+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:45.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:44.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:45.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:44.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:45.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:44.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:45.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:44.770+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:45.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:44.842+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:45.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:44.842+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:45.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:44.842+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:45.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:44.842+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:46.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:45.688+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:46.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:45.688+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:46.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:45.688+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:46.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:45.689+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:46.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:45.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:46.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:45.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:46.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:45.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:46.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:45.889+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:46.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:45.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:46.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:45.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:46.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:45.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:46.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:45.734+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:46.731 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:14:46.330+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 2022 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:14:47.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:46.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:47.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:46.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:47.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:46.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:47.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:46.710+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:47.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:46.728+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:47.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:46.728+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:47.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:46.728+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:47.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:46.729+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:47.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:46.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:47.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:46.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:47.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:46.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:47.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:46.882+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:48.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:47.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:48.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:47.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:48.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:47.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:48.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:47.775+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:48.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:47.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:48.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:47.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:48.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:47.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:48.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:47.669+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:48.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:47.917+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:48.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:47.918+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:48.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:47.918+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:48.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:47.918+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:49.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:48.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:49.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:48.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:49.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:48.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:49.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:48.797+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:49.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:48.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:49.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:48.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:49.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:48.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:49.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:48.716+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:49.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:48.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:49.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:48.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:49.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:48.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:49.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:48.908+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:50.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:49.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:50.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:49.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:50.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:49.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:50.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:49.767+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:50.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:49.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:50.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:49.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:50.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:49.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:50.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:49.712+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:50.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:49.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:50.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:49.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:50.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:49.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:50.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:49.901+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:51.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:50.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:51.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:50.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:51.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:50.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:51.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:50.797+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:51.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:50.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:51.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:50.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:51.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:50.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:51.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:50.685+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:51.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:50.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:51.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:50.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:51.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:50.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:51.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:50.901+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:51.736 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:14:51.331+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 2117 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:14:52.040 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:51.869+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:52.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:51.869+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:52.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:51.869+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:52.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:51.869+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:52.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:51.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:52.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:51.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:52.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:51.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:52.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:51.715+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:52.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:51.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:52.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:51.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:52.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:51.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:52.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:51.846+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:53.040 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:52.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:53.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:52.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:53.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:52.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:53.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:52.694+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:53.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:52.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:53.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:52.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:53.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:52.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:53.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:52.833+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:53.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:52.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:53.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:52.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:53.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:52.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:53.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:52.846+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:53.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:53.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:53.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:53.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:53.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:53.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:53.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:53.658+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:54.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:53.876+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:54.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:53.877+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:54.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:53.877+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:54.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:53.877+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:54.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:53.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:54.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:53.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:54.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:53.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:54.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:53.851+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:54.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:54.625+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:54.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:54.625+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:54.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:54.625+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:54.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:54.626+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:55.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:54.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:55.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:54.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:55.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:54.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:55.305 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:54.882+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:55.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:54.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:55.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:54.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:55.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:54.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:55.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:54.915+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:56.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:55.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:56.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:55.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:56.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:55.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:56.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:55.861+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:56.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:55.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:56.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:55.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:56.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:55.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:56.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:55.672+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:56.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:55.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:56.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:55.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:56.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:55.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:56.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:55.891+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:56.717 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:14:56.332+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 2214 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:14:57.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:56.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:57.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:56.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:57.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:56.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:57.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:56.829+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:57.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:56.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:57.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:56.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:57.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:56.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:57.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:56.696+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:57.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:56.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:57.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:56.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:57.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:56.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:57.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:56.897+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:58.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:57.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:58.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:57.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:58.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:57.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:58.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:57.848+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:58.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:57.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:58.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:57.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:58.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:57.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:58.041 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:57.914+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:58.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:57.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:58.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:57.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:58.041 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:57.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:58.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:57.669+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:58.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:58.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:58.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:58.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:58.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:58.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:14:58.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:58.859+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:14:58.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:58.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:58.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:58.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:58.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:58.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:14:58.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:58.685+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:14:59.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:58.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:59.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:58.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:59.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:58.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:14:59.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:58.949+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:00.002 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:59.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:00.002 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:59.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:00.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:59.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:00.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:14:59.655+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:00.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:59.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:00.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:59.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:00.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:59.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:00.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:14:59.837+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:00.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:59.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:00.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:59.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:00.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:59.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:00.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:14:59.981+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:01.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:00.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:01.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:00.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:01.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:00.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:01.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:00.676+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:01.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:00.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:01.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:00.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:01.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:00.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:01.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:00.838+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:01.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:00.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:01.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:00.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:01.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:00.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:01.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:00.981+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:01.652 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:15:01.333+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 2310 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:15:02.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:01.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:02.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:01.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:02.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:01.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:02.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:01.631+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:02.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:01.814+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:02.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:01.814+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:02.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:01.814+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:02.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:01.815+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:02.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:01.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:02.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:01.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:02.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:01.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:02.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:01.988+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:02.883 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:02.602+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:02.884 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:02.602+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:02.884 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:02.602+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:02.884 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:02.603+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:03.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:03.000+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:03.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:03.000+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:03.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:03.000+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:03.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:03.000+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:03.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:02.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:03.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:02.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:03.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:02.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:03.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:02.862+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:03.913 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:03.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:03.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:03.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:03.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:03.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:03.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:03.591+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:04.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:04.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:04.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:04.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:04.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:04.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:04.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:04.036+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:04.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:03.891+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:04.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:03.891+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:04.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:03.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:04.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:03.893+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:04.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:04.621+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:04.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:04.621+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:04.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:04.621+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:04.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:04.622+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:05.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:05.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:05.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:05.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:05.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:05.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:05.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:05.084+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:05.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:04.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:05.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:04.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:05.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:04.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:05.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:04.929+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:05.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:05.602+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:05.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:05.602+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:05.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:05.602+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:05.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:05.603+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:06.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:06.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:06.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:06.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:06.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:06.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:06.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:06.103+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:06.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:05.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:06.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:05.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:06.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:05.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:06.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:05.964+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:06.642 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:15:06.333+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 2405 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:15:07.014 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:06.620+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:07.014 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:06.620+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:07.014 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:06.620+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:07.014 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:06.621+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:07.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:06.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:07.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:06.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:07.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:06.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:07.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:06.993+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:07.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:07.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:07.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:07.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:07.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:07.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:07.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:07.069+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:08.040 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:07.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:08.040 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:07.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:08.040 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:07.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:08.040 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:07.618+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:08.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:08.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:08.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:08.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:08.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:08.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:08.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:08.038+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:08.540 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:08.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:08.541 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:08.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:08.541 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:08.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:08.541 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:08.020+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:09.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:08.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:09.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:08.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:09.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:08.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:09.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:08.588+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:09.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:09.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:09.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:09.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:09.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:09.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:09.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:09.006+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:09.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:09.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:09.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:09.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:09.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:09.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:09.291 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:09.052+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:10.000 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:09.558+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:10.000 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:09.558+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:10.000 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:09.558+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:10.000 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:09.559+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:10.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:10.057+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:10.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:10.057+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:10.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:10.057+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:10.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:10.058+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:10.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:09.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:10.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:09.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:10.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:09.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:10.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:09.979+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:10.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:10.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:10.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:10.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:10.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:10.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:10.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:10.609+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:11.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:10.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:11.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:10.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:11.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:10.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:11.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:10.935+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:11.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:11.031+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:11.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:11.031+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:11.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:11.031+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:11.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:11.032+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:11.790 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:15:11.334+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 2499 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:15:11.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:11.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:11.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:11.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:11.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:11.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:11.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:11.570+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:11.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:11.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:11.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:11.995+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:12.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:11.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:12.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:11.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:12.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:11.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:12.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:11.985+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:12.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:12.521+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:12.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:12.521+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:12.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:12.521+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:12.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:12.522+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:13.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:13.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:13.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:13.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:13.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:13.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:13.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:13.017+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:13.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:12.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:13.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:12.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:13.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:12.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:13.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:12.950+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:13.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:13.513+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:13.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:13.513+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:13.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:13.513+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:13.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:13.514+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:14.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:14.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:14.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:14.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:14.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:14.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:14.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:14.030+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:14.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:13.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:14.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:13.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:14.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:13.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:14.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:13.992+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:14.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:14.473+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:14.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:14.473+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:14.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:14.473+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:14.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:14.474+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:15.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:14.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:15.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:14.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:15.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:14.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:15.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:14.973+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:15.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:15.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:15.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:15.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:15.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:15.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:15.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:15.067+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:15.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:15.431+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:15.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:15.431+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:15.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:15.431+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:15.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:15.431+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:16.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:15.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:16.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:15.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:16.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:15.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:16.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:15.979+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:16.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:16.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:16.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:16.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:16.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:16.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:16.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:16.114+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:16.789 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:15:16.335+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 2594 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:15:16.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:16.392+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:16.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:16.392+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:16.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:16.392+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:16.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:16.393+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:17.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:17.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:17.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:17.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:17.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:17.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:17.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:17.146+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:17.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:17.000+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:17.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:17.000+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:17.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:17.000+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:17.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:17.000+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:17.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:17.405+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:17.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:17.405+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:17.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:17.405+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:17.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:17.406+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:18.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:18.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:18.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:18.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:18.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:18.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:18.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:18.011+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:18.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:18.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:18.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:18.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:18.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:18.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:18.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:18.150+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:18.664 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:18.356+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:18.664 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:18.356+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:18.664 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:18.356+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:18.664 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:18.357+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:19.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:19.118+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:19.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:19.118+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:19.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:19.118+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:19.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:19.119+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:18.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:18.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:18.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:19.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:18.976+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:19.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:19.311+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:19.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:19.311+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:19.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:19.311+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:19.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:19.312+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:20.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:19.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:20.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:19.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:20.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:19.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:20.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:19.984+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:20.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:20.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:20.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:20.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:20.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:20.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:20.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:20.076+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:20.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:20.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:20.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:20.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:20.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:20.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:20.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:20.336+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:21.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:21.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:21.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:21.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:21.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:21.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:21.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:21.082+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:21.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:20.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:21.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:20.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:21.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:20.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:21.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:21.000+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:21.736 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:15:21.335+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 2690 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:15:21.736 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:21.375+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:21.736 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:21.375+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:21.736 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:21.375+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:21.736 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:21.376+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:22.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:22.122+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:22.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:22.122+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:22.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:22.122+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:22.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:22.123+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:22.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:21.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:22.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:21.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:22.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:21.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:22.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:21.981+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:22.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:22.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:22.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:22.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:22.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:22.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:22.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:22.327+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:23.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:23.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:23.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:23.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:23.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:23.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:23.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:23.118+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:23.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:22.956+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:23.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:22.956+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:23.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:22.956+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:23.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:22.956+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:23.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:23.337+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:23.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:23.337+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:23.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:23.337+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:23.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:23.337+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:24.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:24.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:24.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:24.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:24.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:24.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:24.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:24.149+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:24.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:23.967+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:24.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:23.967+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:24.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:23.967+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:24.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:23.967+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:24.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:24.378+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:24.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:24.378+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:24.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:24.378+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:24.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:24.379+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:25.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:24.956+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:25.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:24.956+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:25.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:24.956+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:25.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:24.956+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:25.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:25.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:25.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:25.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:25.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:25.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:25.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:25.199+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:25.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:25.330+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:25.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:25.330+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:25.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:25.330+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:25.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:25.331+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:26.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:25.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:26.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:25.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:26.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:25.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:26.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:25.975+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:26.540 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:15:26.336+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 2785 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:15:26.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:26.245+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:26.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:26.245+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:26.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:26.245+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:26.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:26.246+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:26.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:26.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:26.541 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:26.321+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:26.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:26.321+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:26.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:26.321+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:27.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:27.212+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:27.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:27.212+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:27.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:27.212+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:27.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:27.212+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:27.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:26.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:27.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:26.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:27.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:26.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:27.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:26.985+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:27.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:27.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:27.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:27.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:27.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:27.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:27.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:27.298+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:28.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:27.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:28.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:27.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:28.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:27.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:28.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:27.942+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:28.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:28.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:28.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:28.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:28.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:28.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:28.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:28.320+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:28.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:28.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:28.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:28.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:28.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:28.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:28.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:28.166+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:29.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:28.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:29.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:28.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:29.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:28.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:29.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:28.959+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:29.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:29.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:29.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:29.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:29.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:29.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:29.290 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:29.198+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:29.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:29.315+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:29.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:29.315+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:29.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:29.315+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:29.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:29.316+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:30.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:29.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:30.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:29.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:30.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:29.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:30.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:29.988+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:30.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:30.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:30.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:30.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:30.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:30.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:30.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:30.240+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:30.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:30.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:30.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:30.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:30.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:30.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:30.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:30.293+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:31.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:30.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:31.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:30.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:31.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:30.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:31.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:30.980+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:31.479 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:15:31.337+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 2880 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:15:31.480 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:31.194+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:31.480 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:31.194+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:31.480 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:31.194+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:31.480 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:31.195+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:31.480 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:31.259+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:31.480 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:31.259+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:31.480 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:31.259+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:31.480 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:31.259+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:32.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:32.192+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:32.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:32.192+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:32.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:32.192+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:32.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:32.193+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:32.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:31.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:32.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:31.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:32.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:31.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:32.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:31.967+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:32.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:32.294+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:32.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:32.294+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:32.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:32.294+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:32.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:32.295+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:33.225 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:32.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:33.225 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:32.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:33.225 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:32.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:33.225 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:32.948+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:33.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:33.272+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:33.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:33.272+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:33.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:33.272+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:33.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:33.272+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:33.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:33.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:33.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:33.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:33.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:33.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:33.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:33.205+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:34.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:33.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:34.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:33.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:34.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:33.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:34.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:33.980+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:34.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:34.295+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:34.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:34.295+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:34.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:34.295+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:34.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:34.296+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:34.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:34.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:34.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:34.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:34.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:34.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:34.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:34.237+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:35.230 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:34.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:35.230 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:34.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:35.230 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:34.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:35.230 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:34.943+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:35.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:35.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:35.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:35.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:35.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:35.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:35.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:35.264+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:35.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:35.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:35.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:35.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:35.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:35.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:35.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:35.210+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:36.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:35.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:36.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:35.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:36.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:35.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:36.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:35.897+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:36.479 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:15:36.338+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 2976 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:15:36.479 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:36.236+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:36.479 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:36.236+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:36.479 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:36.236+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:36.479 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:36.237+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:36.480 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:36.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:36.480 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:36.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:36.480 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:36.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:36.480 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:36.203+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:37.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:36.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:37.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:36.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:37.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:36.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:37.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:36.910+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:37.534 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:37.187+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:37.535 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:37.187+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:37.535 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:37.187+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:37.535 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:37.189+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:37.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:37.265+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:37.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:37.265+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:37.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:37.265+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:37.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:37.266+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:38.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:37.945+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:38.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:37.945+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:38.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:37.945+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:38.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:37.945+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:38.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:38.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:38.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:38.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:38.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:38.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:38.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:38.226+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:38.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:38.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:38.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:38.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:38.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:38.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:38.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:38.300+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:39.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:39.242+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:39.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:39.242+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:39.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:39.242+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:39.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:39.243+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:39.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:38.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:39.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:38.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:39.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:38.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:39.289 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:38.995+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:39.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:39.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:39.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:39.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:39.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:39.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:39.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:39.332+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:40.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:40.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:40.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:40.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:40.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:40.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:40.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:40.004+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:40.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:40.263+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:40.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:40.263+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:40.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:40.263+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:40.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:40.264+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:40.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:40.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:40.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:40.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:40.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:40.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:40.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:40.294+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:41.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:41.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:41.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:41.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:41.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:41.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:41.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:41.258+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:41.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:41.256+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:41.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:41.256+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:41.360 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:41.256+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:41.360 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:41.257+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:41.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:41.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:41.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:41.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:41.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:41.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:41.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:41.049+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:41.788 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:15:41.338+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 3073 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:15:42.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:42.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:42.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:42.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:42.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:42.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:42.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:42.088+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:42.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:42.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:42.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:42.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:42.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:42.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:42.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:42.261+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:42.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:42.295+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:42.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:42.295+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:42.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:42.295+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:42.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:42.296+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:43.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:43.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:43.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:43.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:43.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:43.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:43.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:43.134+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:43.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:43.269+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:43.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:43.269+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:43.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:43.269+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:43.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:43.269+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:43.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:43.301+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:43.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:43.301+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:43.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:43.301+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:43.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:43.302+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:44.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:44.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:44.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:44.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:44.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:44.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:44.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:44.169+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:44.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:44.254+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:44.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:44.254+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:44.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:44.254+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:44.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:44.254+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:44.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:44.311+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:44.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:44.311+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:44.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:44.311+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:44.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:44.312+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:45.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:45.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:45.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:45.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:45.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:45.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:45.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:45.174+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:45.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:45.247+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:45.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:45.247+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:45.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:45.247+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:45.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:45.248+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:45.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:45.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:45.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:45.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:45.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:45.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:45.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:45.316+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:46.482 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:15:46.339+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 3168 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:15:46.482 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:46.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:46.483 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:46.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:46.483 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:46.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:46.483 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:46.289+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:46.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:46.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:46.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:46.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:46.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:46.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:46.484 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:46.336+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:46.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:46.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:46.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:46.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:46.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:46.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:46.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:46.130+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:47.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:47.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:47.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:47.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:47.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:47.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:47.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:47.332+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:47.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:47.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:47.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:47.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:47.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:47.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:47.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:47.163+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:47.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:47.327+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:47.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:47.327+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:47.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:47.327+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:47.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:47.328+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:48.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:48.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:48.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:48.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:48.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:48.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:48.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:48.347+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:48.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:48.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:48.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:48.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:48.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:48.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:48.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:48.156+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:48.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:48.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:48.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:48.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:48.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:48.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:48.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:48.323+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:49.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:49.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:49.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:49.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:49.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:49.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:49.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:49.311+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:49.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:49.362+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:49.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:49.362+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:49.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:49.362+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:49.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:49.362+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:49.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:49.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:49.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:49.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:49.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:49.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:49.540 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:49.141+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:50.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:50.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:50.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:50.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:50.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:50.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:50.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:50.356+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:50.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:50.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:50.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:50.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:50.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:50.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:50.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:50.341+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:50.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:50.154+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:50.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:50.154+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:50.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:50.154+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:50.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:50.154+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:51.481 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:15:51.340+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 3262 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:15:51.481 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:51.301+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:51.481 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:51.301+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:51.481 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:51.301+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:51.481 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:51.301+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:51.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:51.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:51.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:51.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:51.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:51.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:51.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:51.192+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:51.482 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:51.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:51.482 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:51.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:51.482 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:51.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:51.482 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:51.351+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:52.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:52.320+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:52.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:52.320+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:52.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:52.320+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:52.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:52.321+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:52.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:52.220+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:52.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:52.220+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:52.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:52.220+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:52.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:52.220+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:52.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:52.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:52.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:52.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:52.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:52.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:52.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:52.272+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:53.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:53.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:53.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:53.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:53.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:53.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:53.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:53.329+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:53.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:53.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:53.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:53.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:53.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:53.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:53.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:53.270+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:53.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:53.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:53.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:53.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:53.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:53.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:53.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:53.194+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:54.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:54.368+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:54.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:54.368+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:54.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:54.368+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:54.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:54.368+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:54.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:54.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:54.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:54.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:54.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:54.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:54.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:54.258+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:54.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:54.177+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:54.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:54.178+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:54.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:54.178+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:54.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:54.178+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:55.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:55.220+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:55.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:55.220+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:55.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:55.220+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:55.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:55.221+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:55.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:55.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:55.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:55.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:55.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:55.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:55.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:55.339+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:55.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:55.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:55.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:55.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:55.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:55.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:55.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:55.305+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:56.538 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:15:56.341+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 3359 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:15:56.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:56.287+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:56.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:56.287+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:56.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:56.287+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:56.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:56.288+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:56.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:56.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:56.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:56.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:56.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:56.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:56.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:56.302+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:56.541 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:56.255+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:56.541 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:56.255+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:56.541 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:56.255+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:56.541 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:56.255+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:57.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:57.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:57.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:57.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:57.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:57.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:57.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:57.257+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:57.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:57.322+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:57.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:57.322+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:57.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:57.322+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:57.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:57.323+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:57.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:57.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:57.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:57.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:57.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:57.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:57.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:57.296+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:58.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:58.218+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:58.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:58.218+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:58.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:58.218+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:58.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:58.219+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:58.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:58.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:58.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:58.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:58.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:58.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:58.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:58.349+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:58.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:58.343+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:58.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:58.343+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:58.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:58.343+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:58.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:58.343+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:59.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:59.188+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:59.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:59.188+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:59.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:59.188+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:15:59.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:15:59.190+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:15:59.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:59.398+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:59.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:59.398+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:59.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:59.398+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:15:59.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:15:59.398+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:15:59.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:59.360+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:59.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:59.360+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:59.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:59.360+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:15:59.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:15:59.361+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:00.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:00.379+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:00.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:00.379+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:00.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:00.379+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:00.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:00.379+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:00.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:00.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:00.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:00.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:00.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:00.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:00.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:00.214+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:00.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:00.345+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:00.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:00.345+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:00.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:00.345+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:00.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:00.346+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:01.483 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:16:01.341+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 3455 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:16:01.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:01.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:01.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:01.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:01.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:01.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:01.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:01.335+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:01.484 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:01.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:01.484 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:01.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:01.484 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:01.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:01.484 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:01.200+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:01.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:01.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:01.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:01.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:01.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:01.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:01.485 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:01.376+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:02.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:02.330+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:02.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:02.330+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:02.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:02.330+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:02.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:02.330+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:02.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:02.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:02.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:02.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:02.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:02.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:02.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:02.210+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:02.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:02.332+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:02.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:02.332+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:02.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:02.332+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:02.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:02.332+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:03.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:03.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:03.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:03.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:03.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:03.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:03.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:03.312+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:03.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:03.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:03.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:03.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:03.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:03.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:03.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:03.289+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:03.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:03.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:03.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:03.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:03.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:03.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:03.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:03.181+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:04.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:04.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:04.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:04.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:04.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:04.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:04.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:04.171+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:04.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:04.308+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:04.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:04.308+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:04.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:04.308+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:04.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:04.308+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:04.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:04.355+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:04.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:04.355+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:04.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:04.355+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:04.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:04.355+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:05.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:05.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:05.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:05.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:05.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:05.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:05.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:05.158+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:05.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:05.279+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:05.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:05.279+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:05.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:05.279+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:05.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:05.280+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:05.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:05.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:05.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:05.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:05.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:05.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:05.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:05.401+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:06.408 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:16:06.342+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 3552 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:16:06.408 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:06.123+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:06.408 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:06.123+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:06.408 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:06.123+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:06.408 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:06.124+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:06.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:06.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:06.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:06.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:06.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:06.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:06.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:06.233+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:06.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:06.388+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:06.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:06.388+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:06.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:06.388+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:06.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:06.389+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:07.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:07.124+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:07.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:07.124+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:07.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:07.124+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:07.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:07.125+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:07.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:07.255+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:07.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:07.255+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:07.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:07.255+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:07.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:07.256+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:07.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:07.352+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:07.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:07.352+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:07.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:07.352+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:07.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:07.352+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:08.537 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:08.163+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:08.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:08.163+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:08.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:08.163+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:08.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:08.164+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:08.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:08.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:08.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:08.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:08.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:08.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:08.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:08.312+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:08.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:08.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:08.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:08.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:08.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:08.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:08.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:08.264+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:09.537 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:09.168+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:09.537 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:09.168+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:09.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:09.168+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:09.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:09.169+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:09.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:09.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:09.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:09.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:09.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:09.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:09.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:09.293+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:09.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:09.307+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:09.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:09.307+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:09.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:09.307+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:09.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:09.307+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:10.537 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:10.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:10.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:10.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:10.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:10.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:10.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:10.342+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:10.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:10.274+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:10.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:10.274+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:10.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:10.274+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:10.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:10.274+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:10.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:10.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:10.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:10.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:10.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:10.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:10.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:10.120+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:11.482 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:16:11.343+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 3647 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:16:11.483 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:11.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:11.483 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:11.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:11.483 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:11.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:11.483 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:11.157+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:11.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:11.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:11.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:11.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:11.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:11.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:11.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:11.332+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:11.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:11.304+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:11.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:11.304+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:11.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:11.304+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:11.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:11.305+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:12.497 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:12.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:12.497 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:12.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:12.497 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:12.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:12.497 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:12.174+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:12.497 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:12.294+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:12.497 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:12.294+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:12.497 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:12.294+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:12.497 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:12.294+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:12.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:12.345+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:12.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:12.345+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:12.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:12.345+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:12.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:12.345+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:13.537 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:13.195+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:13.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:13.195+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:13.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:13.195+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:13.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:13.196+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:13.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:13.383+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:13.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:13.383+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:13.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:13.383+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:13.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:13.384+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:13.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:13.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:13.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:13.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:13.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:13.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:13.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:13.306+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:14.537 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:14.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:14.537 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:14.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:14.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:14.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:14.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:14.192+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:14.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:14.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:14.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:14.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:14.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:14.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:14.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:14.297+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:14.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:14.386+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:14.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:14.386+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:14.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:14.386+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:14.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:14.386+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:15.537 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:15.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:15.537 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:15.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:15.537 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:15.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:15.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:15.181+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:15.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:15.402+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:15.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:15.402+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:15.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:15.402+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:15.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:15.403+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:15.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:15.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:15.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:15.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:15.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:15.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:15.539 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:15.272+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:16.487 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:16:16.344+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 3745 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:16:16.488 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:16.279+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:16.488 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:16.279+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:16.488 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:16.279+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:16.488 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:16.279+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:16.488 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:16.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:16.488 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:16.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:16.488 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:16.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:16.488 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:16.209+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:16.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:16.386+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:16.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:16.386+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:16.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:16.386+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:16.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:16.386+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:17.537 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:17.245+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:17.537 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:17.245+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:17.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:17.245+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:17.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:17.245+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:17.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:17.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:17.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:17.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:17.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:17.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:17.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:17.309+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:17.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:17.405+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:17.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:17.405+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:17.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:17.405+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:17.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:17.405+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:18.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:18.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:18.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:18.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:18.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:18.356+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:18.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:18.356+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:18.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:18.285+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:18.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:18.285+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:18.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:18.285+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:18.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:18.286+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:18.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:18.433+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:18.789 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:18.433+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:18.789 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:18.433+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:18.789 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:18.434+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:19.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:19.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:19.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:19.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:19.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:19.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:19.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:19.342+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:19.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:19.294+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:19.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:19.294+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:19.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:19.294+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:19.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:19.294+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:19.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:19.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:19.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:19.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:19.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:19.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:19.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:19.469+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:20.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:20.322+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:20.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:20.322+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:20.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:20.322+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:20.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:20.323+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:20.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:20.356+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:20.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:20.356+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:20.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:20.356+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:20.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:20.356+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:20.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:20.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:20.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:20.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:20.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:20.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:20.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:20.505+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:21.787 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:16:21.344+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 3840 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:16:21.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:21.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:21.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:21.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:21.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:21.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:21.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:21.339+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:21.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:21.495+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:21.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:21.495+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:21.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:21.495+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:21.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:21.495+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:21.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:21.405+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:21.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:21.405+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:21.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:21.405+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:21.789 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:21.406+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:22.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:22.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:22.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:22.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:22.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:22.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:22.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:22.373+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:22.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:22.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:22.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:22.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:22.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:22.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:22.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:22.415+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:22.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:22.481+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:22.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:22.481+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:22.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:22.481+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:22.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:22.482+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:23.562 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:23.414+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:23.562 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:23.414+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:23.562 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:23.414+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:23.562 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:23.416+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:23.562 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:23.434+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:23.563 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:23.434+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:23.563 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:23.434+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:23.563 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:23.434+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:23.563 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:23.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:23.563 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:23.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:23.563 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:23.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:23.563 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:23.510+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:24.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:24.370+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:24.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:24.370+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:24.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:24.370+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:24.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:24.371+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:24.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:24.452+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:24.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:24.452+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:24.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:24.452+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:24.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:24.452+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:24.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:24.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:24.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:24.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:24.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:24.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:24.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:24.488+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:25.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:25.403+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:25.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:25.403+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:25.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:25.403+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:25.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:25.403+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:25.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:25.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:25.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:25.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:25.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:25.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:25.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:25.462+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:25.789 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:25.491+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:25.789 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:25.491+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:25.789 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:25.491+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:25.789 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:25.492+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:26.512 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:16:26.345+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 3937 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:16:26.512 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:26.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:26.512 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:26.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:26.512 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:26.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:26.513 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:26.393+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:26.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:26.453+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:26.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:26.453+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:26.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:26.453+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:26.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:26.453+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:26.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:26.493+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:26.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:26.493+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:26.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:26.493+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:26.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:26.494+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:27.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:27.450+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:27.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:27.450+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:27.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:27.450+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:27.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:27.451+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:27.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:27.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:27.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:27.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:27.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:27.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:27.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:27.436+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:27.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:27.427+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:27.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:27.427+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:27.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:27.427+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:27.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:27.428+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:28.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:28.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:28.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:28.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:28.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:28.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:28.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:28.416+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:28.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:28.471+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:28.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:28.471+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:28.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:28.471+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:28.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:28.472+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:28.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:28.488+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:28.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:28.488+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:28.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:28.488+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:28.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:28.489+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:29.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:29.437+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:29.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:29.437+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:29.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:29.437+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:29.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:29.438+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:29.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:29.441+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:29.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:29.441+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:29.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:29.442+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:29.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:29.442+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:29.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:29.422+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:29.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:29.422+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:29.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:29.422+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:29.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:29.423+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:30.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:30.424+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:30.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:30.424+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:30.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:30.424+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:30.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:30.424+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:30.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:30.474+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:30.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:30.474+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:30.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:30.474+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:30.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:30.475+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:30.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:30.447+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:30.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:30.447+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:30.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:30.447+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:30.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:30.447+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:31.787 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:16:31.346+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 4032 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:16:31.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:31.508+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:31.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:31.508+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:31.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:31.508+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:31.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:31.508+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:31.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:31.379+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:31.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:31.379+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:31.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:31.379+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:31.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:31.379+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:31.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:31.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:31.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:31.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:31.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:31.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:31.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:31.484+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:32.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:32.348+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:32.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:32.348+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:32.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:32.348+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:32.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:32.349+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:32.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:32.463+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:32.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:32.463+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:32.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:32.463+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:32.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:32.464+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:32.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:32.480+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:32.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:32.480+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:32.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:32.480+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:32.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:32.480+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:33.444 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:33.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:33.444 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:33.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:33.444 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:33.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:33.444 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:33.325+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:33.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:33.426+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:33.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:33.426+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:33.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:33.426+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:33.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:33.426+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:33.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:33.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:33.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:33.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:33.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:33.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:33.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:33.509+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:34.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:34.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:34.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:34.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:34.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:34.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:34.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:34.482+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:34.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:34.369+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:34.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:34.369+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:34.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:34.369+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:34.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:34.370+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:34.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:34.409+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:34.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:34.409+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:34.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:34.409+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:34.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:34.409+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:35.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:35.459+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:35.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:35.459+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:35.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:35.459+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:35.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:35.459+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:35.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:35.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:35.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:35.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:35.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:35.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:35.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:35.520+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:35.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:35.400+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:35.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:35.400+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:35.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:35.400+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:35.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:35.400+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:36.787 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:16:36.347+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 4130 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:16:36.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:36.457+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:36.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:36.457+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:36.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:36.457+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:36.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:36.458+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:36.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:36.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:36.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:36.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:36.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:36.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:36.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:36.510+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:36.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:36.422+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:36.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:36.422+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:36.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:36.422+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:36.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:36.423+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:37.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:37.445+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:37.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:37.445+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:37.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:37.445+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:37.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:37.445+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:37.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:37.435+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:37.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:37.435+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:37.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:37.435+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:37.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:37.435+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:37.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:37.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:37.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:37.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:37.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:37.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:37.788 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:37.513+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:38.786 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:38.497+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:38.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:38.497+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:38.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:38.497+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:38.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:38.497+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:38.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:38.438+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:38.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:38.438+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:38.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:38.438+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:38.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:38.439+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:38.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:38.450+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:38.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:38.450+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:38.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:38.450+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:38.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:38.450+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:39.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:39.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:39.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:39.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:39.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:39.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:39.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:39.393+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:39.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:39.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:39.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:39.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:39.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:39.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:39.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:39.480+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:39.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:39.437+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:39.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:39.437+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:39.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:39.437+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:39.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:39.438+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:40.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:40.406+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:40.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:40.406+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:40.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:40.406+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:40.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:40.407+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:40.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:40.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:40.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:40.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:40.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:40.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:40.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:40.432+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:40.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:40.402+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:40.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:40.402+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:40.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:40.402+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:40.788 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:40.403+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:41.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:16:41.348+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 4226 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:16:41.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:41.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:41.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:41.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:41.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:41.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:41.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:41.426+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:41.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:41.390+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:41.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:41.390+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:41.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:41.390+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:41.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:41.390+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:41.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:41.418+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:41.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:41.418+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:41.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:41.418+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:41.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:41.419+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:42.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:42.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:42.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:42.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:42.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:42.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:42.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:42.441+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:42.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:42.387+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:42.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:42.387+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:42.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:42.387+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:42.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:42.388+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:42.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:42.456+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:42.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:42.456+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:42.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:42.456+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:42.788 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:42.457+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:43.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:43.457+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:43.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:43.457+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:43.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:43.457+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:43.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:43.458+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:43.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:43.373+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:43.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:43.373+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:43.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:43.373+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:43.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:43.374+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:43.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:43.503+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:43.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:43.503+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:43.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:43.503+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:43.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:43.504+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:44.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:44.489+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:44.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:44.489+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:44.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:44.489+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:44.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:44.490+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:44.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:44.474+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:44.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:44.474+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:44.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:44.474+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:44.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:44.475+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:44.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:44.390+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:44.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:44.390+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:44.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:44.390+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:44.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:44.390+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:45.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:45.461+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:45.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:45.461+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:45.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:45.461+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:45.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:45.462+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:45.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:45.359+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:45.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:45.359+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:45.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:45.359+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:45.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:45.359+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:45.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:45.490+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:45.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:45.490+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:45.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:45.490+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:45.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:45.491+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:46.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:16:46.349+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 4324 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:16:46.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:46.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:46.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:46.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:46.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:46.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:46.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:46.488+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:46.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:46.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:46.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:46.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:46.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:46.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:46.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:46.459+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:46.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:46.365+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:46.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:46.365+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:46.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:46.365+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:46.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:46.365+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:47.749 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:47.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:47.749 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:47.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:47.749 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:47.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:47.749 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:47.501+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:47.749 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:47.375+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:47.749 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:47.375+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:47.749 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:47.375+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:47.749 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:47.375+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:47.750 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:47.513+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:47.750 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:47.513+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:47.750 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:47.513+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:47.750 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:47.514+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:48.536 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:48.511+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:48.536 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:48.511+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:48.536 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:48.511+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:48.537 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:48.511+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:48.537 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:48.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:48.537 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:48.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:48.537 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:48.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:48.537 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:48.420+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:49.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:48.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:49.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:48.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:49.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:48.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:49.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:48.542+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:49.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:49.473+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:49.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:49.473+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:49.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:49.473+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:49.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:49.474+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:49.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:49.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:49.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:49.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:49.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:49.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:49.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:49.441+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:49.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:49.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:49.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:49.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:49.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:49.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:49.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:49.518+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:50.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:50.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:50.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:50.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:50.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:50.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:50.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:50.525+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:50.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:50.497+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:50.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:50.497+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:50.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:50.497+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:50.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:50.497+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:50.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:50.463+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:50.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:50.463+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:50.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:50.463+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:50.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:50.463+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:51.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:16:51.349+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 4420 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:16:51.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:51.495+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:51.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:51.495+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:51.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:51.495+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:51.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:51.496+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:51.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:51.446+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:51.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:51.446+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:51.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:51.446+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:51.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:51.447+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:51.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:51.514+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:51.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:51.514+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:51.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:51.514+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:51.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:51.515+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:52.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:52.517+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:52.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:52.517+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:52.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:52.517+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:52.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:52.517+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:52.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:52.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:52.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:52.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:52.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:52.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:52.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:52.459+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:52.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:52.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:52.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:52.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:52.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:52.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:52.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:52.510+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:53.597 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:53.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:53.597 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:53.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:53.598 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:53.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:53.598 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:53.460+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:53.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:53.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:53.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:53.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:53.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:53.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:53.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:53.507+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:53.598 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:53.494+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:53.598 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:53.494+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:53.599 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:53.494+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:53.599 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:53.494+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:54.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:54.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:54.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:54.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:54.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:54.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:54.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:54.458+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:54.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:54.497+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:54.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:54.497+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:54.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:54.497+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:54.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:54.498+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:54.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:54.483+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:54.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:54.483+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:54.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:54.483+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:54.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:54.484+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:55.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:55.430+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:55.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:55.430+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:55.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:55.430+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:55.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:55.431+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:55.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:55.464+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:55.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:55.464+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:55.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:55.464+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:55.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:55.465+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:55.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:55.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:55.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:55.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:55.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:55.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:55.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:55.485+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:56.687 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:16:56.350+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 4523 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:16:56.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:56.385+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:56.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:56.385+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:56.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:56.385+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:56.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:56.386+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:56.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:56.493+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:56.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:56.493+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:56.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:56.493+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:56.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:56.494+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:56.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:56.515+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:56.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:56.515+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:56.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:56.515+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:56.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:56.516+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:57.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:57.527+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:57.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:57.527+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:57.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:57.527+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:57.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:57.528+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:57.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:57.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:57.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:57.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:57.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:57.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:57.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:57.345+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:57.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:57.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:57.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:57.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:57.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:57.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:57.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:57.536+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:58.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:58.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:58.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:58.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:58.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:58.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:58.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:58.332+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:58.786 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:58.533+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:58.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:58.533+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:58.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:58.533+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:58.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:58.534+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:58.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:58.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:58.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:58.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:58.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:58.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:58.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:58.554+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:59.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:59.592+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:59.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:59.592+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:59.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:59.592+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:16:59.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:16:59.593+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:16:59.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:59.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:59.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:59.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:59.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:59.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:16:59.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:16:59.360+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:16:59.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:59.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:59.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:59.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:59.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:59.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:16:59.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:16:59.485+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:00.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:00.549+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:00.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:00.549+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:00.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:00.549+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:00.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:00.550+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:00.786 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:00.517+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:00.786 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:00.517+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:00.786 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:00.517+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:00.786 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:00.518+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:00.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:00.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:00.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:00.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:00.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:00.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:00.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:00.338+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:01.786 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:17:01.351+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 4638 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:17:01.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:01.547+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:01.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:01.547+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:01.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:01.547+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:01.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:01.548+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:01.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:01.365+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:01.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:01.365+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:01.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:01.365+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:01.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:01.366+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:01.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:01.521+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:01.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:01.521+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:01.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:01.521+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:01.787 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:01.522+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:02.785 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:02.554+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:02.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:02.554+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:02.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:02.554+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:02.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:02.555+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:02.786 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:02.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:02.786 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:02.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:02.786 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:02.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:02.786 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:02.562+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:02.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:02.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:02.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:02.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:02.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:02.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:02.787 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:02.379+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:03.613 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:03.520+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:03.613 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:03.520+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:03.613 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:03.520+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:03.613 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:03.521+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:03.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:03.332+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:03.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:03.332+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:03.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:03.332+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:03.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:03.333+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:04.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:03.596+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:04.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:03.596+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:04.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:03.596+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:04.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:03.596+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:04.630 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:04.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:04.630 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:04.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:04.630 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:04.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:04.630 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:04.332+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:04.630 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:04.488+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:04.630 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:04.488+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:04.631 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:04.488+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:04.631 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:04.489+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:05.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:04.611+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:05.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:04.611+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:05.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:04.611+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:05.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:04.612+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:05.617 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:05.470+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:05.618 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:05.470+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:05.618 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:05.470+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:05.618 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:05.471+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:05.618 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:05.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:05.618 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:05.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:05.618 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:05.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:05.618 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:05.291+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:06.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:05.600+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:06.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:05.600+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:06.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:05.600+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:06.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:05.600+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:06.477 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:17:06.352+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 4735 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:17:06.477 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:06.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:06.478 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:06.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:06.478 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:06.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:06.478 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:06.272+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:06.786 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:06.609+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:06.786 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:06.609+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:06.786 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:06.609+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:06.786 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:06.609+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:06.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:06.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:06.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:06.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:06.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:06.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:06.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:06.461+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:07.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:07.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:07.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:07.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:07.647 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:07.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:07.647 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:07.310+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:07.647 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:07.426+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:07.647 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:07.426+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:07.647 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:07.426+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:07.647 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:07.427+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:08.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:07.628+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:08.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:07.628+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:08.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:07.628+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:08.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:07.629+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:08.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:08.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:08.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:08.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:08.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:08.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:08.690 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:08.310+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:08.690 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:08.409+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:08.690 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:08.409+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:08.690 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:08.409+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:08.690 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:08.410+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:09.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:08.672+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:09.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:08.672+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:09.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:08.672+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:09.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:08.672+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:09.731 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:09.357+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:09.731 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:09.357+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:09.731 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:09.357+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:09.731 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:09.358+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:09.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:09.386+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:09.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:09.386+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:09.732 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:09.386+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:09.732 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:09.387+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:10.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:09.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:10.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:09.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:10.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:09.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:10.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:09.714+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:10.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:10.384+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:10.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:10.384+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:10.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:10.384+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:10.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:10.385+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:10.739 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:10.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:10.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:10.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:10.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:10.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:10.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:10.347+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:11.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:10.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:11.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:10.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:11.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:10.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:11.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:10.722+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:11.719 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:17:11.353+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 4830 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:17:11.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:11.432+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:11.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:11.432+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:11.720 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:11.432+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:11.720 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:11.433+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:11.720 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:11.308+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:11.720 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:11.308+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:11.720 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:11.308+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:11.720 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:11.309+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:12.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:11.702+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:12.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:11.702+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:12.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:11.702+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:12.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:11.702+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:12.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:12.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:12.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:12.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:12.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:12.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:12.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:12.452+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:12.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:12.329+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:12.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:12.329+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:12.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:12.329+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:12.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:12.329+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:13.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:12.656+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:13.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:12.656+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:13.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:12.656+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:13.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:12.656+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:13.720 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:13.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:13.720 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:13.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:13.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:13.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:13.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:13.492+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:13.721 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:13.363+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:13.721 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:13.363+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:13.721 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:13.363+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:13.721 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:13.363+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:14.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:13.703+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:14.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:13.703+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:14.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:13.703+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:14.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:13.704+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:14.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:14.324+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:14.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:14.324+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:14.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:14.324+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:14.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:14.324+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:14.502+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:14.502+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:14.502+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:14.504+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:15.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:14.698+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:15.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:14.698+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:15.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:14.698+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:15.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:14.698+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:15.568 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:15.284+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:15.568 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:15.284+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:15.568 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:15.284+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:15.568 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:15.284+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:16.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:15.551+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:16.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:15.551+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:16.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:15.551+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:16.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:15.552+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:16.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:15.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:16.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:15.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:16.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:15.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:16.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:15.736+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:16.568 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:17:16.353+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 4927 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:17:16.568 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:16.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:16.568 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:16.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:16.568 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:16.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:16.569 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:16.293+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:17.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:16.695+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:17.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:16.695+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:17.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:16.695+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:17.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:16.696+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:17.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:16.551+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:17.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:16.551+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:17.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:16.551+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:17.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:16.552+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:17.603 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:17.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:17.604 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:17.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:17.604 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:17.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:17.604 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:17.271+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:18.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:17.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:18.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:17.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:18.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:17.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:18.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:17.587+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:18.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:17.689+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:18.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:17.689+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:18.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:17.689+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:18.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:17.689+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:18.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:18.261+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:18.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:18.261+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:18.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:18.261+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:18.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:18.261+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:19.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:18.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:19.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:18.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:19.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:18.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:19.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:18.589+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:19.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:18.727+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:19.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:18.727+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:19.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:18.727+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:19.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:18.727+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:19.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:19.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:19.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:19.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:19.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:19.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:19.654 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:19.289+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:20.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:19.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:20.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:19.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:20.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:19.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:20.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:19.637+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:20.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:19.687+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:20.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:19.687+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:20.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:19.687+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:20.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:19.687+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:20.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:20.254+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:20.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:20.254+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:20.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:20.254+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:20.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:20.254+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:21.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:20.660+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:21.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:20.660+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:21.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:20.660+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:21.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:20.660+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:21.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:20.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:21.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:20.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:21.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:20.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:21.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:20.697+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:21.535 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:17:21.354+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 5026 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:17:21.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:21.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:21.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:21.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:21.536 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:21.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:21.536 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:21.235+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:22.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:21.675+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:22.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:21.675+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:22.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:21.675+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:22.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:21.676+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:22.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:21.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:22.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:21.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:22.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:21.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:22.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:21.722+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:22.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:22.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:22.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:22.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:22.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:22.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:22.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:22.220+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:23.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:22.681+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:23.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:22.681+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:23.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:22.681+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:23.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:22.682+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:23.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:22.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:23.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:22.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:23.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:22.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:23.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:22.735+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:23.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:23.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:23.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:23.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:23.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:23.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:23.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:23.188+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:24.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:23.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:24.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:23.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:24.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:23.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:24.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:23.637+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:24.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:23.742+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:24.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:23.742+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:24.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:23.742+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:24.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:23.742+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:24.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:24.184+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:24.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:24.184+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:24.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:24.184+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:24.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:24.185+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:25.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:24.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:25.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:24.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:25.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:24.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:25.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:24.604+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:25.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:24.768+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:25.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:24.768+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:25.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:24.769+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:25.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:24.769+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:25.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:25.143+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:25.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:25.143+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:25.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:25.143+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:25.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:25.144+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:26.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:25.793+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:26.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:25.793+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:26.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:25.793+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:26.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:25.793+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:26.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:25.588+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:26.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:25.588+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:26.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:25.588+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:26.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:25.589+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:26.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:26.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:26.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:26.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:26.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:26.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:26.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:26.095+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:26.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:17:26.355+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 5122 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:17:26.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:26.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:26.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:26.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:26.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:26.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:26.786 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:26.576+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:27.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:26.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:27.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:26.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:27.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:26.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:27.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:26.833+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:27.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:27.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:27.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:27.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:27.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:27.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:27.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:27.096+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:28.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:27.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:28.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:27.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:28.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:27.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:28.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:27.582+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:28.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:27.828+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:28.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:27.828+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:28.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:27.828+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:28.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:27.829+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:28.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:28.122+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:28.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:28.122+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:28.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:28.122+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:28.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:28.123+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:28.857 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:28.540+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:28.857 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:28.540+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:28.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:28.540+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:28.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:28.541+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:29.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:28.840+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:29.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:28.840+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:29.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:28.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:29.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:28.841+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:29.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:29.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:29.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:29.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:29.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:29.085+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:29.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:29.085+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:29.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:29.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:29.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:29.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:29.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:29.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:29.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:29.547+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:30.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:30.080+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:30.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:30.080+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:30.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:30.080+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:30.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:30.081+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:30.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:29.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:30.286 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:29.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:30.286 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:29.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:30.286 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:29.886+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:30.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:30.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:30.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:30.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:30.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:30.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:30.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:30.507+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:31.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:30.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:31.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:30.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:31.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:30.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:31.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:30.932+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:31.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:31.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:31.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:31.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:31.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:31.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:31.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:31.035+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:31.785 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:17:31.356+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 5224 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:17:31.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:31.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:31.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:31.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:31.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:31.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:31.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:31.537+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:32.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:31.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:32.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:31.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:32.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:31.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:32.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:31.936+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:32.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:32.027+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:32.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:32.027+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:32.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:32.027+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:32.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:32.027+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:32.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:32.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:32.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:32.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:32.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:32.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:32.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:32.528+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:33.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:32.969+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:33.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:32.969+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:33.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:32.969+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:33.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:32.970+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:33.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:33.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:33.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:33.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:33.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:33.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:33.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:33.009+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:33.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:33.490+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:33.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:33.490+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:33.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:33.490+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:33.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:33.491+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:34.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:34.005+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:34.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:34.005+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:34.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:34.005+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:34.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:34.006+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:34.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:34.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:34.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:34.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:34.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:34.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:34.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:34.045+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:34.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:34.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:34.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:34.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:34.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:34.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:34.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:34.511+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:35.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:35.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:35.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:35.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:35.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:35.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:35.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:35.032+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:35.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:35.011+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:35.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:35.011+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:35.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:35.011+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:35.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:35.011+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:35.982 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:35.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:35.982 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:35.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:35.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:35.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:35.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:35.538+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:36.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:35.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:36.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:35.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:36.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:35.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:36.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:35.999+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:36.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:35.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:36.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:35.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:36.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:35.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:36.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:35.966+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:36.784 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:17:36.357+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 5340 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:17:36.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:36.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:36.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:36.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:36.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:36.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:36.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:36.506+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:37.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:37.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:37.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:37.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:37.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:37.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:37.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:37.038+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:37.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:36.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:37.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:36.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:37.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:36.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:37.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:36.968+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:37.997 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:37.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:37.997 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:37.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:37.997 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:37.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:37.997 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:37.532+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:38.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:37.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:38.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:37.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:38.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:37.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:38.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:37.981+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:38.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:38.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:38.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:38.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:38.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:38.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:38.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:38.074+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:38.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:38.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:38.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:38.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:38.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:38.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:38.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:38.527+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:39.100 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:39.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:39.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:39.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:39.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:39.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:39.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:39.022+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:39.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:39.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:39.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:39.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:39.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:39.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:39.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:39.085+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:40.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:39.532+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:40.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:39.532+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:40.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:39.532+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:40.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:39.532+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:40.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:40.132+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:40.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:40.132+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:40.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:40.132+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:40.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:40.132+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:40.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:40.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:40.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:40.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:40.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:40.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:40.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:40.011+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:40.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:40.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:40.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:40.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:40.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:40.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:40.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:40.506+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:41.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:41.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:41.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:41.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:41.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:41.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:41.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:41.149+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:41.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:40.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:41.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:40.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:41.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:40.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:41.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:40.965+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:41.784 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:17:41.358+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 5457 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:17:41.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:41.549+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:41.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:41.549+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:41.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:41.549+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:41.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:41.550+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:42.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:42.167+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:42.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:42.167+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:42.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:42.167+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:42.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:42.168+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:42.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:41.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:42.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:41.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:42.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:41.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:42.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:41.934+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:42.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:42.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:42.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:42.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:42.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:42.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:42.990 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:42.545+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:43.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:43.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:43.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:43.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:43.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:43.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:43.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:43.140+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:43.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:42.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:43.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:42.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:43.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:42.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:43.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:42.974+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:43.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:43.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:43.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:43.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:43.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:43.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:43.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:43.499+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:44.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:43.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:44.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:43.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:44.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:43.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:44.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:43.963+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:44.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:44.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:44.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:44.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:44.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:44.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:44.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:44.171+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:45.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:44.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:45.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:44.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:45.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:44.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:45.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:44.547+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:45.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:45.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:45.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:45.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:45.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:45.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:45.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:45.010+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:45.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:45.205+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:45.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:45.205+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:45.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:45.205+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:45.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:45.205+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:45.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:45.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:45.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:45.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:45.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:45.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:45.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:45.518+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:46.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:45.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:46.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:45.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:46.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:45.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:46.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:45.980+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:46.524 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:17:46.359+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 5552 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:17:46.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:46.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:46.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:46.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:46.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:46.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:46.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:46.248+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:46.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:46.508+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:46.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:46.508+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:46.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:46.509+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:46.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:46.509+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:47.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:46.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:47.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:46.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:47.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:46.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:47.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:46.963+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:47.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:47.215+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:47.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:47.215+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:47.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:47.215+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:47.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:47.216+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:47.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:47.521+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:47.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:47.521+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:47.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:47.521+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:47.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:47.522+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:48.239 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:47.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:48.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:47.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:48.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:47.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:48.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:47.950+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:48.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:48.223+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:48.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:48.223+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:48.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:48.223+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:48.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:48.224+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:48.938 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:48.525+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:48.938 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:48.525+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:48.938 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:48.525+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:48.938 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:48.526+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:49.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:48.970+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:49.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:48.970+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:49.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:48.970+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:49.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:48.971+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:49.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:49.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:49.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:49.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:49.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:49.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:49.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:49.270+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:50.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:49.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:50.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:49.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:50.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:49.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:50.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:49.551+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:50.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:50.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:50.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:50.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:50.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:50.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:50.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:50.251+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:50.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:50.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:50.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:50.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:50.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:50.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:50.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:50.016+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:50.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:50.516+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:50.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:50.516+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:50.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:50.516+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:50.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:50.516+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:51.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:50.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:51.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:50.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:51.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:50.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:51.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:50.980+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:51.534 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:17:51.360+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 5645 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:17:51.534 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:51.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:51.535 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:51.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:51.535 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:51.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:51.535 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:51.498+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:51.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:51.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:51.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:51.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:51.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:51.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:51.535 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:51.249+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:52.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:52.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:52.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:52.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:52.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:52.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:52.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:52.009+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:52.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:52.261+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:52.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:52.261+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:52.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:52.261+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:52.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:52.262+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:52.982 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:52.541+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:52.982 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:52.541+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:52.982 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:52.541+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:52.982 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:52.542+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:53.243 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:52.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:53.244 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:52.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:53.244 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:52.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:53.244 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:52.966+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:53.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:53.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:53.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:53.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:53.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:53.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:53.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:53.228+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:53.963 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:53.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:53.963 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:53.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:53.963 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:53.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:53.963 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:53.559+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:54.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:53.947+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:54.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:53.947+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:54.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:53.947+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:54.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:53.948+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:54.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:54.242+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:54.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:54.242+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:54.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:54.242+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:54.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:54.243+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:54.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:54.561+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:54.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:54.561+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:54.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:54.561+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:54.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:54.562+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:55.279 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:54.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:55.279 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:54.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:55.279 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:54.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:55.279 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:54.947+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:55.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:55.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:55.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:55.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:55.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:55.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:55.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:55.264+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:55.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:55.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:55.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:55.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:55.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:55.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:55.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:55.538+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:56.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:55.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:56.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:55.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:56.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:55.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:56.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:55.976+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:56.498 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:17:56.361+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 5741 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:17:56.498 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:56.226+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:56.498 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:56.226+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:56.498 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:56.226+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:56.498 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:56.227+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:56.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:56.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:56.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:56.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:56.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:56.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:56.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:56.576+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:57.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:56.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:57.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:56.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:57.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:56.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:57.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:56.964+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:57.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:57.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:57.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:57.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:57.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:57.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:57.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:57.221+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:57.992 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:57.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:57.992 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:57.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:57.992 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:57.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:57.992 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:57.595+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:58.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:58.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:58.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:58.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:58.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:58.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:58.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:58.217+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:58.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:57.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:58.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:57.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:58.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:57.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:58.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:57.976+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:58.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:58.600+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:58.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:58.600+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:58.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:58.600+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:58.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:58.601+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:59.225 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:58.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:59.226 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:58.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:59.226 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:58.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:17:59.226 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:58.954+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:17:59.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:59.210+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:59.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:59.210+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:59.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:59.210+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:17:59.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:17:59.210+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:17:59.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:59.614+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:59.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:59.614+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:59.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:59.614+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:17:59.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:17:59.615+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:00.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:59.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:00.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:59.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:00.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:59.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:00.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:17:59.978+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:00.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:00.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:00.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:00.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:00.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:00.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:00.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:00.190+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:00.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:00.605+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:00.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:00.605+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:00.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:00.605+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:00.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:00.605+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:01.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:00.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:01.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:00.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:01.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:00.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:01.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:00.978+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:01.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:01.218+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:01.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:01.218+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:01.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:01.218+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:01.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:01.219+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:01.631 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:18:01.362+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 5834 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:18:02.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:01.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:02.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:01.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:02.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:01.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:02.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:01.616+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:02.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:02.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:02.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:02.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:02.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:02.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:02.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:02.172+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:02.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:02.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:02.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:02.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:02.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:02.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:02.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:02.003+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:02.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:02.655+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:02.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:02.655+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:02.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:02.655+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:02.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:02.656+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:03.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:02.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:03.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:02.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:03.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:02.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:03.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:02.963+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:03.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:03.202+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:03.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:03.202+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:03.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:03.202+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:03.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:03.203+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:04.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:03.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:04.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:03.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:04.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:03.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:04.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:03.630+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:04.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:04.006+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:04.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:04.006+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:04.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:04.006+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:04.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:04.006+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:04.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:04.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:04.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:04.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:04.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:04.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:04.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:04.170+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:05.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:04.671+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:05.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:04.671+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:05.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:04.671+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:05.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:04.672+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:05.534 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:05.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:05.534 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:05.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:05.534 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:05.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:05.534 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:05.037+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:05.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:05.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:05.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:05.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:05.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:05.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:05.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:05.130+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:06.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:05.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:06.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:05.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:06.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:05.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:06.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:05.664+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:06.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:05.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:06.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:05.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:06.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:05.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:06.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:05.994+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:06.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:06.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:06.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:06.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:06.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:06.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:06.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:06.178+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:06.728 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:18:06.362+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 5928 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:18:07.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:06.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:07.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:06.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:07.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:06.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:07.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:06.713+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:07.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:07.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:07.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:07.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:07.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:07.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:07.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:07.155+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:07.534 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:07.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:07.534 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:07.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:07.534 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:07.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:07.535 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:07.019+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:08.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:07.696+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:08.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:07.696+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:08.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:07.696+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:08.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:07.697+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:08.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:08.184+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:08.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:08.184+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:08.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:08.184+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:08.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:08.184+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:08.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:08.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:08.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:08.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:08.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:08.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:08.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:08.004+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:08.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:08.683+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:08.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:08.683+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:08.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:08.683+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:08.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:08.684+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:09.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:09.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:09.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:09.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:09.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:09.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:09.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:09.153+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:09.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:08.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:09.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:08.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:09.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:08.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:09.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:08.960+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:09.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:09.642+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:09.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:09.642+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:09.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:09.642+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:09.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:09.643+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:10.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:10.129+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:10.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:10.129+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:10.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:10.129+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:10.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:10.130+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:10.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:09.939+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:10.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:09.939+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:10.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:09.939+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:10.285 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:09.940+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:10.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:10.610+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:10.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:10.610+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:10.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:10.610+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:10.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:10.611+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:11.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:10.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:11.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:10.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:11.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:10.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:11.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:10.939+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:11.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:11.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:11.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:11.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:11.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:11.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:11.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:11.091+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:11.784 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:18:11.364+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 6021 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:18:11.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:11.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:11.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:11.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:11.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:11.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:11.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:11.592+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:12.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:12.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:12.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:12.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:12.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:12.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:12.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:12.058+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:12.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:11.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:12.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:11.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:12.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:11.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:12.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:11.900+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:12.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:12.627+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:12.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:12.627+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:12.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:12.627+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:12.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:12.627+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:13.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:13.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:13.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:13.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:13.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:13.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:13.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:13.086+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:13.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:12.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:13.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:12.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:13.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:12.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:13.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:12.909+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:13.889 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:13.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:13.889 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:13.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:13.889 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:13.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:13.889 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:13.661+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:14.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:14.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:14.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:14.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:14.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:14.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:14.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:14.094+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:14.213 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:13.929+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:14.213 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:13.929+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:14.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:13.929+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:14.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:13.930+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:14.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:14.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:14.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:14.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:14.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:14.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:14.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:14.627+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:15.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:14.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:15.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:14.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:15.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:14.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:15.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:14.953+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:15.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:15.072+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:15.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:15.072+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:15.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:15.072+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:15.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:15.072+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:16.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:15.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:16.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:15.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:16.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:15.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:16.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:15.660+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:16.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:15.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:16.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:15.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:16.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:15.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:16.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:15.991+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:16.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:16.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:16.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:16.071+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:16.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:16.071+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:16.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:16.071+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:16.783 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:18:16.365+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 6116 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:18:16.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:16.613+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:16.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:16.613+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:16.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:16.613+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:16.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:16.614+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:17.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:17.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:17.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:17.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:17.534 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:17.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:17.534 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:17.021+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:17.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:17.042+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:17.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:17.042+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:17.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:17.042+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:17.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:17.043+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:18.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:17.655+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:18.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:17.655+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:18.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:17.655+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:18.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:17.656+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:18.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:18.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:18.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:18.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:18.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:18.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:18.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:18.044+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:18.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:17.995+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:18.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:17.995+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:18.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:17.995+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:18.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:17.995+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:19.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:18.682+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:19.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:18.682+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:19.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:18.682+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:19.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:18.683+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:19.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:19.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:19.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:19.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:19.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:19.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:19.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:19.007+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:19.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:19.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:19.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:19.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:19.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:19.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:19.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:19.013+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:20.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:19.715+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:20.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:19.715+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:20.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:19.715+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:20.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:19.716+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:20.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:20.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:20.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:20.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:20.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:20.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:20.534 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:20.043+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:20.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:20.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:20.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:20.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:20.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:20.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:20.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:20.047+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:21.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:20.736+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:21.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:20.736+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:21.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:20.736+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:21.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:20.737+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:21.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:21.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:21.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:21.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:21.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:21.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:21.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:21.070+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:21.381 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:21.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:21.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:21.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:21.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:21.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:21.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:21.045+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:21.733 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:18:21.366+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 6210 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:18:22.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:21.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:22.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:21.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:22.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:21.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:22.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:21.719+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:22.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:22.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:22.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:22.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:22.534 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:22.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:22.534 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:22.056+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:22.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:22.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:22.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:22.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:22.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:22.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:22.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:22.095+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:23.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:22.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:23.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:22.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:23.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:22.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:23.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:22.755+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:23.533 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:23.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:23.533 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:23.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:23.533 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:23.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:23.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:23.065+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:23.534 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:23.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:23.534 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:23.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:23.534 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:23.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:23.534 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:23.037+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:24.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:23.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:24.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:23.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:24.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:23.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:24.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:23.708+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:24.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:24.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:24.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:24.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:24.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:24.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:24.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:24.082+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:24.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:24.006+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:24.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:24.006+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:24.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:24.007+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:24.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:24.007+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:25.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:24.741+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:25.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:24.741+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:25.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:24.741+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:25.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:24.742+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:25.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:25.033+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:25.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:25.033+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:25.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:25.033+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:25.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:25.033+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:25.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:24.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:25.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:24.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:25.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:24.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:25.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:24.996+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:26.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:25.692+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:26.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:25.692+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:26.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:25.692+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:26.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:25.693+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:26.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:25.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:26.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:25.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:26.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:25.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:26.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:25.991+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:26.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:26.014+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:26.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:26.014+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:26.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:26.014+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:26.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:26.015+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:26.753 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:18:26.367+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 6306 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:18:27.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:26.973+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:27.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:26.973+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:27.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:26.973+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:27.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:26.974+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:27.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:26.739+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:27.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:26.739+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:27.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:26.739+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:27.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:26.740+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:27.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:27.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:27.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:27.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:27.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:27.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:27.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:27.018+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:28.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:27.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:28.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:27.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:28.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:27.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:28.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:27.945+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:28.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:27.709+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:28.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:27.709+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:28.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:27.709+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:28.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:27.710+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:28.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:28.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:28.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:28.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:28.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:28.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:28.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:28.021+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:28.964 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:28.664+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:28.964 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:28.664+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:28.964 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:28.664+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:28.964 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:28.665+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:29.249 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:28.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:29.249 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:28.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:29.249 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:28.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:29.249 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:28.950+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:29.250 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:28.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:29.250 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:28.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:29.250 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:28.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:29.250 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:28.990+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:29.952 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:29.630+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:29.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:29.630+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:29.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:29.630+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:29.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:29.631+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:30.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:29.944+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:30.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:29.944+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:30.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:29.944+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:30.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:29.944+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:30.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:29.937+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:30.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:29.937+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:30.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:29.937+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:30.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:29.938+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:30.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:30.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:30.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:30.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:30.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:30.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:30.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:30.626+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:31.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:30.974+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:31.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:30.974+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:31.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:30.974+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:31.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:30.974+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:31.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:30.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:31.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:30.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:31.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:30.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:31.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:30.906+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:31.783 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:18:31.368+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 6401 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:18:31.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:31.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:31.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:31.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:31.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:31.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:31.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:31.605+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:32.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:31.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:32.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:31.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:32.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:31.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:32.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:31.936+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:32.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:31.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:32.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:31.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:32.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:31.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:32.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:31.991+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:32.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:32.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:32.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:32.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:32.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:32.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:32.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:32.579+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:33.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:32.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:33.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:32.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:33.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:32.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:33.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:32.979+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:33.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:33.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:33.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:33.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:33.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:33.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:33.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:33.031+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:34.030 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:33.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:34.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:33.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:34.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:33.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:34.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:33.617+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:34.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:34.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:34.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:34.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:34.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:34.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:34.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:34.059+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:34.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:34.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:34.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:34.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:34.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:34.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:34.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:34.016+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:34.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:34.627+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:34.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:34.627+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:34.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:34.627+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:34.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:34.628+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:35.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:35.000+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:35.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:35.000+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:35.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:35.000+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:35.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:35.001+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:35.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:35.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:35.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:35.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:35.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:35.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:35.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:35.028+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:35.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:35.580+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:35.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:35.580+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:35.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:35.580+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:35.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:35.581+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:36.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:36.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:36.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:36.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:36.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:36.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:36.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:36.026+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:36.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:35.974+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:36.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:35.974+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:36.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:35.974+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:36.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:35.975+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:36.783 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:18:36.369+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 6499 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:18:36.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:36.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:36.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:36.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:36.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:36.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:36.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:36.570+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:37.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:37.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:37.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:37.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:37.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:37.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:37.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:37.052+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:37.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:36.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:37.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:36.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:37.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:36.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:37.284 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:36.962+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:37.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:37.585+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:37.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:37.585+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:37.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:37.585+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:37.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:37.585+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:38.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:37.962+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:38.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:37.962+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:38.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:37.962+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:38.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:37.963+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:38.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:38.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:38.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:38.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:38.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:38.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:38.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:38.082+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:38.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:38.584+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:38.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:38.584+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:38.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:38.584+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:38.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:38.585+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:39.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:38.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:39.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:38.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:39.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:38.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:39.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:38.915+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:39.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:39.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:39.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:39.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:39.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:39.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:39.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:39.099+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:39.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:39.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:39.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:39.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:39.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:39.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:39.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:39.583+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:40.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:39.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:40.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:39.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:40.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:39.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:40.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:39.895+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:40.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:40.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:40.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:40.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:40.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:40.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:40.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:40.110+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:40.891 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:40.603+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:40.891 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:40.603+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:40.891 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:40.603+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:40.891 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:40.604+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:41.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:41.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:41.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:41.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:41.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:41.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:41.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:41.062+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:41.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:40.876+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:41.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:40.876+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:41.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:40.876+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:41.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:40.877+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:41.783 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:18:41.370+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 6595 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:18:41.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:41.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:41.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:41.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:41.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:41.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:41.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:41.594+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:42.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:42.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:42.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:42.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:42.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:42.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:42.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:42.075+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:42.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:41.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:42.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:41.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:42.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:41.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:42.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:41.884+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:42.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:42.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:42.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:42.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:42.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:42.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:42.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:42.547+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:43.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:43.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:43.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:43.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:43.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:43.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:43.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:43.086+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:43.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:42.900+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:43.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:42.900+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:43.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:42.900+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:43.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:42.900+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:43.908 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:43.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:43.908 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:43.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:43.908 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:43.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:43.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:43.547+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:44.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:43.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:44.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:43.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:44.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:43.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:44.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:43.894+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:44.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:44.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:44.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:44.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:44.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:44.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:44.284 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:44.096+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:44.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:44.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:44.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:44.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:44.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:44.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:44.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:44.547+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:45.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:45.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:45.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:45.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:45.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:45.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:45.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:45.055+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:45.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:44.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:45.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:44.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:45.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:44.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:45.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:44.854+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:45.840 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:45.552+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:45.840 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:45.552+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:45.840 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:45.552+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:45.840 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:45.553+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:46.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:46.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:46.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:46.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:46.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:46.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:46.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:46.028+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:46.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:45.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:46.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:45.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:46.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:45.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:46.283 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:45.826+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:46.783 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:18:46.371+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 6692 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:18:46.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:46.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:46.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:46.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:46.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:46.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:46.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:46.543+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:47.092 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:46.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:47.092 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:46.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:47.093 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:46.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:47.093 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:46.805+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:47.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:47.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:47.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:47.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:47.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:47.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:47.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:47.078+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:47.810 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:47.520+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:47.810 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:47.520+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:47.810 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:47.520+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:47.810 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:47.521+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:48.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:47.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:48.122 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:47.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:48.122 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:47.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:48.122 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:47.796+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:48.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:48.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:48.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:48.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:48.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:48.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:48.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:48.108+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:48.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:48.516+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:48.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:48.516+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:48.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:48.516+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:48.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:48.516+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:48.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:48.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:48.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:48.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:48.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:48.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:48.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:48.752+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:49.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:49.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:49.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:49.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:49.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:49.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:49.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:49.145+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:50.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:49.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:50.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:49.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:50.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:49.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:50.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:49.755+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:50.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:49.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:50.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:49.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:50.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:49.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:50.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:49.540+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:50.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:50.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:50.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:50.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:50.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:50.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:50.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:50.156+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:50.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:50.513+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:50.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:50.513+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:50.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:50.513+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:50.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:50.513+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:51.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:50.790+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:51.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:50.790+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:51.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:50.790+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:51.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:50.791+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:51.489 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:18:51.372+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 6787 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:18:51.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:51.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:51.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:51.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:51.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:51.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:51.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:51.172+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:51.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:51.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:51.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:51.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:51.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:51.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:51.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:51.475+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:52.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:51.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:52.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:51.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:52.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:51.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:52.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:51.805+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:52.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:52.188+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:52.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:52.188+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:52.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:52.188+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:52.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:52.189+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:52.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:52.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:52.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:52.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:52.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:52.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:52.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:52.476+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:53.222 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:52.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:53.222 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:52.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:53.222 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:52.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:53.222 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:52.779+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:53.494 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:53.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:53.494 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:53.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:53.494 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:53.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:53.494 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:53.208+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:53.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:53.480+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:53.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:53.480+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:53.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:53.480+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:53.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:53.481+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:54.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:53.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:54.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:53.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:54.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:53.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:54.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:53.792+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:54.532 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:54.469+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:54.532 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:54.469+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:54.533 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:54.469+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:54.533 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:54.470+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:54.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:54.252+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:54.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:54.252+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:54.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:54.252+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:54.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:54.252+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:55.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:54.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:55.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:54.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:55.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:54.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:55.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:54.768+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:55.515 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:55.219+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:55.515 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:55.220+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:55.515 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:55.220+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:55.515 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:55.220+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:55.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:55.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:55.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:55.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:55.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:55.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:55.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:55.501+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:55.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:55.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:55.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:55.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:55.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:55.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:55.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:55.739+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:56.482 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:18:56.373+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 6884 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:18:56.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:56.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:56.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:56.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:56.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:56.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:56.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:56.215+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:56.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:56.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:56.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:56.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:56.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:56.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:56.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:56.469+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:57.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:56.743+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:57.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:56.743+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:57.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:56.743+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:57.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:56.744+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:57.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:57.218+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:57.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:57.218+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:57.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:57.218+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:57.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:57.219+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:57.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:57.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:57.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:57.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:57.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:57.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:57.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:57.701+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:57.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:57.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:57.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:57.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:57.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:57.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:57.784 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:57.476+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:58.532 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:58.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:58.533 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:58.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:58.533 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:58.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:58.533 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:58.488+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:58.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:58.261+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:58.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:58.261+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:58.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:58.261+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:58.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:58.262+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:59.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:58.704+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:59.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:58.704+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:59.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:58.704+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:59.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:58.704+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:18:59.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:59.232+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:59.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:59.232+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:59.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:59.232+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:18:59.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:18:59.233+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:59.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:59.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:59.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:59.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:59.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:59.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:18:59.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:18:59.511+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:18:59.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:59.743+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:59.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:59.743+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:59.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:59.743+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:18:59.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:18:59.743+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:00.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:00.188+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:19:00.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:00.188+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:19:00.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:00.188+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:19:00.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:00.188+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:00.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:00.485+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:00.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:00.485+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:00.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:00.485+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:00.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:00.485+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:00.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:00.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:00.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:00.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:00.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:00.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:00.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:00.695+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:01.509 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:19:01.374+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 6981 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:19:01.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:01.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:01.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:01.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:01.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:01.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:01.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:01.479+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:01.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:01.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:32.884896+0000 front 2024-02-19T19:08:32.884867+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:19:01.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:01.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:32.884799+0000 front 2024-02-19T19:08:32.884834+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:19:01.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:01.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:32.884818+0000 front 2024-02-19T19:08:32.884849+0000 (oldest deadline 2024-02-19T19:08:56.384730+0000) 2024-02-19T19:19:01.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:01.231+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:01.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:01.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:01.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:01.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:01.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:01.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:01.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:01.663+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:02.491 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:02.212+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:02.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:02.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:02.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:02.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:02.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:02.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:02.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:02.619+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:02.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:02.477+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:02.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:02.477+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:02.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:02.477+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:02.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:02.478+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:03.471 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:03.217+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:03.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:03.627+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:03.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:03.627+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:03.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:03.627+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:03.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:03.627+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:03.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:03.457+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:03.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:03.457+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:03.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:03.457+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:03.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:03.458+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:04.509 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:04.201+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:04.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:04.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:04.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:04.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:04.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:04.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:04.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:04.655+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:04.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:04.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:04.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:04.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:04.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:04.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:04.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:04.496+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:05.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:05.190+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:05.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:05.659+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:05.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:05.659+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:05.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:05.659+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:05.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:05.660+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:05.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:05.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:05.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:05.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:05.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:05.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:05.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:05.469+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:06.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:06.141+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:06.690 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:19:06.375+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 7077 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:19:06.690 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:06.447+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:06.690 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:06.447+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:06.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:06.447+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:06.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:06.448+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:07.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:06.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:07.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:06.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:07.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:06.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:07.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:06.677+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:07.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:07.159+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:07.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:07.650+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:07.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:07.650+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:07.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:07.650+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:07.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:07.650+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:07.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:07.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:07.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:07.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:07.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:07.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:07.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:07.497+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:08.491 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:08.122+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:08.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:08.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:08.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:08.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:08.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:08.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:08.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:08.478+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:08.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:08.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:08.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:08.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:08.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:08.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:08.783 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:08.693+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:09.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:09.172+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:09.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-02-19T19:19:09.448+0000 7fca237f5700 0 monclient(hunting): authenticate timed out after 300 2024-02-19T19:19:09.463 INFO:teuthology.orchestra.run.smithi114.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-19T19:19:09.746 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:09.443+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:09.746 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:09.443+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:09.746 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:09.443+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:09.746 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:09.444+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:10.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:09.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:35.395466+0000 front 2024-02-19T19:08:35.395592+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:10.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:09.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:35.395671+0000 front 2024-02-19T19:08:35.395700+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:10.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:09.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:35.395621+0000 front 2024-02-19T19:08:35.395654+0000 (oldest deadline 2024-02-19T19:08:58.295438+0000) 2024-02-19T19:19:10.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:09.732+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:10.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T19:19:10.266 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi114 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph orch ps'" 2024-02-19T19:19:10.483 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4681efabf223408f9f0bbe4aaae5d81d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi114 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph orch ps'" 2024-02-19T19:19:10.486 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-02-19T19:19:10.497 INFO:tasks.kclient:Unmounting kernel clients... 2024-02-19T19:19:10.498 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T19:19:10.498 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:19:10.502 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:10.174+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:10.519 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T19:19:10.519 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:19:10.575 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T19:19:10.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:10.707+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:10.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:10.479+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:10.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:10.479+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:10.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:10.479+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:10.783 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:10.480+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:11.401 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:19:11.376+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 7172 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:19:11.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:11.128+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:11.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:11.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:11.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:11.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:11.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:11.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:11.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:11.507+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:12.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:11.700+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:12.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:12.110+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:12.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:12.710+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:12.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:12.511+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:12.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:12.511+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:12.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:12.511+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:12.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:12.512+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:13.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:13.146+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:14.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:13.678+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:14.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:13.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 since back 2024-02-19T19:08:33.213274+0000 front 2024-02-19T19:08:33.213207+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:14.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:13.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 since back 2024-02-19T19:08:33.213221+0000 front 2024-02-19T19:08:33.213257+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:14.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:13.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 since back 2024-02-19T19:08:33.213294+0000 front 2024-02-19T19:08:33.213306+0000 (oldest deadline 2024-02-19T19:08:57.913392+0000) 2024-02-19T19:19:14.033 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:13.560+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:14.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:14.147+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:15.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:14.606+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:15.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:14.669+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:15.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:15.105+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:16.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:15.598+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:16.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:15.673+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:16.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:16.109+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:16.656 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:19:16.377+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 7268 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:19:16.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:16.565+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:17.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:16.643+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:17.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:17.100+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:18.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:17.591+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:18.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:17.625+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:18.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:18.142+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:19.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:18.559+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:19.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:18.579+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:19.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:19.136+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:19.901 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:19.609+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:19.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:19.540+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:20.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:20.146+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:21.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:20.553+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:21.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:20.589+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:21.472 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:19:21.378+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 7364 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:19:21.473 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:21.188+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:21.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:21.543+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:21.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:21.530+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:22.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:22.219+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:23.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:22.551+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:23.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:22.558+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:23.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:23.186+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:23.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:23.186+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:23.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:23.186+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:23.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:23.187+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:24.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:23.580+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:24.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:23.545+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:24.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:24.197+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:24.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:24.197+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:24.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:24.197+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:24.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:24.197+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:24.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:24.541+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:24.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:24.593+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:25.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:25.214+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:25.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:25.214+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:25.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:25.214+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:25.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:25.214+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:26.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:25.536+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:26.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:25.593+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:26.393 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:26.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:26.393 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:26.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:26.393 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:26.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:26.393 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:26.199+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:26.781 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:26.565+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:26.782 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:19:26.379+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 7460 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:19:26.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:26.614+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:27.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:27.197+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:27.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:27.197+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:27.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:27.197+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:27.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:27.198+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:28.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:27.611+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:28.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:27.566+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:28.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:28.165+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:28.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:28.165+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:28.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:28.165+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:28.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:28.166+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:29.031 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:28.636+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:29.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:28.543+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:29.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:29.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:29.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:29.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:29.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:29.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:29.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:29.209+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:30.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:29.541+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:30.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:29.602+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:30.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:30.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:30.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:30.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:30.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:30.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:30.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:30.239+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:31.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:30.559+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:31.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:30.591+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:31.531 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:19:31.380+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 7556 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:19:31.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:31.263+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:31.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:31.263+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:31.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:31.263+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:31.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:31.263+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:32.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:31.611+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:32.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:31.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:32.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:31.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:32.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:31.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:32.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:31.605+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:32.577 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:32.274+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:32.577 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:32.274+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:32.577 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:32.274+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:32.577 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:32.274+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:33.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:32.587+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:33.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:32.563+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:33.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:32.563+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:33.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:32.563+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:33.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:32.563+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:33.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:33.230+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:33.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:33.230+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:33.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:33.230+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:33.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:33.230+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:34.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:33.610+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:34.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:33.584+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:34.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:33.584+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:34.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:33.584+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:34.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:33.585+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:34.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:34.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:34.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:34.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:34.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:34.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:34.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:34.236+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:35.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:34.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:35.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:34.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:35.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:34.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:35.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:34.605+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:35.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:34.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:35.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:34.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:35.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:34.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:35.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:34.635+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:35.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:35.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:35.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:35.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:35.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:35.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:35.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:35.191+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:36.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:35.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:36.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:35.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:36.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:35.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:36.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:35.637+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:36.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:35.681+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:36.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:35.681+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:36.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:35.681+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:36.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:35.682+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:36.518 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:19:36.381+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 7653 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:19:36.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:36.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:36.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:36.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:36.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:36.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:36.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:36.176+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:36.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:36.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:36.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:36.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:36.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:36.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:36.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:36.637+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:36.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:36.664+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:36.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:36.664+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:36.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:36.664+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:36.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:36.665+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:37.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:37.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:37.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:37.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:37.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:37.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:37.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:37.204+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:38.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:37.647+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:38.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:37.647+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:38.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:37.647+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:38.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:37.648+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:38.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:37.599+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:38.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:37.599+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:38.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:37.599+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:38.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:37.600+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:38.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:38.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:38.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:38.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:38.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:38.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:38.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:38.181+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:39.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:38.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:39.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:38.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:39.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:38.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:39.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:38.675+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:39.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:38.575+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:39.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:38.575+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:39.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:38.575+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:39.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:38.575+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:39.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:39.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:39.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:39.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:39.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:39.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:39.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:39.193+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:40.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:39.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:40.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:39.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:40.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:39.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:40.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:39.708+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:40.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:39.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:40.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:39.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:40.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:39.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:40.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:39.554+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:40.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:40.206+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:40.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:40.206+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:40.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:40.206+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:40.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:40.206+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:41.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:40.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:41.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:40.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:41.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:40.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:41.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:40.712+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:41.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:40.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:41.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:40.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:41.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:40.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:41.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:40.534+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:41.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:19:41.382+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 7750 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:19:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:41.230+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:41.230+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:41.230+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:41.231+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:41.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:41.565+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:41.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:41.565+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:41.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:41.565+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:41.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:41.566+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:41.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:41.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:41.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:41.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:41.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:41.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:41.782 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:41.686+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:42.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:42.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:42.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:42.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:42.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:42.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:42.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:42.196+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:42.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:42.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:42.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:42.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:42.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:42.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:42.976 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:42.707+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:42.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:42.601+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:42.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:42.601+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:42.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:42.601+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:42.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:42.601+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:43.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:43.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:43.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:43.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:43.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:43.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:43.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:43.152+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:44.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:43.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:44.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:43.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:44.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:43.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:44.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:43.722+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:44.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:43.582+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:44.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:43.582+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:44.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:43.582+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:44.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:43.583+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:44.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:44.200+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:44.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:44.200+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:44.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:44.200+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:44.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:44.200+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:45.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:44.745+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:45.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:44.745+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:45.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:44.745+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:45.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:44.746+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:45.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:44.546+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:45.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:44.546+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:45.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:44.546+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:45.032 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:44.547+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:45.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:45.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:45.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:45.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:45.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:45.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:45.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:45.153+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:45.799 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:45.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:45.799 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:45.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:45.799 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:45.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:45.799 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:45.535+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:46.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:45.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:46.117 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:45.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:46.117 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:45.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:46.117 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:45.787+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:46.396 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:46.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:46.397 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:46.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:46.397 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:46.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:46.397 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:46.104+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:46.781 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:19:46.383+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 7846 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:19:46.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:46.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:46.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:46.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:46.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:46.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:46.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:46.488+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:47.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:46.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:47.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:46.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:47.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:46.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:47.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:46.792+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:47.494 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:47.154+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:47.495 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:47.154+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:47.495 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:47.154+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:47.495 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:47.154+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:47.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:47.482+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:47.838 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:47.482+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:47.838 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:47.482+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:47.838 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:47.482+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:48.204 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:47.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:48.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:47.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:48.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:47.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:48.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:47.792+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:48.477 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:48.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:48.477 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:48.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:48.477 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:48.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:48.477 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:48.192+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:48.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:48.464+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:48.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:48.464+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:48.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:48.464+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:48.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:48.465+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:49.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:48.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:49.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:48.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:49.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:48.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:49.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:48.761+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:49.478 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:49.199+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:49.478 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:49.199+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:49.479 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:49.199+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:49.479 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:49.200+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:49.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:49.465+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:49.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:49.465+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:49.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:49.465+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:49.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:49.466+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:50.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:49.748+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:50.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:49.748+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:50.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:49.748+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:50.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:49.749+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:50.471 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:50.187+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:50.471 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:50.187+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:50.471 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:50.187+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:50.471 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:50.188+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:50.763 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:50.458+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:50.763 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:50.458+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:50.764 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:50.458+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:50.764 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:50.459+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:51.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:50.751+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:51.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:50.751+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:51.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:50.751+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:51.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:50.751+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:51.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:19:51.385+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 7942 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:19:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:51.445+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:51.445+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:51.445+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:51.446+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:51.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:51.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:51.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:51.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:51.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:51.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:51.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:51.237+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:52.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:51.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:52.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:51.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:52.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:51.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:52.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:51.793+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:52.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:52.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:52.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:52.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:52.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:52.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:52.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:52.192+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:52.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:52.476+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:52.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:52.476+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:52.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:52.476+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:52.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:52.477+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:53.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:52.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:53.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:52.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:53.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:52.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:53.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:52.802+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:53.449 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:53.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:53.449 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:53.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:53.449 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:53.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:53.449 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:53.167+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:53.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:53.436+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:53.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:53.436+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:53.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:53.436+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:53.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:53.437+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:54.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:53.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:54.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:53.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:54.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:53.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:54.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:53.786+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:54.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:54.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:54.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:54.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:54.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:54.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:54.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:54.123+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:54.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:54.486+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:54.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:54.486+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:54.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:54.486+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:54.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:54.486+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:55.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:54.773+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:55.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:54.773+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:55.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:54.773+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:55.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:54.773+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:55.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:55.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:55.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:55.142+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:55.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:55.506+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:55.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:55.506+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:55.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:55.506+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:55.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:55.507+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:56.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:55.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:56.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:55.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:56.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:55.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:56.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:55.810+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:56.522 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:19:56.386+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 8039 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:19:56.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:56.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:56.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:56.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:56.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:56.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:56.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:56.181+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:56.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:56.510+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:56.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:56.510+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:56.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:56.510+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:56.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:56.510+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:57.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:56.788+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:57.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:56.788+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:57.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:56.788+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:57.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:56.788+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:57.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:57.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:57.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:57.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:57.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:57.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:57.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:57.141+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:57.846 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:57.540+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:57.846 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:57.540+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:57.846 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:57.540+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:57.847 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:57.541+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:58.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:57.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:58.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:57.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:58.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:57.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:58.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:57.835+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:58.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:58.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:58.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:58.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:58.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:58.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:58.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:58.147+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:58.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:58.559+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:58.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:58.559+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:58.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:58.559+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:58.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:58.560+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:19:59.208 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:58.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:59.208 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:58.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:59.209 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:58.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:19:59.209 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:58.870+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:59.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:59.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:59.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:59.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:59.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:59.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:19:59.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:19:59.196+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:19:59.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:59.514+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:59.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:59.514+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:59.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:59.514+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:19:59.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:19:59.515+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:00.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:59.822+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:00.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:59.822+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:00.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:59.822+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:00.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:19:59.823+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:00.488 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:00.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:00.488 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:00.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:00.488 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:00.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:00.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:00.163+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:00.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:00.475+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:00.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:00.475+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:00.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:00.475+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:00.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:00.476+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:01.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:00.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:01.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:00.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:01.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:00.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:01.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:00.846+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:01.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:01.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:01.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:01.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:01.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:01.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:01.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:01.135+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:01.781 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:20:01.387+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 8136 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:20:01.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:01.453+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:01.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:01.453+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:01.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:01.453+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:01.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:01.454+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:02.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:02.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:02.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:02.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:02.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:02.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:02.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:02.123+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:02.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:01.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:02.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:01.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:02.282 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:01.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:02.282 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:01.882+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:02.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:02.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:02.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:02.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:02.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:02.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:02.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:02.433+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:03.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:02.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:03.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:02.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:03.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:02.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:03.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:02.919+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:03.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:03.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:03.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:03.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:03.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:03.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:03.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:03.109+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:03.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:03.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:03.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:03.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:03.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:03.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:03.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:03.391+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:04.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:03.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:04.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:03.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:04.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:03.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:04.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:03.874+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:04.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:04.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:04.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:04.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:04.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:04.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:04.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:04.074+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:04.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:04.431+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:04.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:04.431+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:04.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:04.431+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:04.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:04.432+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:05.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:04.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:05.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:04.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:05.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:04.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:05.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:04.875+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:05.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:05.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:05.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:05.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:05.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:05.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:05.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:05.045+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:05.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:05.462+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:05.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:05.462+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:05.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:05.462+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:05.782 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:05.462+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:06.280 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:05.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:06.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:05.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:06.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:05.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:06.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:05.838+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:06.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:06.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:06.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:06.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:06.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:06.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:06.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:06.016+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:06.780 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:20:06.388+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 8232 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:20:06.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:06.413+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:06.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:06.413+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:06.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:06.413+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:06.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:06.413+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:07.280 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:06.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:07.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:06.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:07.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:06.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:07.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:06.874+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:07.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:07.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:07.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:07.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:07.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:07.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:07.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:07.052+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:07.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:07.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:07.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:07.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:07.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:07.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:07.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:07.420+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:08.280 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:07.894+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:08.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:07.894+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:08.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:07.894+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:08.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:07.895+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:08.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:08.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:08.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:08.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:08.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:08.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:08.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:08.078+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:08.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:08.412+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:08.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:08.412+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:08.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:08.412+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:08.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:08.412+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:09.280 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:08.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:09.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:08.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:09.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:08.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:09.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:08.917+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:09.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:09.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:09.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:09.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:09.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:09.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:09.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:09.074+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:09.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:09.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:09.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:09.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:09.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:09.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:09.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:09.432+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:10.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:09.871+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:10.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:09.871+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:10.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:09.871+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:10.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:09.872+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:10.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:10.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:10.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:10.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:10.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:10.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:10.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:10.080+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:10.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:10.464+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:10.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:10.464+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:10.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:10.464+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:10.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:10.465+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:11.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:10.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:11.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:10.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:11.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:10.832+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:11.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:10.832+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:11.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:11.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:11.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:11.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:11.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:11.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:11.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:11.057+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:11.780 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:20:11.390+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 8328 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:20:11.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:11.514+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:11.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:11.514+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:11.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:11.514+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:11.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:11.515+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:12.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:11.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:12.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:11.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:12.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:11.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:12.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:11.808+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:12.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:12.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:12.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:12.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:12.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:12.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:12.282 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:12.020+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:12.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:12.501+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:12.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:12.501+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:12.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:12.501+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:12.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:12.501+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:13.030 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:12.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:13.030 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:12.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:13.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:12.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:13.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:12.760+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:13.031 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:12.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:13.031 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:12.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:13.031 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:12.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:13.031 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:12.976+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:13.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:13.481+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:13.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:13.481+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:13.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:13.481+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:13.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:13.482+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:14.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:13.741+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:14.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:13.741+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:14.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:13.741+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:14.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:13.743+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:14.280 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:14.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:14.280 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:14.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:14.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:14.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:14.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:14.003+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:14.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:14.500+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:14.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:14.500+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:14.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:14.500+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:14.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:14.501+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:15.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:14.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:15.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:14.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:15.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:14.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:15.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:14.760+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:15.032 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:14.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:15.032 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:14.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:15.032 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:14.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:15.032 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:14.996+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:15.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:15.472+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:15.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:15.472+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:15.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:15.472+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:15.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:15.473+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:16.030 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:15.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:16.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:15.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:16.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:15.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:16.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:15.768+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:16.403 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:16.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:16.403 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:16.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:16.403 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:16.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:16.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:16.030+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:16.762 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:20:16.391+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 8425 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:20:16.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:16.476+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:16.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:16.476+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:16.763 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:16.476+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:16.763 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:16.477+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:17.030 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:16.750+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:17.030 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:16.750+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:17.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:16.750+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:17.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:16.751+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:17.493 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:17.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:17.493 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:17.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:17.494 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:17.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:17.494 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:17.077+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:17.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:17.481+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:17.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:17.481+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:17.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:17.481+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:17.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:17.482+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:18.030 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:17.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:18.030 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:17.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:18.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:17.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:18.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:17.757+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:18.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:18.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:18.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:18.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:18.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:18.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:18.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:18.055+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:18.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:18.514+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:18.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:18.514+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:18.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:18.514+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:18.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:18.515+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:19.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:18.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:19.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:18.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:19.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:18.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:19.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:18.796+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:19.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:19.101+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:19.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:19.101+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:19.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:19.101+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:19.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:19.102+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:19.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:19.546+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:19.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:19.546+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:19.849 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:19.546+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:19.849 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:19.546+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:20.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:19.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:20.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:19.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:20.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:19.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:20.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:19.837+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:20.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:20.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:20.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:20.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:20.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:20.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:20.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:20.141+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:20.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:20.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:20.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:20.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:20.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:20.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:20.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:20.594+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:21.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:20.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:21.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:20.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:21.200 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:20.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:21.200 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:20.859+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:21.527 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:20:21.392+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 8522 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:20:21.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:21.187+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:21.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:21.187+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:21.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:21.187+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:21.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:21.187+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:21.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:21.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:21.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:21.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:21.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:21.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:21.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:21.581+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:22.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:21.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:22.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:21.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:22.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:21.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:22.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:21.908+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:22.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:22.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:22.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:22.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:22.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:22.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:22.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:22.160+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:22.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:22.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:22.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:22.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:22.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:22.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:22.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:22.620+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:23.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:22.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:23.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:22.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:23.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:22.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:23.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:22.929+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:23.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:23.178+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:23.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:23.178+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:23.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:23.178+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:23.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:23.178+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:23.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:23.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:23.926 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:23.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:23.926 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:23.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:23.926 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:23.637+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:24.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:23.913+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:24.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:23.913+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:24.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:23.913+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:24.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:23.914+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:24.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:24.201+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:24.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:24.201+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:24.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:24.201+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:24.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:24.201+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:24.926 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:24.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:24.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:24.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:24.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:24.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:24.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:24.624+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:25.280 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:24.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:25.280 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:24.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:25.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:24.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:25.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:24.915+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:25.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:25.154+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:25.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:25.154+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:25.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:25.154+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:25.281 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:25.155+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:25.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:25.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:25.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:25.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:25.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:25.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:25.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:25.645+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:26.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:25.905+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:26.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:25.905+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:26.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:25.905+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:26.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:25.906+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:26.529 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:20:26.393+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 8618 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:20:26.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:26.200+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:26.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:26.200+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:26.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:26.200+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:26.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:26.200+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:26.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:26.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:26.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:26.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:26.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:26.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:26.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:26.631+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:27.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:26.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:27.245 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:26.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:27.245 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:26.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:27.245 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:26.931+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:27.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:27.233+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:27.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:27.233+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:27.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:27.233+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:27.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:27.233+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:27.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:27.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:27.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:27.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:27.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:27.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:27.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:27.659+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:28.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:27.951+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:28.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:27.951+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:28.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:27.951+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:28.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:27.952+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:28.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:28.264+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:28.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:28.264+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:28.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:28.264+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:28.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:28.265+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:28.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:28.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:28.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:28.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:28.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:28.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:28.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:28.635+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:29.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:28.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:29.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:28.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:29.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:28.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:29.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:28.974+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:29.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:29.242+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:29.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:29.242+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:29.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:29.242+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:29.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:29.243+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:30.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:29.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:30.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:29.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:30.031 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:29.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:30.031 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:29.655+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:30.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:30.023+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:30.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:30.023+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:30.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:30.023+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:30.531 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:30.024+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:30.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:30.256+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:30.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:30.256+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:30.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:30.256+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:30.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:30.256+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:31.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:30.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:31.031 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:30.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:31.031 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:30.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:31.031 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:30.670+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:31.407 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:31.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:31.408 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:31.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:31.408 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:31.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:31.408 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:31.054+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:31.408 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:31.250+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:31.408 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:31.250+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:31.409 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:31.250+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:31.409 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:31.251+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:31.682 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:20:31.394+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 8714 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:20:32.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:31.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:32.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:31.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:32.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:31.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:32.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:31.671+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:32.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:32.265+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:32.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:32.265+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:32.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:32.265+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:32.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:32.265+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:32.531 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:32.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:32.531 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:32.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:32.531 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:32.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:32.531 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:32.020+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:33.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:32.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:33.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:32.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:33.031 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:32.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:33.031 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:32.702+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:33.324 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:33.021+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:33.324 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:33.021+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:33.324 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:33.021+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:33.325 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:33.022+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:33.589 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:33.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:33.590 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:33.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:33.590 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:33.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:33.590 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:33.313+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:34.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:33.677+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:34.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:33.677+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:34.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:33.677+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:34.031 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:33.678+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:34.361 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:34.065+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:34.361 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:34.065+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:34.361 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:34.065+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:34.361 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:34.066+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:34.654 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:34.349+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:34.655 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:34.349+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:34.655 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:34.349+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:34.655 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:34.349+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:35.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:34.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:35.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:34.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:35.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:34.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:35.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:34.643+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:35.391 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:35.107+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:35.391 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:35.107+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:35.392 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:35.107+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:35.392 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:35.108+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:35.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:35.614+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:35.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:35.614+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:35.719 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:35.614+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:35.719 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:35.615+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:35.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:35.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:35.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:35.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:35.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:35.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:35.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:35.380+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:36.417 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:20:36.395+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 8811 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:20:36.418 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:36.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:36.418 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:36.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:36.418 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:36.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:36.418 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:36.148+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:36.418 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:36.381+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:36.419 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:36.382+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:36.419 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:36.382+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:36.419 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:36.382+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:36.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:36.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:36.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:36.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:36.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:36.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:36.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:36.646+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:37.404 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:37.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:37.404 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:37.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:37.404 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:37.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:37.404 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:37.134+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:37.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:37.392+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:37.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:37.392+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:37.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:37.392+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:37.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:37.393+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:38.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:37.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:38.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:37.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:38.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:37.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:38.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:37.665+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:38.435 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:38.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:38.435 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:38.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:38.435 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:38.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:38.436 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:38.156+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:38.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:38.638+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:38.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:38.638+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:38.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:38.638+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:38.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:38.639+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:38.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:38.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:38.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:38.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:38.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:38.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:38.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:38.424+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:39.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:39.141+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:39.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:39.141+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:39.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:39.141+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:39.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:39.142+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:39.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:39.609+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:39.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:39.609+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:39.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:39.609+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:39.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:39.609+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:39.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:39.455+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:39.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:39.455+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:39.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:39.455+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:39.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:39.455+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:40.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:40.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:40.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:40.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:40.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:40.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:40.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:40.114+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:40.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:40.633+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:40.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:40.633+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:40.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:40.633+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:40.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:40.633+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:40.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:40.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:40.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:40.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:40.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:40.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:40.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:40.413+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:41.408 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:41.097+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:41.409 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:41.097+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:41.409 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:41.097+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:41.409 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:41.098+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:41.780 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:20:41.397+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 8909 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:20:41.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:41.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:41.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:41.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:41.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:41.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:41.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:41.615+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:41.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:41.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:41.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:41.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:41.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:41.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:41.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:41.450+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:42.436 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:42.087+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:42.437 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:42.087+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:42.437 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:42.087+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:42.437 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:42.088+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:42.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:42.425+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:42.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:42.425+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:42.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:42.425+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:42.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:42.425+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:42.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:42.582+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:42.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:42.583+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:42.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:42.583+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:42.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:42.583+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:43.471 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:43.127+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:43.472 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:43.127+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:43.472 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:43.127+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:43.472 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:43.128+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:43.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:43.580+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:43.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:43.580+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:43.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:43.580+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:43.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:43.580+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:43.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:43.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:43.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:43.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:43.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:43.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:43.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:43.460+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:44.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:44.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:44.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:44.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:44.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:44.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:44.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:44.106+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:44.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:44.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:44.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:44.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:44.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:44.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:44.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:44.569+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:44.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:44.469+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:44.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:44.469+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:44.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:44.469+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:44.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:44.469+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:45.495 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:45.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:45.495 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:45.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:45.495 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:45.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:45.495 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:45.154+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:45.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:45.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:45.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:45.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:45.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:45.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:45.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:45.604+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:45.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:45.483+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:45.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:45.483+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:45.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:45.483+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:45.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:45.483+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:46.432 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:20:46.398+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 9005 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:20:46.432 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:46.166+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:46.433 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:46.166+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:46.433 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:46.166+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:46.433 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:46.167+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:46.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:46.574+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:46.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:46.574+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:46.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:46.574+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:46.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:46.574+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:46.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:46.515+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:46.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:46.515+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:46.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:46.515+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:46.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:46.515+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:47.496 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:47.182+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:47.496 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:47.182+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:47.496 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:47.182+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:47.496 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:47.183+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:47.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:47.551+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:47.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:47.551+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:47.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:47.551+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:47.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:47.551+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:47.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:47.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:47.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:47.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:47.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:47.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:47.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:47.485+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:48.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:48.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:48.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:48.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:48.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:48.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:48.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:48.198+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:48.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:48.595+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:48.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:48.595+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:48.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:48.595+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:48.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:48.596+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:48.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:48.495+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:48.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:48.495+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:48.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:48.495+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:48.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:48.496+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:49.460 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:49.179+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:49.460 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:49.179+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:49.461 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:49.179+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:49.461 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:49.180+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:49.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:49.612+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:49.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:49.612+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:49.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:49.612+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:49.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:49.613+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:49.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:49.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:49.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:49.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:49.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:49.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:49.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:49.449+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:50.175+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:50.175+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:50.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:50.175+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:50.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:50.176+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:50.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:50.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:50.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:50.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:50.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:50.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:50.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:50.655+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:50.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:50.455+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:50.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:50.455+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:50.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:50.455+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:50.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:50.455+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:51.424 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:20:51.399+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 9101 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:20:51.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:51.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:51.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:51.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:51.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:51.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:51.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:51.162+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:51.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:51.637+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:51.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:51.637+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:51.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:51.637+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:51.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:51.637+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:51.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:51.453+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:51.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:51.453+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:51.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:51.453+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:51.781 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:51.454+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:52.493 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:52.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:52.493 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:52.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:52.493 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:52.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:52.494 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:52.208+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:52.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:52.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:52.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:52.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:52.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:52.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:52.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:52.482+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:52.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:52.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:52.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:52.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:52.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:52.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:52.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:52.650+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:53.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:53.253+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:53.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:53.253+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:53.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:53.253+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:53.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:53.254+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:54.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:53.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:54.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:53.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:54.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:53.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:54.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:53.531+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:54.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:53.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:54.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:53.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:54.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:53.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:54.031 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:53.625+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:54.517 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:54.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:54.517 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:54.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:54.517 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:54.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:54.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:54.217+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:54.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:54.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:54.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:54.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:54.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:54.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:54.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:54.506+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:54.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:54.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:54.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:54.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:54.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:54.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:54.781 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:54.636+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:55.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:55.250+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:55.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:55.250+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:55.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:55.250+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:55.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:55.251+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:55.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:55.628+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:55.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:55.628+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:55.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:55.628+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:55.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:55.629+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:55.935 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:55.525+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:55.935 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:55.525+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:55.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:55.525+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:55.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:55.526+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:56.536 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:20:56.400+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 9198 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:20:56.537 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:56.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:56.537 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:56.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:56.537 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:56.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:56.537 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:56.509+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:56.537 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:56.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:56.537 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:56.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:56.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:56.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:56.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:56.273+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:57.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:56.663+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:57.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:56.663+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:57.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:56.663+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:57.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:56.664+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:57.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:57.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:57.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:57.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:57.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:57.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:57.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:57.559+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:57.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:57.318+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:57.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:57.318+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:57.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:57.318+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:57.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:57.318+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:58.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:57.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:58.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:57.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:58.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:57.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:58.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:57.632+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:58.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:58.538+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:58.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:58.538+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:58.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:58.538+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:58.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:58.538+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:58.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:58.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:58.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:58.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:58.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:58.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:58.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:58.305+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:59.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:58.656+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:59.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:58.656+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:59.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:58.656+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:20:59.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:58.657+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:20:59.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:59.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:59.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:59.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:59.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:59.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:20:59.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:20:59.296+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:20:59.687 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:59.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:59.687 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:59.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:59.688 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:59.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:20:59.688 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:20:59.498+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:00.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:59.675+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:00.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:59.675+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:00.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:59.675+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:00.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:20:59.676+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:00.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:00.280+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:00.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:00.280+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:00.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:00.280+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:00.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:00.281+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:00.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:00.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:00.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:00.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:00.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:00.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:00.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:00.714+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:00.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:00.539+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:00.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:00.539+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:00.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:00.539+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:00.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:00.540+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:01.758 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:21:01.402+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 9295 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:21:01.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:01.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:01.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:01.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:01.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:01.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:01.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:01.555+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:01.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:01.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:01.760 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:01.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:01.760 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:01.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:01.760 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:01.324+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:02.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:01.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:02.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:01.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:02.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:01.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:02.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:01.748+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:02.585 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:02.306+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:02.586 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:02.306+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:02.586 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:02.306+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:02.586 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:02.307+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:02.845 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:02.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:02.845 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:02.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:02.846 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:02.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:02.846 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:02.709+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:02.846 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:02.574+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:02.846 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:02.574+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:02.846 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:02.574+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:02.846 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:02.575+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:03.591 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:03.292+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:03.591 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:03.292+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:03.591 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:03.292+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:03.591 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:03.292+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:04.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:03.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:04.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:03.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:04.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:03.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:04.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:03.734+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:04.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:03.619+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:04.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:03.619+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:04.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:03.619+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:04.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:03.620+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:04.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:04.254+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:04.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:04.254+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:04.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:04.254+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:04.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:04.255+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:05.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:04.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:05.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:04.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:05.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:04.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:05.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:04.754+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:05.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:04.593+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:05.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:04.593+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:05.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:04.593+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:05.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:04.593+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:05.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:05.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:05.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:05.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:05.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:05.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:05.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:05.209+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:05.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:05.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:05.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:05.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:05.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:05.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:05.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:05.559+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:05.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:05.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:05.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:05.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:05.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:05.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:05.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:05.763+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:06.529 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:21:06.403+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 9388 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:21:06.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:06.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:06.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:06.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:06.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:06.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:06.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:06.177+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:07.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:06.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:07.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:06.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:07.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:06.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:07.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:06.769+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:07.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:06.567+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:07.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:06.567+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:07.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:06.567+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:07.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:06.567+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:07.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:07.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:07.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:07.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:07.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:07.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:07.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:07.193+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:08.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:07.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:08.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:07.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:08.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:07.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:08.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:07.811+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:08.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:07.585+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:08.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:07.585+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:08.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:07.585+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:08.031 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:07.585+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:08.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:08.214+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:08.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:08.214+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:08.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:08.214+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:08.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:08.215+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:09.029 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:08.627+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:09.029 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:08.627+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:09.029 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:08.627+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:09.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:08.628+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:09.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:08.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:09.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:08.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:09.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:08.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:09.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:08.807+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:09.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:09.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:09.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:09.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:09.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:09.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:09.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:09.199+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:10.029 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:09.598+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:10.029 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:09.598+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:10.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:09.598+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:10.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:09.598+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:10.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:09.781+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:10.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:09.781+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:10.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:09.781+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:10.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:09.781+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:10.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:10.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:10.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:10.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:10.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:10.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:10.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:10.162+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:10.646 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-19T19:21:10.646 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-02-19T19:21:10.647 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T19:21:10.647 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:21:10.664 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-02-19T19:21:10.664 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:21:10.721 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T19:21:11.029 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:10.554+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:11.029 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:10.554+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:11.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:10.554+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:11.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:10.554+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:11.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:10.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:11.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:10.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:11.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:10.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:11.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:10.740+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:11.415 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:11.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:11.415 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:11.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:11.416 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:11.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:11.416 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:11.165+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:11.739 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:21:11.404+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 9478 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:21:11.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:11.524+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:11.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:11.524+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:11.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:11.524+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:11.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:11.525+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:12.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:11.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:12.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:11.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:12.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:11.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:12.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:11.729+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:12.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:12.150+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:12.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:12.150+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:12.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:12.150+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:12.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:12.151+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:12.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:12.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:12.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:12.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:12.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:12.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:12.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:12.511+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:12.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:12.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:12.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:12.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:12.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:12.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:12.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:12.706+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:13.497 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:13.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:13.497 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:13.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:13.497 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:13.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:13.497 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:13.170+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:13.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:13.485+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:13.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:13.485+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:13.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:13.485+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:13.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:13.486+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:13.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:13.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:13.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:13.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:13.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:13.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:13.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:13.697+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:14.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:14.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:14.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:14.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:14.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:14.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:14.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:14.175+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:14.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:14.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:14.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:14.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:14.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:14.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:14.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:14.491+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:15.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:14.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:15.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:14.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:15.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:14.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:15.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:14.747+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:15.498 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:15.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:15.498 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:15.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:15.498 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:15.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:15.498 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:15.171+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:15.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:15.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:15.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:15.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:15.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:15.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:15.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:15.714+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:15.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:15.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:15.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:15.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:15.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:15.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:15.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:15.488+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:16.416 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:16.188+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:16.417 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:16.188+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:16.417 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:16.188+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:16.417 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:16.189+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:16.739 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:21:16.405+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 9566 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:21:16.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:16.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:16.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:16.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:16.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:16.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:16.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:16.495+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:17.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:16.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:17.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:16.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:17.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:16.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:17.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:16.729+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:17.472 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:17.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:17.472 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:17.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:17.472 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:17.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:17.473 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:17.170+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:17.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:17.461+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:17.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:17.461+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:17.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:17.461+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:17.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:17.462+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:18.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:17.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:18.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:17.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:18.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:17.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:18.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:17.758+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:18.442 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:18.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:18.442 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:18.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:18.442 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:18.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:18.442 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:18.179+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:18.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:18.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:18.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:18.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:18.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:18.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:18.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:18.432+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:19.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:18.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:19.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:18.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:19.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:18.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:19.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:18.795+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:19.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:19.434+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:19.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:19.434+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:19.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:19.434+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:19.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:19.434+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:19.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:19.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:19.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:19.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:19.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:19.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:19.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:19.200+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:20.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:19.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:20.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:19.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:20.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:19.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:20.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:19.810+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:20.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:20.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:20.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:20.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:20.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:20.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:20.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:20.219+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:20.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:20.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:20.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:20.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:20.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:20.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:20.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:20.408+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:21.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:20.788+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:21.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:20.788+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:21.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:20.788+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:21.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:20.789+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:21.529 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:21:21.407+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 9654 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:21:21.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:21.364+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:21.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:21.364+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:21.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:21.364+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:21.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:21.365+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:21.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:21.190+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:21.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:21.190+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:21.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:21.190+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:21.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:21.191+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:22.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:21.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:22.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:21.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:22.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:21.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:22.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:21.741+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:22.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:22.179+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:22.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:22.179+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:22.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:22.179+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:22.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:22.179+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:22.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:22.370+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:22.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:22.370+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:22.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:22.370+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:22.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:22.370+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:23.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:22.721+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:23.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:22.721+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:23.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:22.721+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:23.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:22.721+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:23.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:23.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:23.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:23.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:23.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:23.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:23.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:23.175+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:23.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:23.381+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:23.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:23.381+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:23.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:23.381+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:23.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:23.381+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:24.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:23.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:24.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:23.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:24.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:23.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:24.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:23.764+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:24.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:24.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:24.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:24.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:24.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:24.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:24.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:24.148+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:24.737 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:24.403+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:24.737 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:24.403+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:24.737 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:24.403+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:24.737 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:24.404+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:25.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:24.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:25.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:24.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:25.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:24.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:25.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:24.727+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:25.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:25.152+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:25.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:25.152+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:25.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:25.152+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:25.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:25.153+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:25.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:25.359+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:25.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:25.359+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:25.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:25.359+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:25.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:25.359+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:26.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:25.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:26.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:25.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:26.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:25.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:26.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:25.713+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:26.330 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:26.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:26.330 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:26.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:26.330 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:26.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:26.330 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:26.166+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:26.699 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:21:26.408+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 9744 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:21:26.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:26.319+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:26.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:26.319+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:26.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:26.319+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:26.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:26.319+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:27.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:26.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:27.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:26.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:27.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:26.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:27.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:26.689+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:27.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:27.321+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:27.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:27.321+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:27.422 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:27.321+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:27.422 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:27.321+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:27.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:27.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:27.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:27.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:27.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:27.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:27.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:27.141+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:28.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:27.722+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:28.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:27.722+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:28.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:27.722+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:28.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:27.723+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:28.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:28.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:28.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:28.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:28.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:28.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:28.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:28.162+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:28.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:28.334+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:28.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:28.334+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:28.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:28.334+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:28.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:28.334+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:29.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:28.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:29.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:28.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:29.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:28.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:29.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:28.709+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:29.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:29.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:29.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:29.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:29.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:29.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:29.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:29.174+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:29.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:29.305+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:29.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:29.305+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:29.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:29.305+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:29.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:29.305+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:30.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:29.675+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:30.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:29.675+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:30.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:29.675+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:30.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:29.675+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:30.493 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:30.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:30.493 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:30.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:30.493 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:30.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:30.494 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:30.162+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:30.494 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:30.316+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:30.494 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:30.316+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:30.494 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:30.316+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:30.494 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:30.316+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:30.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:30.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:30.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:30.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:30.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:30.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:30.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:30.661+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:31.529 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:21:31.409+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 9834 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:21:31.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:31.361+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:31.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:31.361+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:31.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:31.361+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:31.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:31.361+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:31.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:31.190+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:31.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:31.190+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:31.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:31.190+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:31.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:31.191+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:32.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:31.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:32.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:31.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:32.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:31.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:32.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:31.687+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:32.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:32.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:32.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:32.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:32.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:32.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:32.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:32.174+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:32.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:32.383+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:32.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:32.383+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:32.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:32.383+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:32.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:32.383+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:33.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:32.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:33.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:32.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:33.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:32.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:33.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:32.645+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:33.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:33.361+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:33.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:33.361+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:33.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:33.361+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:33.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:33.361+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:33.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:33.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:33.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:33.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:33.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:33.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:33.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:33.194+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:34.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:33.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:34.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:33.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:34.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:33.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:34.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:33.671+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:34.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:34.352+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:34.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:34.352+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:34.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:34.352+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:34.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:34.353+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:34.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:34.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:34.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:34.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:34.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:34.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:34.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:34.240+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:35.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:34.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:35.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:34.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:35.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:34.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:35.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:34.654+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:35.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:35.346+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:35.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:35.346+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:35.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:35.346+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:35.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:35.346+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:35.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:35.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:35.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:35.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:35.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:35.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:35.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:35.209+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:36.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:35.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:36.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:35.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:36.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:35.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:36.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:35.617+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:36.445 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:21:36.411+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 9931 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:21:36.445 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:36.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:36.445 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:36.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:36.446 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:36.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:36.446 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:36.179+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:36.446 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:36.369+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:36.446 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:36.369+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:36.446 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:36.369+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:36.446 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:36.370+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:36.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:36.586+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:36.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:36.586+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:36.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:36.586+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:36.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:36.586+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:37.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:37.332+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:37.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:37.332+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:37.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:37.332+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:37.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:37.332+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:37.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:37.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:37.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:37.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:37.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:37.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:37.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:37.159+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:38.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:37.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:38.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:37.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:38.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:37.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:38.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:37.553+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:38.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:38.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:38.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:38.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:38.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:38.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:38.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:38.152+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:38.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:38.328+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:38.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:38.328+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:38.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:38.328+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:38.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:38.329+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:39.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:38.589+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:39.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:38.589+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:39.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:38.589+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:39.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:38.590+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:39.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:39.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:39.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:39.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:39.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:39.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:39.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:39.115+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:39.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:39.348+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:39.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:39.348+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:39.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:39.348+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:39.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:39.348+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:40.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:39.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:40.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:39.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:40.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:39.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:40.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:39.632+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:40.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:40.082+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:40.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:40.082+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:40.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:40.082+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:40.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:40.083+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:40.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:40.320+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:40.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:40.320+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:40.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:40.320+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:40.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:40.320+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:41.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:40.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:41.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:40.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:41.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:40.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:41.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:40.616+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:41.420 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:41.350+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:41.420 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:41.350+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:41.420 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:41.350+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:41.420 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:41.351+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:41.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:41.115+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:41.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:41.115+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:41.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:41.115+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:41.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:41.116+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:41.779 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:21:41.412+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 10028 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:21:41.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:41.575+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:41.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:41.575+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:41.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:41.575+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:41.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:41.576+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:42.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:42.134+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:42.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:42.134+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:42.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:42.134+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:42.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:42.135+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:42.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:42.326+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:42.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:42.326+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:42.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:42.326+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:42.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:42.326+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:43.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:42.539+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:43.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:42.539+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:43.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:42.539+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:43.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:42.540+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:43.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:43.098+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:43.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:43.098+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:43.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:43.098+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:43.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:43.099+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:43.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:43.340+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:43.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:43.340+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:43.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:43.340+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:43.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:43.340+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:44.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:43.589+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:44.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:43.589+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:44.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:43.589+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:44.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:43.590+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:44.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:44.305+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:44.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:44.305+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:44.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:44.305+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:44.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:44.305+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:44.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:44.137+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:44.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:44.137+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:44.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:44.137+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:44.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:44.138+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:45.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:44.632+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:45.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:44.632+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:45.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:44.632+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:45.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:44.633+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:45.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:45.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:45.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:45.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:45.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:45.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:45.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:45.324+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:45.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:45.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:45.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:45.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:45.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:45.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:45.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:45.107+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:46.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:45.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:46.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:45.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:46.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:45.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:46.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:45.657+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:46.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:46.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:46.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:46.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:46.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:46.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:46.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:46.081+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:46.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:46.321+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:46.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:46.321+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:46.425 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:46.321+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:46.425 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:46.322+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:46.684 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:21:46.413+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 10124 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:21:47.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:46.674+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:47.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:46.674+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:47.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:46.674+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:47.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:46.675+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:47.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:47.293+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:47.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:47.293+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:47.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:47.293+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:47.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:47.293+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:47.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:47.089+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:47.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:47.089+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:47.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:47.089+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:47.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:47.090+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:48.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:47.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:48.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:47.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:48.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:47.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:48.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:47.627+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:48.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:48.267+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:48.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:48.267+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:48.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:48.267+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:48.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:48.268+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:48.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:48.059+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:48.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:48.059+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:48.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:48.059+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:48.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:48.059+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:49.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:48.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:49.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:48.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:49.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:48.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:49.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:48.670+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:49.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:49.281+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:49.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:49.281+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:49.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:49.281+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:49.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:49.282+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:49.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:49.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:49.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:49.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:49.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:49.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:49.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:49.107+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:50.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:49.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:50.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:49.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:50.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:49.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:50.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:49.672+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:50.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:50.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:50.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:50.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:50.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:50.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:50.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:50.067+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:50.343 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:50.331+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:50.343 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:50.331+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:50.343 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:50.331+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:50.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:50.332+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:51.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:50.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:51.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:50.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:51.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:50.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:51.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:50.660+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:51.303 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:51.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:51.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:51.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:51.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:51.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:51.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:51.036+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:51.642 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:21:51.414+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 10219 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:21:51.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:51.293+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:51.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:51.293+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:51.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:51.293+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:51.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:51.294+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:51.996 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:51.633+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:51.996 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:51.633+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:51.996 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:51.633+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:51.996 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:51.633+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:52.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:52.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:52.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:52.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:52.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:52.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:52.279 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:52.023+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:52.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:52.295+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:52.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:52.295+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:52.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:52.295+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:52.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:52.296+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:53.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:52.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:53.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:52.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:53.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:52.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:53.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:52.640+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:53.311 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:53.043+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:53.311 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:53.043+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:53.311 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:53.043+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:53.311 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:53.044+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:53.590 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:53.300+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:53.591 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:53.300+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:53.591 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:53.300+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:53.591 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:53.300+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:54.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:53.613+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:54.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:53.613+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:54.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:53.613+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:54.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:53.613+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:54.325 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:54.052+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:54.325 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:54.052+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:54.325 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:54.052+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:54.325 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:54.053+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:54.599 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:54.315+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:54.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:54.315+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:54.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:54.315+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:54.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:54.315+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:55.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:54.589+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:55.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:54.589+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:55.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:54.589+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:55.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:54.590+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:55.330 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:55.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:55.330 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:55.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:55.330 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:55.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:55.330 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:55.073+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:55.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:55.543+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:55.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:55.543+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:55.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:55.543+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:55.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:55.544+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:55.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:55.320+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:55.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:55.320+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:55.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:55.320+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:55.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:55.320+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:56.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:56.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:56.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:56.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:56.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:56.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:56.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:56.122+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:56.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:56.350+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:56.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:56.350+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:56.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:56.350+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:56.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:56.350+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:56.779 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:21:56.416+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 10317 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:21:56.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:56.536+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:56.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:56.536+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:56.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:56.536+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:56.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:56.537+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:57.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:57.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:57.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:57.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:57.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:57.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:57.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:57.324+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:57.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:57.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:57.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:57.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:57.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:57.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:57.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:57.166+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:58.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:57.558+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:58.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:57.558+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:58.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:57.558+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:58.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:57.558+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:58.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:58.345+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:58.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:58.345+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:58.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:58.345+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:58.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:58.346+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:58.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:58.137+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:58.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:58.137+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:58.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:58.137+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:58.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:58.138+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:59.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:58.599+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:59.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:58.599+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:59.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:58.599+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:21:59.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:58.600+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:21:59.528 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:59.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:59.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:59.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:59.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:59.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:21:59.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:21:59.170+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:21:59.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:59.300+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:59.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:59.300+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:59.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:59.300+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:21:59.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:21:59.300+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:00.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:59.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:00.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:59.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:00.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:59.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:00.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:21:59.636+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:00.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:00.271+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:00.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:00.271+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:00.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:00.271+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:00.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:00.272+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:00.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:00.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:00.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:00.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:00.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:00.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:00.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:00.198+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:01.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:00.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:01.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:00.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:01.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:00.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:01.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:00.604+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:01.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:01.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:01.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:01.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:01.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:01.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:01.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:01.154+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:01.429 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:01.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:01.429 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:01.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:01.429 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:01.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:01.429 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:01.231+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:01.778 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:22:01.418+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 10413 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:22:01.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:01.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:01.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:01.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:01.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:01.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:01.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:01.644+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:02.528 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:02.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:02.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:02.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:02.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:02.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:02.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:02.154+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:02.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:02.229+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:02.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:02.229+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:02.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:02.230+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:02.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:02.230+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:03.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:02.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:03.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:02.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:03.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:02.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:03.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:02.609+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:03.528 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:03.129+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:03.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:03.129+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:03.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:03.129+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:03.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:03.129+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:03.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:03.278+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:03.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:03.278+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:03.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:03.278+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:03.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:03.278+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:04.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:03.637+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:04.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:03.637+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:04.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:03.637+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:04.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:03.638+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:04.528 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:04.150+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:04.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:04.150+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:04.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:04.150+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:04.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:04.150+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:04.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:04.281+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:04.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:04.281+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:04.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:04.281+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:04.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:04.281+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:05.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:04.595+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:05.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:04.595+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:05.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:04.595+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:05.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:04.596+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:05.528 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:05.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:05.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:05.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:05.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:05.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:05.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:05.191+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:05.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:05.295+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:05.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:05.295+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:05.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:05.295+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:05.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:05.295+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:06.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:05.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:06.036 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:05.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:06.036 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:05.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:06.037 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:05.630+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:06.430 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:06.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:06.430 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:06.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:06.430 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:06.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:06.430 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:06.156+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:06.430 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:06.326+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:06.431 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:06.326+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:06.431 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:06.326+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:06.431 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:06.326+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:06.692 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:22:06.419+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 10510 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:22:06.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:06.659+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:06.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:06.659+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:06.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:06.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:06.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:06.660+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:07.528 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:07.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:07.528 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:07.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:07.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:07.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:07.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:07.160+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:07.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:07.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:07.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:07.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:07.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:07.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:07.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:07.325+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:08.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:07.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:08.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:07.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:08.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:07.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:08.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:07.677+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:08.528 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:08.192+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:08.528 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:08.192+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:08.528 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:08.192+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:08.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:08.193+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:08.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:08.298+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:08.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:08.298+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:08.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:08.298+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:08.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:08.299+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:09.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:08.710+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:09.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:08.710+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:09.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:08.710+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:09.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:08.711+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:09.528 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:09.215+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:09.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:09.215+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:09.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:09.215+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:09.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:09.216+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:09.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:09.340+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:09.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:09.340+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:09.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:09.340+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:09.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:09.340+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:10.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:09.737+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:10.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:09.737+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:10.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:09.737+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:10.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:09.737+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:10.428 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:10.319+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:10.428 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:10.319+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:10.428 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:10.319+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:10.428 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:10.319+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:10.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:10.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:10.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:10.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:10.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:10.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:10.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:10.258+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:11.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:10.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:11.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:10.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:11.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:10.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:11.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:10.776+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:11.528 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:22:11.421+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 10605 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:22:11.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:11.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:11.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:11.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:11.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:11.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:11.529 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:11.312+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:11.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:11.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:11.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:11.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:11.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:11.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:11.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:11.263+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:12.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:11.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:12.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:11.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:12.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:11.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:12.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:11.733+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:12.698 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:12.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:12.698 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:12.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:12.698 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:12.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:12.699 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:12.292+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:12.699 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:12.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:12.699 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:12.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:12.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:12.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:12.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:12.325+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:13.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:12.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:13.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:12.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:13.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:12.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:13.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:12.689+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:13.589 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:13.327+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:13.590 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:13.327+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:13.590 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:13.327+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:13.590 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:13.327+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:13.590 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:13.336+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:13.590 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:13.336+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:13.590 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:13.336+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:13.591 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:13.337+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:14.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:13.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:14.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:13.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:14.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:13.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:14.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:13.687+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:14.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:14.354+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:14.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:14.354+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:14.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:14.354+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:14.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:14.355+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:14.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:14.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:14.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:14.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:14.728 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:14.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:14.728 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:14.357+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:15.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:14.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:15.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:14.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:15.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:14.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:15.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:14.717+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:15.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:15.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:15.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:15.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:15.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:15.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:15.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:15.348+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:15.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:15.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:15.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:15.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:15.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:15.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:15.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:15.405+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:16.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:15.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:16.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:15.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:16.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:15.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:16.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:15.747+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:16.763 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:22:16.422+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 10703 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:22:16.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:16.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:16.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:16.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:16.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:16.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:16.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:16.353+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:16.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:16.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:16.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:16.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:16.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:16.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:16.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:16.454+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:17.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:16.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:17.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:16.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:17.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:16.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:17.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:16.754+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:17.741 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:17.375+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:17.741 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:17.375+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:17.741 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:17.375+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:17.741 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:17.376+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:17.741 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:17.442+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:17.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:17.442+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:17.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:17.442+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:17.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:17.442+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:18.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:17.730+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:18.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:17.730+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:18.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:17.730+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:18.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:17.731+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:18.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:18.390+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:18.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:18.390+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:18.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:18.390+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:18.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:18.391+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:18.776 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:18.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:18.777 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:18.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:18.777 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:18.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:18.777 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:18.432+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:19.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:18.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:19.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:18.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:19.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:18.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:19.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:18.766+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:19.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:19.394+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:19.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:19.394+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:19.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:19.394+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:19.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:19.396+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:19.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:19.383+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:19.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:19.383+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:19.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:19.383+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:19.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:19.383+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:20.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:19.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:20.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:19.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:20.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:19.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:20.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:19.796+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:20.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:20.423+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:20.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:20.423+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:20.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:20.423+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:20.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:20.424+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:20.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:20.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:20.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:20.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:20.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:20.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:20.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:20.405+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:21.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:20.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:21.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:20.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:21.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:20.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:21.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:20.842+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:21.778 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:22:21.423+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 10799 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:22:21.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:21.430+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:21.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:21.430+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:21.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:21.430+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:21.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:21.432+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:21.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:21.375+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:21.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:21.375+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:21.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:21.375+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:21.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:21.375+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:22.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:21.820+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:22.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:21.820+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:22.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:21.820+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:22.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:21.821+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:22.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:22.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:22.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:22.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:22.473+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:22.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:22.402+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:22.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:22.402+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:22.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:22.402+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:22.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:22.402+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:23.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:22.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:23.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:22.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:23.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:22.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:23.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:22.818+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:23.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:23.414+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:23.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:23.414+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:23.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:23.414+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:23.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:23.415+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:23.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:23.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:23.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:23.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:23.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:23.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:23.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:23.489+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:24.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:23.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:24.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:23.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:24.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:23.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:24.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:23.862+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:24.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:24.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:24.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:24.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:24.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:24.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:24.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:24.441+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:24.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:24.500+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:24.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:24.500+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:24.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:24.500+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:24.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:24.501+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:25.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:24.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:25.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:24.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:25.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:24.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:25.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:24.875+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:25.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:25.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:25.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:25.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:25.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:25.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:25.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:25.472+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:25.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:25.530+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:25.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:25.530+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:25.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:25.530+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:25.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:25.531+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:26.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:25.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:26.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:25.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:26.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:25.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:26.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:25.902+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:26.558 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:22:26.425+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 10896 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:22:26.558 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:26.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:26.558 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:26.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:26.558 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:26.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:26.558 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:26.432+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:26.868 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:26.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:26.868 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:26.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:26.868 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:26.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:26.868 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:26.577+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:27.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:26.858+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:27.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:26.858+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:27.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:26.858+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:27.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:26.859+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:27.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:27.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:27.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:27.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:27.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:27.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:27.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:27.429+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:27.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:27.622+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:27.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:27.622+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:27.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:27.622+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:27.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:27.623+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:28.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:27.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:28.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:27.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:28.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:27.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:28.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:27.834+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:28.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:28.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:28.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:28.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:28.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:28.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:28.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:28.446+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:28.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:28.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:28.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:28.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:28.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:28.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:28.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:28.608+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:29.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:28.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:29.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:28.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:29.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:28.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:29.278 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:28.797+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:29.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:29.566+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:29.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:29.566+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:29.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:29.566+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:29.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:29.567+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:29.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:29.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:29.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:29.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:29.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:29.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:29.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:29.490+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:30.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:29.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:30.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:29.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:30.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:29.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:30.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:29.754+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:31.028 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:30.539+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:31.028 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:30.539+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:31.028 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:30.539+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:31.028 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:30.539+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:31.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:30.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:31.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:30.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:31.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:30.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:31.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:30.581+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:31.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:30.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:31.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:30.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:31.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:30.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:31.030 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:30.743+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:31.726 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:22:31.426+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 10991 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:22:31.726 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:31.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:31.726 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:31.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:31.726 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:31.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:31.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:31.540+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:31.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:31.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:31.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:31.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:31.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:31.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:31.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:31.540+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:32.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:31.716+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:32.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:31.716+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:32.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:31.716+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:32.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:31.717+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:32.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:32.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:32.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:32.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:32.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:32.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:32.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:32.510+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:32.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:32.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:32.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:32.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:32.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:32.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:32.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:32.508+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:32.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:32.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:32.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:32.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:32.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:32.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:32.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:32.690+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:33.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:33.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:33.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:33.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:33.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:33.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:33.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:33.487+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:33.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:33.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:33.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:33.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:33.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:33.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:33.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:33.720+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:33.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:33.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:33.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:33.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:33.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:33.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:33.780 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:33.528+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:34.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:34.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:34.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:34.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:34.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:34.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:34.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:34.508+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:34.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:34.509+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:34.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:34.509+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:34.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:34.509+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:34.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:34.509+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:34.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:34.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:34.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:34.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:34.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:34.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:34.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:34.748+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:35.777 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:35.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:35.777 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:35.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:35.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:35.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:35.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:35.512+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:35.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:35.508+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:35.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:35.508+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:35.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:35.508+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:35.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:35.508+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:36.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:35.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:36.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:35.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:36.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:35.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:36.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:35.768+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:36.728 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:22:36.428+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 11088 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:22:36.729 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:36.547+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:36.729 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:36.547+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:36.729 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:36.547+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:36.729 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:36.547+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:36.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:36.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:36.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:36.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:36.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:36.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:36.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:36.479+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:37.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:36.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:37.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:36.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:37.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:36.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:37.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:36.720+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:37.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:37.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:37.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:37.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:37.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:37.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:37.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:37.546+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:37.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:37.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:37.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:37.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:37.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:37.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:37.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:37.441+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:38.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:37.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:38.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:37.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:38.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:37.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:38.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:37.692+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:38.728 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:38.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:38.728 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:38.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:38.728 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:38.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:38.728 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:38.527+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:38.729 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:38.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:38.729 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:38.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:38.729 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:38.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:38.729 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:38.401+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:39.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:38.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:39.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:38.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:39.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:38.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:39.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:38.719+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:39.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:39.548+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:39.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:39.548+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:39.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:39.548+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:39.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:39.549+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:39.773 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:39.448+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:39.773 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:39.448+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:39.773 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:39.448+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:39.773 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:39.448+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:40.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:39.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:40.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:39.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:40.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:39.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:40.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:39.763+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:40.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:40.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:40.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:40.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:40.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:40.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:40.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:40.441+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:40.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:40.522+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:40.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:40.522+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:40.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:40.522+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:40.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:40.523+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:41.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:40.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:41.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:40.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:41.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:40.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:41.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:40.753+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:41.735 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:22:41.429+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 11182 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:22:41.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:41.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:41.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:41.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:41.736 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:41.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:41.736 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:41.506+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:41.736 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:41.467+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:41.736 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:41.467+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:41.736 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:41.467+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:41.736 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:41.467+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:42.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:41.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:42.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:41.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:42.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:41.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:42.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:41.727+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:42.767 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:42.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:42.767 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:42.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:42.767 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:42.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:42.767 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:42.474+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:42.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:42.540+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:42.768 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:42.540+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:42.768 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:42.540+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:42.768 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:42.540+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:43.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:42.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:43.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:42.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:43.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:42.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:43.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:42.758+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:43.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:43.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:43.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:43.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:43.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:43.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:43.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:43.582+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:43.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:43.450+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:43.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:43.450+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:43.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:43.450+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:43.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:43.450+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:44.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:43.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:44.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:43.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:44.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:43.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:44.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:43.753+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:44.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:44.608+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:44.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:44.608+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:44.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:44.608+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:44.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:44.609+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:44.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:44.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:44.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:44.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:44.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:44.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:44.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:44.486+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:45.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:44.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:45.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:44.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:45.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:44.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:45.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:44.749+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:46.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:45.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:46.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:45.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:46.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:45.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:46.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:45.758+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:46.029 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:45.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:46.029 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:45.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:46.029 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:45.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:46.029 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:45.528+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:46.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:45.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:46.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:45.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:46.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:45.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:46.030 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:45.602+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:46.755 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:22:46.430+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 11278 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:22:46.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:46.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:46.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:46.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:46.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:46.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:46.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:46.555+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:46.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:46.501+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:46.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:46.501+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:46.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:46.501+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:46.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:46.501+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:47.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:46.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:47.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:46.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:47.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:46.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:47.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:46.747+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:47.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:47.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:47.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:47.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:47.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:47.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:47.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:47.591+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:47.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:47.477+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:47.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:47.477+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:47.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:47.477+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:47.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:47.478+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:48.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:47.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:48.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:47.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:48.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:47.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:48.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:47.763+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:48.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:48.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:48.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:48.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:48.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:48.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:48.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:48.598+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:48.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:48.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:48.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:48.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:48.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:48.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:48.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:48.453+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:49.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:48.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:49.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:48.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:49.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:48.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:49.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:48.761+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:49.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:49.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:49.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:49.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:49.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:49.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:49.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:49.418+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:49.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:49.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:49.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:49.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:49.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:49.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:49.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:49.580+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:50.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:49.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:50.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:49.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:50.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:49.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:50.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:49.760+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:50.737 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:50.620+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:50.737 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:50.620+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:50.738 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:50.620+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:50.738 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:50.621+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:50.738 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:50.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:50.738 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:50.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:50.738 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:50.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:50.738 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:50.449+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:51.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:50.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:51.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:50.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:51.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:50.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:51.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:50.728+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:51.717 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:22:51.431+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 11372 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:22:51.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:51.605+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:51.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:51.605+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:51.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:51.605+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:51.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:51.606+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:51.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:51.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:51.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:51.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:51.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:51.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:51.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:51.415+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:52.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:51.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:52.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:51.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:52.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:51.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:52.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:51.708+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:52.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:52.682+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:52.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:52.682+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:52.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:52.682+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:52.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:52.683+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:52.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:52.437+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:52.778 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:52.437+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:52.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:52.437+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:52.779 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:52.438+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:52.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:52.574+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:52.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:52.574+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:52.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:52.574+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:52.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:52.575+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:53.726 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:53.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:53.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:53.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:53.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:53.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:53.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:53.577+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:53.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:53.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:53.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:53.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:53.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:53.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:53.728 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:53.472+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:54.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:53.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:54.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:53.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:54.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:53.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:54.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:53.718+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:54.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:54.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:54.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:54.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:54.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:54.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:54.734 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:54.540+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:54.734 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:54.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:54.734 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:54.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:54.734 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:54.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:54.734 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:54.460+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:55.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:54.724+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:55.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:54.724+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:55.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:54.724+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:55.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:54.724+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:55.707 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:55.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:55.707 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:55.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:55.707 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:55.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:55.707 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:55.503+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:55.708 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:55.433+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:55.708 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:55.433+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:55.708 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:55.433+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:55.708 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:55.434+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:56.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:55.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:56.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:55.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:56.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:55.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:56.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:55.698+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:56.717 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:22:56.433+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 11469 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:22:56.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:56.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:56.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:56.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:56.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:56.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:56.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:56.413+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:56.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:56.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:56.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:56.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:56.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:56.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:56.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:56.540+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:57.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:56.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:57.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:56.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:57.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:56.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:57.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:56.709+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:57.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:57.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:57.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:57.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:57.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:57.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:57.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:57.376+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:58.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:57.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:58.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:57.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:58.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:57.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:58.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:57.715+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:58.028 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:57.551+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:58.028 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:57.551+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:58.028 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:57.551+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:58.028 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:57.552+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:58.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:58.350+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:58.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:58.350+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:58.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:58.350+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:58.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:58.351+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:59.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:58.667+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:59.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:58.667+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:59.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:58.667+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:22:59.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:58.668+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:22:59.028 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:58.601+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:59.028 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:58.601+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:59.028 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:58.601+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:59.028 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:58.602+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:59.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:59.335+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:59.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:59.335+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:59.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:59.335+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:22:59.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:22:59.335+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:22:59.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:59.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:59.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:59.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:59.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:59.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:22:59.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:22:59.569+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:00.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:59.627+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:00.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:59.627+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:00.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:59.627+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:00.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:22:59.627+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:00.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:00.328+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:00.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:00.328+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:00.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:00.328+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:00.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:00.329+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:01.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:00.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:01.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:00.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:01.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:00.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:01.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:00.672+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:01.028 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:00.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:01.028 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:00.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:01.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:00.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:01.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:00.523+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:01.673 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:23:01.435+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 11564 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:23:01.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:01.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:01.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:01.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:01.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:01.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:01.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:01.554+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:01.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:01.311+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:01.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:01.311+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:01.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:01.311+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:01.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:01.312+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:02.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:01.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:02.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:01.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:02.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:01.665+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:02.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:01.665+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:02.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:02.316+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:02.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:02.316+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:02.644 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:02.316+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:02.644 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:02.317+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:02.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:02.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:02.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:02.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:02.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:02.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:02.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:02.518+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:03.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:02.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:03.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:02.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:03.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:02.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:03.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:02.635+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:03.589 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:03.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:03.589 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:03.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:03.589 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:03.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:03.589 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:03.525+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:03.589 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:03.278+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:03.590 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:03.278+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:03.590 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:03.278+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:03.590 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:03.278+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:04.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:03.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:04.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:03.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:04.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:03.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:04.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:03.612+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:04.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:04.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:04.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:04.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:04.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:04.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:04.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:04.531+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:04.541 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:04.281+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:04.542 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:04.281+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:04.542 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:04.281+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:04.542 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:04.282+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:05.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:04.605+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:05.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:04.605+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:05.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:04.605+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:05.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:04.605+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:05.577 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:05.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:05.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:05.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:05.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:05.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:05.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:05.519+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:05.578 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:05.272+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:05.578 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:05.272+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:05.578 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:05.272+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:05.579 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:05.272+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:06.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:05.568+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:06.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:05.568+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:06.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:05.568+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:06.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:05.569+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:06.527 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:23:06.436+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 11660 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:23:06.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:06.266+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:06.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:06.266+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:06.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:06.266+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:06.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:06.266+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:06.528 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:06.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:06.528 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:06.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:06.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:06.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:06.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:06.493+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:07.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:06.567+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:07.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:06.567+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:07.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:06.567+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:07.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:06.567+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:07.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:07.220+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:07.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:07.220+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:07.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:07.220+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:07.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:07.221+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:07.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:07.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:07.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:07.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:07.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:07.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:07.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:07.539+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:07.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:07.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:07.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:07.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:07.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:07.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:07.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:07.499+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:08.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:08.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:08.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:08.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:08.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:08.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:08.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:08.213+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:08.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:08.491+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:08.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:08.491+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:08.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:08.491+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:08.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:08.492+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:08.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:08.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:08.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:08.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:08.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:08.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:08.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:08.525+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:09.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:09.207+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:09.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:09.207+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:09.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:09.207+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:09.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:09.208+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:09.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:09.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:09.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:09.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:09.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:09.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:09.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:09.461+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:09.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:09.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:09.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:09.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:09.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:09.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:09.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:09.515+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:10.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:10.186+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:10.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:10.186+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:10.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:10.186+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:10.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:10.187+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:10.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:10.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:10.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:10.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:10.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:10.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:10.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:10.496+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:10.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:10.459+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:10.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:10.459+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:10.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:10.459+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:10.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:10.460+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-19T19:23:10.792 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-02-19T19:23:10.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:23:10.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-02-19T19:23:10.858 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip: Cannot mkdir: Input/output error 2024-02-19T19:23:10.858 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip6-dstaddr: Cannot open: Input/output error 2024-02-19T19:23:10.858 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip6-nexthdr: Cannot open: Input/output error 2024-02-19T19:23:10.859 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-protcol: Cannot open: Input/output error 2024-02-19T19:23:10.859 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-checksum: Cannot open: Input/output error 2024-02-19T19:23:10.859 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip6-version: Cannot open: Input/output error 2024-02-19T19:23:10.859 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-ihl: Cannot open: Input/output error 2024-02-19T19:23:10.859 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.859 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-fragment: Cannot open: Input/output error 2024-02-19T19:23:10.859 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-totlen: Cannot open: Input/output error 2024-02-19T19:23:10.859 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.859 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-version: Cannot open: Input/output error 2024-02-19T19:23:10.860 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip6-plen: Cannot open: Input/output error 2024-02-19T19:23:10.860 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/broken_ip/broken_ip4-dstaddr: Cannot open: Input/output error 2024-02-19T19:23:10.860 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast: Cannot mkdir: Input/output error 2024-02-19T19:23:10.860 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation: Cannot mkdir: Input/output error 2024-02-19T19:23:10.860 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast6-grpope01: Cannot open: Input/output error 2024-02-19T19:23:10.860 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast6-grpope02: Cannot open: Input/output error 2024-02-19T19:23:10.860 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast4-grpope03: Cannot open: Input/output error 2024-02-19T19:23:10.860 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.860 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.861 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast4-grpope04: Cannot open: Input/output error 2024-02-19T19:23:10.861 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast6-grpope03: Cannot open: Input/output error 2024-02-19T19:23:10.861 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast4-grpope01: Cannot open: Input/output error 2024-02-19T19:23:10.861 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast4-grpope02: Cannot open: Input/output error 2024-02-19T19:23:10.861 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/grp-operation/mcast6-grpope04: Cannot open: Input/output error 2024-02-19T19:23:10.861 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood: Cannot mkdir: Input/output error 2024-02-19T19:23:10.861 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld01: Cannot open: Input/output error 2024-02-19T19:23:10.861 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld03: Cannot open: Input/output error 2024-02-19T19:23:10.861 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld05: Cannot open: Input/output error 2024-02-19T19:23:10.861 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld02: Cannot open: Input/output error 2024-02-19T19:23:10.862 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld05: Cannot open: Input/output error 2024-02-19T19:23:10.862 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld01: Cannot open: Input/output error 2024-02-19T19:23:10.862 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld02: Cannot open: Input/output error 2024-02-19T19:23:10.862 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.862 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.862 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld06: Cannot open: Input/output error 2024-02-19T19:23:10.862 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld03: Cannot open: Input/output error 2024-02-19T19:23:10.862 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld04: Cannot open: Input/output error 2024-02-19T19:23:10.862 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast4-queryfld06: Cannot open: Input/output error 2024-02-19T19:23:10.862 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/query-flood/mcast6-queryfld04: Cannot open: Input/output error 2024-02-19T19:23:10.863 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood: Cannot mkdir: Input/output error 2024-02-19T19:23:10.863 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/mcast6-pktfld02: Cannot open: Input/output error 2024-02-19T19:23:10.863 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.863 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/mcast6-pktfld01: Cannot open: Input/output error 2024-02-19T19:23:10.863 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.863 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/mcast4-pktfld02: Cannot open: Input/output error 2024-02-19T19:23:10.863 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/packet-flood/mcast4-pktfld01: Cannot open: Input/output error 2024-02-19T19:23:10.863 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/multicast/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.863 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/route: Cannot mkdir: Input/output error 2024-02-19T19:23:10.864 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-redirect: Cannot open: Input/output error 2024-02-19T19:23:10.864 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-change-if: Cannot open: Input/output error 2024-02-19T19:23:10.864 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/route/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.864 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/route/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.864 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-change-gw: Cannot open: Input/output error 2024-02-19T19:23:10.864 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-ifdown: Cannot open: Input/output error 2024-02-19T19:23:10.864 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-change-if: Cannot open: Input/output error 2024-02-19T19:23:10.864 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-change-dst: Cannot open: Input/output error 2024-02-19T19:23:10.864 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-change-dst: Cannot open: Input/output error 2024-02-19T19:23:10.865 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-change-gw: Cannot open: Input/output error 2024-02-19T19:23:10.865 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-ifdown: Cannot open: Input/output error 2024-02-19T19:23:10.865 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route4-rmmod: Cannot open: Input/output error 2024-02-19T19:23:10.865 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-redirect: Cannot open: Input/output error 2024-02-19T19:23:10.865 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/route/route6-rmmod: Cannot open: Input/output error 2024-02-19T19:23:10.865 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.865 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp: Cannot mkdir: Input/output error 2024-02-19T19:23:10.865 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff: Cannot mkdir: Input/output error 2024-02-19T19:23:10.865 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff14: Cannot open: Input/output error 2024-02-19T19:23:10.866 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff06: Cannot open: Input/output error 2024-02-19T19:23:10.866 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff09: Cannot open: Input/output error 2024-02-19T19:23:10.866 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff02: Cannot open: Input/output error 2024-02-19T19:23:10.866 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff13: Cannot open: Input/output error 2024-02-19T19:23:10.866 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff01: Cannot open: Input/output error 2024-02-19T19:23:10.866 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff09: Cannot open: Input/output error 2024-02-19T19:23:10.867 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff03: Cannot open: Input/output error 2024-02-19T19:23:10.867 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff04: Cannot open: Input/output error 2024-02-19T19:23:10.867 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff10: Cannot open: Input/output error 2024-02-19T19:23:10.867 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff05: Cannot open: Input/output error 2024-02-19T19:23:10.867 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff04: Cannot open: Input/output error 2024-02-19T19:23:10.867 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.867 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.867 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff07: Cannot open: Input/output error 2024-02-19T19:23:10.867 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff11: Cannot open: Input/output error 2024-02-19T19:23:10.868 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff12: Cannot open: Input/output error 2024-02-19T19:23:10.868 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff05: Cannot open: Input/output error 2024-02-19T19:23:10.868 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff10: Cannot open: Input/output error 2024-02-19T19:23:10.868 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff07: Cannot open: Input/output error 2024-02-19T19:23:10.868 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff11: Cannot open: Input/output error 2024-02-19T19:23:10.868 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff12: Cannot open: Input/output error 2024-02-19T19:23:10.868 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff13: Cannot open: Input/output error 2024-02-19T19:23:10.868 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff08: Cannot open: Input/output error 2024-02-19T19:23:10.868 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff14: Cannot open: Input/output error 2024-02-19T19:23:10.869 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff03: Cannot open: Input/output error 2024-02-19T19:23:10.869 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff02: Cannot open: Input/output error 2024-02-19T19:23:10.869 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp4-uni-dsackoff08: Cannot open: Input/output error 2024-02-19T19:23:10.869 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff06: Cannot open: Input/output error 2024-02-19T19:23:10.869 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-dsackoff/tcp6-uni-dsackoff01: Cannot open: Input/output error 2024-02-19T19:23:10.869 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport: Cannot mkdir: Input/output error 2024-02-19T19:23:10.869 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport04: Cannot open: Input/output error 2024-02-19T19:23:10.869 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport03: Cannot open: Input/output error 2024-02-19T19:23:10.869 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport12: Cannot open: Input/output error 2024-02-19T19:23:10.870 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport07: Cannot open: Input/output error 2024-02-19T19:23:10.870 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport05: Cannot open: Input/output error 2024-02-19T19:23:10.870 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport04: Cannot open: Input/output error 2024-02-19T19:23:10.870 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport14: Cannot open: Input/output error 2024-02-19T19:23:10.870 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport10: Cannot open: Input/output error 2024-02-19T19:23:10.870 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport02: Cannot open: Input/output error 2024-02-19T19:23:10.870 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport05: Cannot open: Input/output error 2024-02-19T19:23:10.870 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.870 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.871 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport06: Cannot open: Input/output error 2024-02-19T19:23:10.871 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport01: Cannot open: Input/output error 2024-02-19T19:23:10.871 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport09: Cannot open: Input/output error 2024-02-19T19:23:10.871 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport13: Cannot open: Input/output error 2024-02-19T19:23:10.871 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport10: Cannot open: Input/output error 2024-02-19T19:23:10.871 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport07: Cannot open: Input/output error 2024-02-19T19:23:10.871 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport02: Cannot open: Input/output error 2024-02-19T19:23:10.871 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport08: Cannot open: Input/output error 2024-02-19T19:23:10.871 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport12: Cannot open: Input/output error 2024-02-19T19:23:10.871 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport01: Cannot open: Input/output error 2024-02-19T19:23:10.872 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport08: Cannot open: Input/output error 2024-02-19T19:23:10.872 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport03: Cannot open: Input/output error 2024-02-19T19:23:10.872 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport11: Cannot open: Input/output error 2024-02-19T19:23:10.872 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport13: Cannot open: Input/output error 2024-02-19T19:23:10.872 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport11: Cannot open: Input/output error 2024-02-19T19:23:10.872 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp6-multi-diffport14: Cannot open: Input/output error 2024-02-19T19:23:10.872 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport06: Cannot open: Input/output error 2024-02-19T19:23:10.872 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffport/tcp4-multi-diffport09: Cannot open: Input/output error 2024-02-19T19:23:10.873 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic: Cannot mkdir: Input/output error 2024-02-19T19:23:10.873 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic04: Cannot open: Input/output error 2024-02-19T19:23:10.873 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic06: Cannot open: Input/output error 2024-02-19T19:23:10.873 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic13: Cannot open: Input/output error 2024-02-19T19:23:10.873 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic14: Cannot open: Input/output error 2024-02-19T19:23:10.873 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic06: Cannot open: Input/output error 2024-02-19T19:23:10.873 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic11: Cannot open: Input/output error 2024-02-19T19:23:10.873 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic05: Cannot open: Input/output error 2024-02-19T19:23:10.873 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.874 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic01: Cannot open: Input/output error 2024-02-19T19:23:10.874 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic12: Cannot open: Input/output error 2024-02-19T19:23:10.874 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic11: Cannot open: Input/output error 2024-02-19T19:23:10.874 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.874 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic10: Cannot open: Input/output error 2024-02-19T19:23:10.874 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic07: Cannot open: Input/output error 2024-02-19T19:23:10.874 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic10: Cannot open: Input/output error 2024-02-19T19:23:10.874 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic08: Cannot open: Input/output error 2024-02-19T19:23:10.874 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic13: Cannot open: Input/output error 2024-02-19T19:23:10.875 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic03: Cannot open: Input/output error 2024-02-19T19:23:10.875 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic08: Cannot open: Input/output error 2024-02-19T19:23:10.875 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic02: Cannot open: Input/output error 2024-02-19T19:23:10.875 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic09: Cannot open: Input/output error 2024-02-19T19:23:10.875 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic03: Cannot open: Input/output error 2024-02-19T19:23:10.875 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic01: Cannot open: Input/output error 2024-02-19T19:23:10.875 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic04: Cannot open: Input/output error 2024-02-19T19:23:10.875 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic07: Cannot open: Input/output error 2024-02-19T19:23:10.875 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic09: Cannot open: Input/output error 2024-02-19T19:23:10.876 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic02: Cannot open: Input/output error 2024-02-19T19:23:10.876 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic12: Cannot open: Input/output error 2024-02-19T19:23:10.876 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp6-uni-basic14: Cannot open: Input/output error 2024-02-19T19:23:10.876 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-basic/tcp4-uni-basic05: Cannot open: Input/output error 2024-02-19T19:23:10.876 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport: Cannot mkdir: Input/output error 2024-02-19T19:23:10.876 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport07: Cannot open: Input/output error 2024-02-19T19:23:10.876 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport14: Cannot open: Input/output error 2024-02-19T19:23:10.876 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport11: Cannot open: Input/output error 2024-02-19T19:23:10.876 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport06: Cannot open: Input/output error 2024-02-19T19:23:10.877 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport09: Cannot open: Input/output error 2024-02-19T19:23:10.877 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport02: Cannot open: Input/output error 2024-02-19T19:23:10.877 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport04: Cannot open: Input/output error 2024-02-19T19:23:10.877 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport01: Cannot open: Input/output error 2024-02-19T19:23:10.877 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.877 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport08: Cannot open: Input/output error 2024-02-19T19:23:10.877 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport12: Cannot open: Input/output error 2024-02-19T19:23:10.877 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.877 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport09: Cannot open: Input/output error 2024-02-19T19:23:10.878 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport05: Cannot open: Input/output error 2024-02-19T19:23:10.878 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport13: Cannot open: Input/output error 2024-02-19T19:23:10.878 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport05: Cannot open: Input/output error 2024-02-19T19:23:10.878 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport08: Cannot open: Input/output error 2024-02-19T19:23:10.878 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport02: Cannot open: Input/output error 2024-02-19T19:23:10.878 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport14: Cannot open: Input/output error 2024-02-19T19:23:10.878 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport12: Cannot open: Input/output error 2024-02-19T19:23:10.878 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport11: Cannot open: Input/output error 2024-02-19T19:23:10.879 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport10: Cannot open: Input/output error 2024-02-19T19:23:10.879 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport03: Cannot open: Input/output error 2024-02-19T19:23:10.879 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport04: Cannot open: Input/output error 2024-02-19T19:23:10.879 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport13: Cannot open: Input/output error 2024-02-19T19:23:10.879 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport06: Cannot open: Input/output error 2024-02-19T19:23:10.879 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport07: Cannot open: Input/output error 2024-02-19T19:23:10.879 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport10: Cannot open: Input/output error 2024-02-19T19:23:10.879 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp4-multi-sameport01: Cannot open: Input/output error 2024-02-19T19:23:10.879 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-sameport/tcp6-multi-sameport03: Cannot open: Input/output error 2024-02-19T19:23:10.880 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.880 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend: Cannot mkdir: Input/output error 2024-02-19T19:23:10.880 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend12: Cannot open: Input/output error 2024-02-19T19:23:10.880 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend09: Cannot open: Input/output error 2024-02-19T19:23:10.880 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend03: Cannot open: Input/output error 2024-02-19T19:23:10.880 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend06: Cannot open: Input/output error 2024-02-19T19:23:10.880 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend11: Cannot open: Input/output error 2024-02-19T19:23:10.880 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend06: Cannot open: Input/output error 2024-02-19T19:23:10.880 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend07: Cannot open: Input/output error 2024-02-19T19:23:10.881 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend04: Cannot open: Input/output error 2024-02-19T19:23:10.881 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend11: Cannot open: Input/output error 2024-02-19T19:23:10.881 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend01: Cannot open: Input/output error 2024-02-19T19:23:10.881 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.881 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend10: Cannot open: Input/output error 2024-02-19T19:23:10.881 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend02: Cannot open: Input/output error 2024-02-19T19:23:10.881 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.881 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend04: Cannot open: Input/output error 2024-02-19T19:23:10.881 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend14: Cannot open: Input/output error 2024-02-19T19:23:10.882 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend13: Cannot open: Input/output error 2024-02-19T19:23:10.882 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend12: Cannot open: Input/output error 2024-02-19T19:23:10.882 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend01: Cannot open: Input/output error 2024-02-19T19:23:10.882 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend05: Cannot open: Input/output error 2024-02-19T19:23:10.882 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend14: Cannot open: Input/output error 2024-02-19T19:23:10.882 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend02: Cannot open: Input/output error 2024-02-19T19:23:10.882 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend07: Cannot open: Input/output error 2024-02-19T19:23:10.882 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend08: Cannot open: Input/output error 2024-02-19T19:23:10.882 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend03: Cannot open: Input/output error 2024-02-19T19:23:10.883 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend05: Cannot open: Input/output error 2024-02-19T19:23:10.883 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend10: Cannot open: Input/output error 2024-02-19T19:23:10.883 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend08: Cannot open: Input/output error 2024-02-19T19:23:10.883 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp4-uni-smallsend13: Cannot open: Input/output error 2024-02-19T19:23:10.883 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-smallsend/tcp6-uni-smallsend09: Cannot open: Input/output error 2024-02-19T19:23:10.883 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic: Cannot mkdir: Input/output error 2024-02-19T19:23:10.883 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic04: Cannot open: Input/output error 2024-02-19T19:23:10.883 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic05: Cannot open: Input/output error 2024-02-19T19:23:10.883 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic09: Cannot open: Input/output error 2024-02-19T19:23:10.883 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic01: Cannot open: Input/output error 2024-02-19T19:23:10.884 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic02: Cannot open: Input/output error 2024-02-19T19:23:10.884 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic07: Cannot open: Input/output error 2024-02-19T19:23:10.884 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic03: Cannot open: Input/output error 2024-02-19T19:23:10.884 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic03: Cannot open: Input/output error 2024-02-19T19:23:10.884 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic04: Cannot open: Input/output error 2024-02-19T19:23:10.884 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.884 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic01: Cannot open: Input/output error 2024-02-19T19:23:10.884 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic05: Cannot open: Input/output error 2024-02-19T19:23:10.884 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic14: Cannot open: Input/output error 2024-02-19T19:23:10.885 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.885 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic11: Cannot open: Input/output error 2024-02-19T19:23:10.885 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic12: Cannot open: Input/output error 2024-02-19T19:23:10.885 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic06: Cannot open: Input/output error 2024-02-19T19:23:10.885 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic13: Cannot open: Input/output error 2024-02-19T19:23:10.885 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic08: Cannot open: Input/output error 2024-02-19T19:23:10.885 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic02: Cannot open: Input/output error 2024-02-19T19:23:10.885 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic14: Cannot open: Input/output error 2024-02-19T19:23:10.885 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic06: Cannot open: Input/output error 2024-02-19T19:23:10.886 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic11: Cannot open: Input/output error 2024-02-19T19:23:10.886 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic09: Cannot open: Input/output error 2024-02-19T19:23:10.886 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic07: Cannot open: Input/output error 2024-02-19T19:23:10.886 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic10: Cannot open: Input/output error 2024-02-19T19:23:10.886 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic13: Cannot open: Input/output error 2024-02-19T19:23:10.886 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic08: Cannot open: Input/output error 2024-02-19T19:23:10.886 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp6-multi-diffnic12: Cannot open: Input/output error 2024-02-19T19:23:10.886 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffnic/tcp4-multi-diffnic10: Cannot open: Input/output error 2024-02-19T19:23:10.886 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup: Cannot mkdir: Input/output error 2024-02-19T19:23:10.886 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup09: Cannot open: Input/output error 2024-02-19T19:23:10.887 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup06: Cannot open: Input/output error 2024-02-19T19:23:10.887 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup13: Cannot open: Input/output error 2024-02-19T19:23:10.887 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup10: Cannot open: Input/output error 2024-02-19T19:23:10.887 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup09: Cannot open: Input/output error 2024-02-19T19:23:10.887 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup03: Cannot open: Input/output error 2024-02-19T19:23:10.887 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup01: Cannot open: Input/output error 2024-02-19T19:23:10.887 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup03: Cannot open: Input/output error 2024-02-19T19:23:10.887 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup02: Cannot open: Input/output error 2024-02-19T19:23:10.887 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup11: Cannot open: Input/output error 2024-02-19T19:23:10.888 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup01: Cannot open: Input/output error 2024-02-19T19:23:10.888 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup10: Cannot open: Input/output error 2024-02-19T19:23:10.888 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup14: Cannot open: Input/output error 2024-02-19T19:23:10.888 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.888 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup08: Cannot open: Input/output error 2024-02-19T19:23:10.888 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.888 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup05: Cannot open: Input/output error 2024-02-19T19:23:10.888 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup04: Cannot open: Input/output error 2024-02-19T19:23:10.888 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup13: Cannot open: Input/output error 2024-02-19T19:23:10.889 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup07: Cannot open: Input/output error 2024-02-19T19:23:10.889 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup06: Cannot open: Input/output error 2024-02-19T19:23:10.889 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup12: Cannot open: Input/output error 2024-02-19T19:23:10.889 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup04: Cannot open: Input/output error 2024-02-19T19:23:10.889 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup05: Cannot open: Input/output error 2024-02-19T19:23:10.889 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup12: Cannot open: Input/output error 2024-02-19T19:23:10.889 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup11: Cannot open: Input/output error 2024-02-19T19:23:10.889 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup02: Cannot open: Input/output error 2024-02-19T19:23:10.890 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup14: Cannot open: Input/output error 2024-02-19T19:23:10.890 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp4-uni-pktlossdup08: Cannot open: Input/output error 2024-02-19T19:23:10.890 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-pktlossdup/tcp6-uni-pktlossdup07: Cannot open: Input/output error 2024-02-19T19:23:10.890 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff: Cannot mkdir: Input/output error 2024-02-19T19:23:10.890 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff14: Cannot open: Input/output error 2024-02-19T19:23:10.890 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff02: Cannot open: Input/output error 2024-02-19T19:23:10.890 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff06: Cannot open: Input/output error 2024-02-19T19:23:10.890 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff02: Cannot open: Input/output error 2024-02-19T19:23:10.890 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff06: Cannot open: Input/output error 2024-02-19T19:23:10.890 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff09: Cannot open: Input/output error 2024-02-19T19:23:10.891 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff08: Cannot open: Input/output error 2024-02-19T19:23:10.891 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff11: Cannot open: Input/output error 2024-02-19T19:23:10.891 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff10: Cannot open: Input/output error 2024-02-19T19:23:10.891 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff11: Cannot open: Input/output error 2024-02-19T19:23:10.891 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff09: Cannot open: Input/output error 2024-02-19T19:23:10.891 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff10: Cannot open: Input/output error 2024-02-19T19:23:10.891 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.891 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.891 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff13: Cannot open: Input/output error 2024-02-19T19:23:10.891 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff05: Cannot open: Input/output error 2024-02-19T19:23:10.892 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff07: Cannot open: Input/output error 2024-02-19T19:23:10.892 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff13: Cannot open: Input/output error 2024-02-19T19:23:10.892 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff01: Cannot open: Input/output error 2024-02-19T19:23:10.892 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff14: Cannot open: Input/output error 2024-02-19T19:23:10.892 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff01: Cannot open: Input/output error 2024-02-19T19:23:10.892 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff12: Cannot open: Input/output error 2024-02-19T19:23:10.892 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff03: Cannot open: Input/output error 2024-02-19T19:23:10.892 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff04: Cannot open: Input/output error 2024-02-19T19:23:10.893 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff05: Cannot open: Input/output error 2024-02-19T19:23:10.893 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff03: Cannot open: Input/output error 2024-02-19T19:23:10.893 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff07: Cannot open: Input/output error 2024-02-19T19:23:10.893 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff08: Cannot open: Input/output error 2024-02-19T19:23:10.893 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp6-uni-sackoff12: Cannot open: Input/output error 2024-02-19T19:23:10.893 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-sackoff/tcp4-uni-sackoff04: Cannot open: Input/output error 2024-02-19T19:23:10.893 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip: Cannot mkdir: Input/output error 2024-02-19T19:23:10.893 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip09: Cannot open: Input/output error 2024-02-19T19:23:10.893 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip08: Cannot open: Input/output error 2024-02-19T19:23:10.893 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip06: Cannot open: Input/output error 2024-02-19T19:23:10.894 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip02: Cannot open: Input/output error 2024-02-19T19:23:10.894 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip01: Cannot open: Input/output error 2024-02-19T19:23:10.894 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip04: Cannot open: Input/output error 2024-02-19T19:23:10.894 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip12: Cannot open: Input/output error 2024-02-19T19:23:10.894 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip03: Cannot open: Input/output error 2024-02-19T19:23:10.894 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip03: Cannot open: Input/output error 2024-02-19T19:23:10.894 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip13: Cannot open: Input/output error 2024-02-19T19:23:10.894 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip14: Cannot open: Input/output error 2024-02-19T19:23:10.894 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.895 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip02: Cannot open: Input/output error 2024-02-19T19:23:10.895 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip11: Cannot open: Input/output error 2024-02-19T19:23:10.895 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip04: Cannot open: Input/output error 2024-02-19T19:23:10.895 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.895 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip06: Cannot open: Input/output error 2024-02-19T19:23:10.895 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip10: Cannot open: Input/output error 2024-02-19T19:23:10.895 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip12: Cannot open: Input/output error 2024-02-19T19:23:10.895 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip09: Cannot open: Input/output error 2024-02-19T19:23:10.895 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip14: Cannot open: Input/output error 2024-02-19T19:23:10.895 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip07: Cannot open: Input/output error 2024-02-19T19:23:10.896 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip08: Cannot open: Input/output error 2024-02-19T19:23:10.896 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip13: Cannot open: Input/output error 2024-02-19T19:23:10.896 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp6-multi-diffip05: Cannot open: Input/output error 2024-02-19T19:23:10.896 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip10: Cannot open: Input/output error 2024-02-19T19:23:10.896 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip07: Cannot open: Input/output error 2024-02-19T19:23:10.896 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip01: Cannot open: Input/output error 2024-02-19T19:23:10.896 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip11: Cannot open: Input/output error 2024-02-19T19:23:10.896 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/multi-diffip/tcp4-multi-diffip05: Cannot open: Input/output error 2024-02-19T19:23:10.896 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso: Cannot mkdir: Input/output error 2024-02-19T19:23:10.897 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso11: Cannot open: Input/output error 2024-02-19T19:23:10.897 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso07: Cannot open: Input/output error 2024-02-19T19:23:10.897 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso14: Cannot open: Input/output error 2024-02-19T19:23:10.897 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso05: Cannot open: Input/output error 2024-02-19T19:23:10.897 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso08: Cannot open: Input/output error 2024-02-19T19:23:10.897 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso04: Cannot open: Input/output error 2024-02-19T19:23:10.897 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso10: Cannot open: Input/output error 2024-02-19T19:23:10.897 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso06: Cannot open: Input/output error 2024-02-19T19:23:10.897 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso08: Cannot open: Input/output error 2024-02-19T19:23:10.898 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso12: Cannot open: Input/output error 2024-02-19T19:23:10.898 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.898 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso03: Cannot open: Input/output error 2024-02-19T19:23:10.898 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso01: Cannot open: Input/output error 2024-02-19T19:23:10.898 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.898 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso06: Cannot open: Input/output error 2024-02-19T19:23:10.898 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso13: Cannot open: Input/output error 2024-02-19T19:23:10.898 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso07: Cannot open: Input/output error 2024-02-19T19:23:10.898 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso02: Cannot open: Input/output error 2024-02-19T19:23:10.899 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso04: Cannot open: Input/output error 2024-02-19T19:23:10.899 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso12: Cannot open: Input/output error 2024-02-19T19:23:10.899 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso01: Cannot open: Input/output error 2024-02-19T19:23:10.899 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso10: Cannot open: Input/output error 2024-02-19T19:23:10.899 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso09: Cannot open: Input/output error 2024-02-19T19:23:10.899 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso14: Cannot open: Input/output error 2024-02-19T19:23:10.899 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso02: Cannot open: Input/output error 2024-02-19T19:23:10.899 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso09: Cannot open: Input/output error 2024-02-19T19:23:10.899 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso11: Cannot open: Input/output error 2024-02-19T19:23:10.899 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso03: Cannot open: Input/output error 2024-02-19T19:23:10.900 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp6-uni-tso05: Cannot open: Input/output error 2024-02-19T19:23:10.900 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-tso/tcp4-uni-tso13: Cannot open: Input/output error 2024-02-19T19:23:10.900 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale: Cannot mkdir: Input/output error 2024-02-19T19:23:10.900 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale07: Cannot open: Input/output error 2024-02-19T19:23:10.900 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale01: Cannot open: Input/output error 2024-02-19T19:23:10.900 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale13: Cannot open: Input/output error 2024-02-19T19:23:10.900 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale06: Cannot open: Input/output error 2024-02-19T19:23:10.900 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale12: Cannot open: Input/output error 2024-02-19T19:23:10.900 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale05: Cannot open: Input/output error 2024-02-19T19:23:10.901 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale11: Cannot open: Input/output error 2024-02-19T19:23:10.901 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale02: Cannot open: Input/output error 2024-02-19T19:23:10.901 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale02: Cannot open: Input/output error 2024-02-19T19:23:10.901 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale07: Cannot open: Input/output error 2024-02-19T19:23:10.901 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale01: Cannot open: Input/output error 2024-02-19T19:23:10.901 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale11: Cannot open: Input/output error 2024-02-19T19:23:10.901 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale14: Cannot open: Input/output error 2024-02-19T19:23:10.901 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.902 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.902 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale13: Cannot open: Input/output error 2024-02-19T19:23:10.902 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale08: Cannot open: Input/output error 2024-02-19T19:23:10.902 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale06: Cannot open: Input/output error 2024-02-19T19:23:10.902 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale10: Cannot open: Input/output error 2024-02-19T19:23:10.902 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale05: Cannot open: Input/output error 2024-02-19T19:23:10.902 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale04: Cannot open: Input/output error 2024-02-19T19:23:10.902 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale10: Cannot open: Input/output error 2024-02-19T19:23:10.902 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale09: Cannot open: Input/output error 2024-02-19T19:23:10.902 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale09: Cannot open: Input/output error 2024-02-19T19:23:10.903 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale03: Cannot open: Input/output error 2024-02-19T19:23:10.903 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale03: Cannot open: Input/output error 2024-02-19T19:23:10.903 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp4-uni-winscale04: Cannot open: Input/output error 2024-02-19T19:23:10.903 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale08: Cannot open: Input/output error 2024-02-19T19:23:10.903 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale12: Cannot open: Input/output error 2024-02-19T19:23:10.903 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/tcp/uni-winscale/tcp6-uni-winscale14: Cannot open: Input/output error 2024-02-19T19:23:10.903 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/dns: Cannot mkdir: Input/output error 2024-02-19T19:23:10.903 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/dns6-stress: Cannot open: Input/output error 2024-02-19T19:23:10.903 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.904 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.904 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/dns4-stress: Cannot open: Input/output error 2024-02-19T19:23:10.904 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/dns-stress01-rmt: Cannot open: Input/output error 2024-02-19T19:23:10.904 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/dns/dns-stress02-rmt: Cannot open: Input/output error 2024-02-19T19:23:10.904 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp: Cannot mkdir: Input/output error 2024-02-19T19:23:10.904 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp6-download-stress: Cannot open: Input/output error 2024-02-19T19:23:10.904 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp-upload-stress01-rmt: Cannot open: Input/output error 2024-02-19T19:23:10.904 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp-download-stress02-rmt: Cannot open: Input/output error 2024-02-19T19:23:10.904 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.905 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp4-upload-stress: Cannot open: Input/output error 2024-02-19T19:23:10.905 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.905 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp-upload-stress02-rmt: Cannot open: Input/output error 2024-02-19T19:23:10.905 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp6-upload-stress: Cannot open: Input/output error 2024-02-19T19:23:10.905 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp-download-stress01-rmt: Cannot open: Input/output error 2024-02-19T19:23:10.905 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ftp/ftp4-download-stress: Cannot open: Input/output error 2024-02-19T19:23:10.905 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/README: Cannot open: Input/output error 2024-02-19T19:23:10.905 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/interface: Cannot mkdir: Input/output error 2024-02-19T19:23:10.906 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-updown: Cannot open: Input/output error 2024-02-19T19:23:10.906 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.906 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.906 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-updown: Cannot open: Input/output error 2024-02-19T19:23:10.906 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-route-addlarge: Cannot open: Input/output error 2024-02-19T19:23:10.906 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-alias-adddel: Cannot open: Input/output error 2024-02-19T19:23:10.906 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-addr-change: Cannot open: Input/output error 2024-02-19T19:23:10.906 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-alias-addlarge: Cannot open: Input/output error 2024-02-19T19:23:10.906 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-addr-addlarge: Cannot open: Input/output error 2024-02-19T19:23:10.906 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-mtu-change: Cannot open: Input/output error 2024-02-19T19:23:10.907 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-route-addlarge: Cannot open: Input/output error 2024-02-19T19:23:10.907 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-mtu-change: Cannot open: Input/output error 2024-02-19T19:23:10.907 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-addr-adddel: Cannot open: Input/output error 2024-02-19T19:23:10.907 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if6-route-adddel: Cannot open: Input/output error 2024-02-19T19:23:10.907 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/interface/if4-route-adddel: Cannot open: Input/output error 2024-02-19T19:23:10.907 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp: Cannot mkdir: Input/output error 2024-02-19T19:23:10.907 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport: Cannot mkdir: Input/output error 2024-02-19T19:23:10.907 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport03: Cannot open: Input/output error 2024-02-19T19:23:10.908 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport05: Cannot open: Input/output error 2024-02-19T19:23:10.908 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport02: Cannot open: Input/output error 2024-02-19T19:23:10.908 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport05: Cannot open: Input/output error 2024-02-19T19:23:10.908 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.908 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport01: Cannot open: Input/output error 2024-02-19T19:23:10.908 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport07: Cannot open: Input/output error 2024-02-19T19:23:10.908 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.908 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport06: Cannot open: Input/output error 2024-02-19T19:23:10.908 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport02: Cannot open: Input/output error 2024-02-19T19:23:10.909 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport06: Cannot open: Input/output error 2024-02-19T19:23:10.909 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport03: Cannot open: Input/output error 2024-02-19T19:23:10.909 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport04: Cannot open: Input/output error 2024-02-19T19:23:10.909 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport04: Cannot open: Input/output error 2024-02-19T19:23:10.909 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp6-multi-diffport01: Cannot open: Input/output error 2024-02-19T19:23:10.909 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffport/udp4-multi-diffport07: Cannot open: Input/output error 2024-02-19T19:23:10.909 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic: Cannot mkdir: Input/output error 2024-02-19T19:23:10.909 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic05: Cannot open: Input/output error 2024-02-19T19:23:10.910 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic04: Cannot open: Input/output error 2024-02-19T19:23:10.910 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic06: Cannot open: Input/output error 2024-02-19T19:23:10.910 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic03: Cannot open: Input/output error 2024-02-19T19:23:10.910 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.910 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.910 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic02: Cannot open: Input/output error 2024-02-19T19:23:10.910 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic06: Cannot open: Input/output error 2024-02-19T19:23:10.910 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic01: Cannot open: Input/output error 2024-02-19T19:23:10.910 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic07: Cannot open: Input/output error 2024-02-19T19:23:10.911 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic07: Cannot open: Input/output error 2024-02-19T19:23:10.911 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic05: Cannot open: Input/output error 2024-02-19T19:23:10.911 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic02: Cannot open: Input/output error 2024-02-19T19:23:10.911 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic03: Cannot open: Input/output error 2024-02-19T19:23:10.911 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp4-uni-basic01: Cannot open: Input/output error 2024-02-19T19:23:10.911 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/uni-basic/udp6-uni-basic04: Cannot open: Input/output error 2024-02-19T19:23:10.911 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.911 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic: Cannot mkdir: Input/output error 2024-02-19T19:23:10.911 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic01: Cannot open: Input/output error 2024-02-19T19:23:10.911 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic02: Cannot open: Input/output error 2024-02-19T19:23:10.912 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic06: Cannot open: Input/output error 2024-02-19T19:23:10.912 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic04: Cannot open: Input/output error 2024-02-19T19:23:10.912 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.912 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic07: Cannot open: Input/output error 2024-02-19T19:23:10.912 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic01: Cannot open: Input/output error 2024-02-19T19:23:10.912 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.912 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic03: Cannot open: Input/output error 2024-02-19T19:23:10.912 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic05: Cannot open: Input/output error 2024-02-19T19:23:10.912 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic02: Cannot open: Input/output error 2024-02-19T19:23:10.913 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic04: Cannot open: Input/output error 2024-02-19T19:23:10.913 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic03: Cannot open: Input/output error 2024-02-19T19:23:10.913 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic06: Cannot open: Input/output error 2024-02-19T19:23:10.913 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp4-multi-diffnic05: Cannot open: Input/output error 2024-02-19T19:23:10.913 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffnic/udp6-multi-diffnic07: Cannot open: Input/output error 2024-02-19T19:23:10.913 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip: Cannot mkdir: Input/output error 2024-02-19T19:23:10.913 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip02: Cannot open: Input/output error 2024-02-19T19:23:10.913 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.913 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.913 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip06: Cannot open: Input/output error 2024-02-19T19:23:10.914 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip04: Cannot open: Input/output error 2024-02-19T19:23:10.914 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip05: Cannot open: Input/output error 2024-02-19T19:23:10.914 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip06: Cannot open: Input/output error 2024-02-19T19:23:10.914 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip07: Cannot open: Input/output error 2024-02-19T19:23:10.914 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip01: Cannot open: Input/output error 2024-02-19T19:23:10.914 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip03: Cannot open: Input/output error 2024-02-19T19:23:10.914 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip02: Cannot open: Input/output error 2024-02-19T19:23:10.914 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip01: Cannot open: Input/output error 2024-02-19T19:23:10.914 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip05: Cannot open: Input/output error 2024-02-19T19:23:10.915 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip03: Cannot open: Input/output error 2024-02-19T19:23:10.915 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp6-multi-diffip07: Cannot open: Input/output error 2024-02-19T19:23:10.915 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/udp/multi-diffip/udp4-multi-diffip04: Cannot open: Input/output error 2024-02-19T19:23:10.915 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/http: Cannot mkdir: Input/output error 2024-02-19T19:23:10.915 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/http/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.915 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/http/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.915 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/http/http-stress01-rmt: Cannot open: Input/output error 2024-02-19T19:23:10.915 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/http/http4-stress: Cannot open: Input/output error 2024-02-19T19:23:10.915 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/http/http-stress02-rmt: Cannot open: Input/output error 2024-02-19T19:23:10.916 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/http/http6-stress: Cannot open: Input/output error 2024-02-19T19:23:10.916 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh: Cannot mkdir: Input/output error 2024-02-19T19:23:10.916 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.916 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/ssh-stress02-rmt: Cannot open: Input/output error 2024-02-19T19:23:10.916 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/ssh6-stress: Cannot open: Input/output error 2024-02-19T19:23:10.916 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/ssh4-stress: Cannot open: Input/output error 2024-02-19T19:23:10.916 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/00_Descriptions.txt: Cannot open: Input/output error 2024-02-19T19:23:10.916 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/ssh-stress03-rmt: Cannot open: Input/output error 2024-02-19T19:23:10.916 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress/ssh/ssh-stress01-rmt: Cannot open: Input/output error 2024-02-19T19:23:10.917 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/stress: Cannot utime: Input/output error 2024-02-19T19:23:10.917 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4: Cannot mkdir: Input/output error 2024-02-19T19:23:10.917 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl: Cannot mkdir: Input/output error 2024-02-19T19:23:10.917 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/random_gen.py: Cannot open: Input/output error 2024-02-19T19:23:10.917 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/runtest: Cannot open: Input/output error 2024-02-19T19:23:10.917 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/setacl_stress.py: Cannot open: Input/output error 2024-02-19T19:23:10.917 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/test_long_acl.py: Cannot open: Input/output error 2024-02-19T19:23:10.917 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.917 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/cleanusers.py: Cannot open: Input/output error 2024-02-19T19:23:10.918 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/README: Cannot open: Input/output error 2024-02-19T19:23:10.918 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/create_users.py: Cannot open: Input/output error 2024-02-19T19:23:10.918 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/test_acl.py: Cannot open: Input/output error 2024-02-19T19:23:10.918 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/acl1.c: Cannot open: Input/output error 2024-02-19T19:23:10.918 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/acl/cleangroups.py: Cannot open: Input/output error 2024-02-19T19:23:10.918 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.918 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks: Cannot mkdir: Input/output error 2024-02-19T19:23:10.918 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/netsync.c: Cannot open: Input/output error 2024-02-19T19:23:10.918 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/deploy_info: Cannot open: Input/output error 2024-02-19T19:23:10.918 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/locktests.h: Cannot open: Input/output error 2024-02-19T19:23:10.919 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.919 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/locktests.py: Cannot open: Input/output error 2024-02-19T19:23:10.919 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/locktests.c: Cannot open: Input/output error 2024-02-19T19:23:10.919 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/README: Cannot open: Input/output error 2024-02-19T19:23:10.919 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/deploy: Cannot mkdir: Input/output error 2024-02-19T19:23:10.919 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/deploy/locktests.tar.gz: Cannot open: Input/output error 2024-02-19T19:23:10.919 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfsv4/locks/VERSION: Cannot open: Input/output error 2024-02-19T19:23:10.919 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/iproute: Cannot mkdir: Input/output error 2024-02-19T19:23:10.919 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/iproute/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.919 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/iproute/ip_tests.sh: Cannot open: Input/output error 2024-02-19T19:23:10.920 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/iptables: Cannot mkdir: Input/output error 2024-02-19T19:23:10.920 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-02-19T19:23:10.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:23:10.921 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-19T19:23:10.931 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-02-19T19:23:10.931 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/iptables/iptables_tests.sh: Cannot open: Input/output error 2024-02-19T19:23:10.931 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/iptables/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.932 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp: Cannot mkdir: Input/output error 2024-02-19T19:23:10.932 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.932 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib: Cannot mkdir: Input/output error 2024-02-19T19:23:10.932 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/addrs.c: Cannot open: Input/output error 2024-02-19T19:23:10.932 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/connectx.c: Cannot open: Input/output error 2024-02-19T19:23:10.932 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/bindx.c: Cannot open: Input/output error 2024-02-19T19:23:10.932 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/opt_info.c: Cannot open: Input/output error 2024-02-19T19:23:10.932 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.932 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/peeloff.c: Cannot open: Input/output error 2024-02-19T19:23:10.933 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/sendmsg.c: Cannot open: Input/output error 2024-02-19T19:23:10.933 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/lib/recvmsg.c: Cannot open: Input/output error 2024-02-19T19:23:10.933 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests: Cannot mkdir: Input/output error 2024-02-19T19:23:10.933 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_inaddr_any.c: Cannot open: Input/output error 2024-02-19T19:23:10.933 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_sctp_sendrecvmsg.c: Cannot open: Input/output error 2024-02-19T19:23:10.933 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_autoclose.c: Cannot open: Input/output error 2024-02-19T19:23:10.933 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_basic.c: Cannot open: Input/output error 2024-02-19T19:23:10.933 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_send.c: Cannot open: Input/output error 2024-02-19T19:23:10.933 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_threads.c: Cannot open: Input/output error 2024-02-19T19:23:10.933 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_getname.c: Cannot open: Input/output error 2024-02-19T19:23:10.934 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_peeloff.c: Cannot open: Input/output error 2024-02-19T19:23:10.934 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_connectx.c: Cannot open: Input/output error 2024-02-19T19:23:10.934 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_socket_bind_listen.c: Cannot open: Input/output error 2024-02-19T19:23:10.934 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_shutdown.c: Cannot open: Input/output error 2024-02-19T19:23:10.934 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_recvmsg.c: Cannot open: Input/output error 2024-02-19T19:23:10.934 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.934 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_accept_close.c: Cannot open: Input/output error 2024-02-19T19:23:10.934 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_nonblock.c: Cannot open: Input/output error 2024-02-19T19:23:10.934 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_recvfrom.c: Cannot open: Input/output error 2024-02-19T19:23:10.934 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_connect.c: Cannot open: Input/output error 2024-02-19T19:23:10.935 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_initmsg_connect.c: Cannot open: Input/output error 2024-02-19T19:23:10.935 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_sendmsg.c: Cannot open: Input/output error 2024-02-19T19:23:10.935 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_sendto.c: Cannot open: Input/output error 2024-02-19T19:23:10.935 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_connect.c: Cannot open: Input/output error 2024-02-19T19:23:10.935 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_tcp_style.c: Cannot open: Input/output error 2024-02-19T19:23:10.935 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_sockopt.c: Cannot open: Input/output error 2024-02-19T19:23:10.935 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_connectx.c: Cannot open: Input/output error 2024-02-19T19:23:10.935 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_fragments.c: Cannot open: Input/output error 2024-02-19T19:23:10.935 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_timetolive.c: Cannot open: Input/output error 2024-02-19T19:23:10.936 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_recvmsg.c: Cannot open: Input/output error 2024-02-19T19:23:10.936 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_assoc_shutdown.c: Cannot open: Input/output error 2024-02-19T19:23:10.936 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_events.c: Cannot open: Input/output error 2024-02-19T19:23:10.936 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_sockopt.c: Cannot open: Input/output error 2024-02-19T19:23:10.936 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_assoc_abort.c: Cannot open: Input/output error 2024-02-19T19:23:10.936 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_rtoinfo.c: Cannot open: Input/output error 2024-02-19T19:23:10.936 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/func_tests/test_1_to_1_addrs.c: Cannot open: Input/output error 2024-02-19T19:23:10.936 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/testlib: Cannot mkdir: Input/output error 2024-02-19T19:23:10.936 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/testlib/sctputil.h: Cannot open: Input/output error 2024-02-19T19:23:10.937 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/testlib/sctputil.c: Cannot open: Input/output error 2024-02-19T19:23:10.937 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/testlib/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.937 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/include: Cannot mkdir: Input/output error 2024-02-19T19:23:10.937 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/include/netinet: Cannot mkdir: Input/output error 2024-02-19T19:23:10.937 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/sctp/include/netinet/sctp.h: Cannot open: Input/output error 2024-02-19T19:23:10.937 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs: Cannot mkdir: Input/output error 2024-02-19T19:23:10.937 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/fsx-linux: Cannot mkdir: Input/output error 2024-02-19T19:23:10.937 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/fsx-linux/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.937 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/fsx-linux/fsx.sh: Cannot open: Input/output error 2024-02-19T19:23:10.937 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstress: Cannot mkdir: Input/output error 2024-02-19T19:23:10.938 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstress/make_tree.c: Cannot open: Input/output error 2024-02-19T19:23:10.938 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstress/nfsstress: Cannot open: Input/output error 2024-02-19T19:23:10.938 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstress/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.938 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs04: Cannot mkdir: Input/output error 2024-02-19T19:23:10.938 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs04/nfs04: Cannot open: Input/output error 2024-02-19T19:23:10.938 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs04/nfs04_create_file.c: Cannot open: Input/output error 2024-02-19T19:23:10.938 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs04/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.938 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01: Cannot mkdir: Input/output error 2024-02-19T19:23:10.938 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfs_flock.c: Cannot open: Input/output error 2024-02-19T19:23:10.939 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfs_flock.h: Cannot open: Input/output error 2024-02-19T19:23:10.939 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfslock01: Cannot open: Input/output error 2024-02-19T19:23:10.939 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.939 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfs_flock_dgen.c: Cannot open: Input/output error 2024-02-19T19:23:10.939 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfs_flock_frk.c: Cannot open: Input/output error 2024-02-19T19:23:10.939 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfslock01/nfs_flock_func.c: Cannot open: Input/output error 2024-02-19T19:23:10.939 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.939 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstat01: Cannot mkdir: Input/output error 2024-02-19T19:23:10.939 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstat01/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.940 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfsstat01/nfsstat01: Cannot open: Input/output error 2024-02-19T19:23:10.940 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs02: Cannot mkdir: Input/output error 2024-02-19T19:23:10.940 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs02/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.940 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs02/nfs02: Cannot open: Input/output error 2024-02-19T19:23:10.940 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs01: Cannot mkdir: Input/output error 2024-02-19T19:23:10.940 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs01/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.940 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs01/nfs01: Cannot open: Input/output error 2024-02-19T19:23:10.940 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs01/open_files.c: Cannot open: Input/output error 2024-02-19T19:23:10.940 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs03: Cannot mkdir: Input/output error 2024-02-19T19:23:10.941 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs03/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.941 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs03/nfs03: Cannot open: Input/output error 2024-02-19T19:23:10.941 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/README: Cannot open: Input/output error 2024-02-19T19:23:10.941 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/CTHON04_README: Cannot open: Input/output error 2024-02-19T19:23:10.941 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress: Cannot mkdir: Input/output error 2024-02-19T19:23:10.941 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress/global.h: Cannot open: Input/output error 2024-02-19T19:23:10.941 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress/nfs_fsstress.sh: Cannot open: Input/output error 2024-02-19T19:23:10.941 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.941 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress/xfscompat.h: Cannot open: Input/output error 2024-02-19T19:23:10.941 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network/nfs/nfs_fsstress/fsstress.c: Cannot open: Input/output error 2024-02-19T19:23:10.942 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/network: Cannot utime: Input/output error 2024-02-19T19:23:10.942 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21: Cannot mkdir: Input/output error 2024-02-19T19:23:10.942 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/COPYING: Cannot open: Input/output error 2024-02-19T19:23:10.942 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/debug.c: Cannot open: Input/output error 2024-02-19T19:23:10.942 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts: Cannot mkdir: Input/output error 2024-02-19T19:23:10.942 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/00check_cdrom_presence: Cannot open: Input/output error 2024-02-19T19:23:10.942 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/mem_alloc: Cannot open: Input/output error 2024-02-19T19:23:10.942 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/ramsnake: Cannot open: Input/output error 2024-02-19T19:23:10.942 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/ltp: Cannot open: Input/output error 2024-02-19T19:23:10.942 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/memxfer5b: Cannot open: Input/output error 2024-02-19T19:23:10.943 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/memtest: Cannot open: Input/output error 2024-02-19T19:23:10.943 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/cpufreq: Cannot open: Input/output error 2024-02-19T19:23:10.943 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/00xbonkers: Cannot open: Input/output error 2024-02-19T19:23:10.943 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/coords: Cannot open: Input/output error 2024-02-19T19:23:10.943 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/00checklatest: Cannot open: Input/output error 2024-02-19T19:23:10.943 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/build_kernel: Cannot open: Input/output error 2024-02-19T19:23:10.943 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/schedutils: Cannot open: Input/output error 2024-02-19T19:23:10.943 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/time_test: Cannot open: Input/output error 2024-02-19T19:23:10.943 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/lame: Cannot open: Input/output error 2024-02-19T19:23:10.943 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/ipmitool: Cannot open: Input/output error 2024-02-19T19:23:10.944 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/bonnie++: Cannot open: Input/output error 2024-02-19T19:23:10.944 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/random_syscall: Cannot open: Input/output error 2024-02-19T19:23:10.944 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/build_scripts/nasm: Cannot open: Input/output error 2024-02-19T19:23:10.944 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/QUICKSTART: Cannot open: Input/output error 2024-02-19T19:23:10.944 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/CHANGELOG: Cannot open: Input/output error 2024-02-19T19:23:10.944 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/nfs_logging: Cannot open: Input/output error 2024-02-19T19:23:10.944 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/Makefile: Cannot open: Input/output error 2024-02-19T19:23:10.944 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/test-tests.tar.gz: Cannot open: Input/output error 2024-02-19T19:23:10.944 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/fast-tests.tar.gz: Cannot open: Input/output error 2024-02-19T19:23:10.945 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/run.c: Cannot open: Input/output error 2024-02-19T19:23:10.945 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/README: Cannot open: Input/output error 2024-02-19T19:23:10.945 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/Install: Cannot open 2024-02-19T19:23:10.945 INFO:tasks.workunit.client.0.smithi114.stderr:gzip: : Input/output error 2024-02-19T19:23:10.945 INFO:tasks.workunit.client.0.smithi114.stderr:stdin: Stale file handle 2024-02-19T19:23:10.945 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases/pounder21/fancy_timed_loop.c: Cannot open: Input/output error 2024-02-19T19:23:10.945 INFO:tasks.workunit.client.0.smithi114.stderr:tar: Child returned status 1 2024-02-19T19:23:10.945 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Input/output error 2024-02-19T19:23:10.945 INFO:tasks.workunit.client.0.smithi114.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-02-19T19:23:10.946 INFO:tasks.workunit.client.0.smithi114.stderr:tar: Error is not recoverable: exiting now 2024-02-19T19:23:10.946 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-02-19T19:23:10.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-19T19:23:10.996 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-02-19T19:23:10.997 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:23:11.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:11.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:11.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:11.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:11.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:11.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:11.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:11.138+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:11.777 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:23:11.437+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 11754 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:23:11.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:11.429+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:11.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:11.429+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:11.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:11.429+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:11.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:11.430+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:11.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:11.465+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:11.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:11.465+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:11.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:11.465+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:11.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:11.465+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:12.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:12.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:12.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:12.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:12.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:12.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:12.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:12.100+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:12.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:12.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:12.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:12.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:12.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:12.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:12.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:12.464+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:12.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:12.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:12.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:12.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:12.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:12.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:12.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:12.381+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:13.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:13.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:13.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:13.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:13.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:13.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:13.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:13.123+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:13.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:13.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:13.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:13.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:13.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:13.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:13.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:13.360+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:13.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:13.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:13.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:13.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:13.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:13.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:13.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:13.456+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:14.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:14.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:14.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:14.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:14.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:14.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:14.405 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:14.096+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:14.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:14.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:14.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:14.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:14.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:14.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:14.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:14.489+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:14.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:14.395+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:14.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:14.395+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:14.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:14.395+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:14.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:14.396+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:15.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:15.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:15.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:15.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:15.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:15.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:15.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:15.075+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:15.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:15.521+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:15.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:15.521+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:15.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:15.521+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:15.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:15.522+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:15.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:15.360+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:15.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:15.360+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:15.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:15.360+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:15.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:15.361+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:16.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:16.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:16.408 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:16.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:16.408 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:16.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:16.408 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:16.061+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:16.777 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:23:16.439+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 11851 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:23:16.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:16.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:16.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:16.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:16.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:16.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:16.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:16.543+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:16.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:16.399+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:16.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:16.399+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:16.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:16.399+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:16.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:16.400+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:17.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:17.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:17.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:17.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:17.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:17.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:17.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:17.100+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:17.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:17.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:17.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:17.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:17.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:17.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:17.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:17.497+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:17.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:17.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:17.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:17.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:17.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:17.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:17.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:17.383+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:18.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:18.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:18.422 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:18.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:18.422 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:18.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:18.422 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:18.124+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:18.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:18.412+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:18.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:18.412+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:18.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:18.412+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:18.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:18.413+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:18.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:18.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:18.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:18.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:18.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:18.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:18.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:18.537+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:19.445 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:19.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:19.446 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:19.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:19.446 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:19.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:19.446 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:19.150+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:19.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:19.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:19.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:19.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:19.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:19.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:19.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:19.546+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:19.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:19.437+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:19.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:19.437+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:19.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:19.437+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:19.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:19.437+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:20.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:20.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:20.475 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:20.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:20.475 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:20.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:20.475 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:20.106+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:20.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:20.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:20.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:20.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:20.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:20.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:20.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:20.527+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:20.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:20.466+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:20.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:20.466+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:20.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:20.466+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:20.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:20.467+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:21.449 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:21.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:21.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:21.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:21.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:21.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:21.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:21.154+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:21.777 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:23:21.440+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 11946 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:23:21.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:21.456+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:21.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:21.456+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:21.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:21.456+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:21.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:21.457+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:21.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:21.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:21.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:21.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:21.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:21.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:21.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:21.514+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:22.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:22.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:22.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:22.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:22.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:22.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:22.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:22.198+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:22.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:22.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:22.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:22.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:22.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:22.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:22.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:22.488+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:22.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:22.491+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:22.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:22.492+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:22.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:22.492+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:22.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:22.492+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:23.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:23.222+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:23.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:23.222+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:23.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:23.222+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:23.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:23.222+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:23.528 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:23.468+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:23.528 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:23.468+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:23.528 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:23.468+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:23.528 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:23.469+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:24.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:23.532+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:24.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:23.532+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:24.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:23.532+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:24.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:23.533+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:24.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:24.223+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:24.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:24.223+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:24.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:24.223+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:24.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:24.224+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:24.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:24.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:24.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:24.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:24.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:24.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:24.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:24.556+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:24.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:24.515+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:24.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:24.516+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:24.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:24.516+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:24.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:24.516+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:25.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:25.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:25.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:25.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:25.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:25.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:25.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:25.180+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:25.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:25.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:25.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:25.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:25.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:25.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:25.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:25.599+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:25.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:25.490+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:25.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:25.490+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:25.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:25.490+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:25.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:25.491+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:26.473 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:23:26.441+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 12041 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:23:26.473 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:26.207+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:26.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:26.207+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:26.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:26.207+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:26.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:26.207+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:26.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:26.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:26.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:26.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:26.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:26.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:26.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:26.569+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:26.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:26.505+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:26.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:26.505+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:26.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:26.505+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:26.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:26.506+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:27.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:27.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:27.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:27.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:27.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:27.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:27.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:27.461+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:27.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:27.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:27.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:27.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:27.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:27.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:27.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:27.212+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:28.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:27.522+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:28.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:27.522+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:28.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:27.522+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:28.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:27.523+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:28.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:28.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:28.483 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:28.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:28.483 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:28.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:28.483 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:28.193+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:28.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:28.528+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:28.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:28.528+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:28.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:28.528+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:28.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:28.528+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:28.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:28.474+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:28.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:28.474+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:28.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:28.474+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:28.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:28.475+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:29.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:29.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:29.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:29.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:29.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:29.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:29.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:29.173+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:29.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:29.488+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:29.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:29.488+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:29.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:29.488+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:29.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:29.489+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:29.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:29.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:29.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:29.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:29.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:29.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:29.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:29.546+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:30.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:30.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:30.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:30.140+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:30.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:30.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:30.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:30.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:30.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:30.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:30.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:30.539+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:30.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:30.511+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:30.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:30.511+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:30.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:30.511+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:30.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:30.512+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:31.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:31.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:31.452 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:31.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:31.452 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:31.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:31.452 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:31.104+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:31.777 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:23:31.443+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 12135 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:23:31.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:31.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:31.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:31.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:31.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:31.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:31.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:31.589+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:31.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:31.548+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:31.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:31.548+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:31.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:31.548+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:31.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:31.549+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:32.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:32.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:32.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:32.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:32.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:32.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:32.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:32.079+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:33.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:32.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:33.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:32.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:33.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:32.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:33.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:32.582+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:33.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:32.554+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:33.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:32.554+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:33.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:32.554+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:33.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:32.555+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:33.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:33.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:33.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:33.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:33.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:33.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:33.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:33.110+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:33.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:33.618+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:33.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:33.618+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:33.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:33.618+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:33.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:33.618+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:33.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:33.510+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:33.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:33.510+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:33.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:33.510+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:33.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:33.511+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:34.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:34.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:34.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:34.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:34.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:34.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:34.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:34.159+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:35.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:34.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:35.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:34.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:35.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:34.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:35.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:34.577+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:35.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:34.536+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:35.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:34.536+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:35.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:34.536+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:35.028 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:34.537+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:35.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:35.147+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:35.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:35.147+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:35.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:35.147+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:35.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:35.148+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:35.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:35.556+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:35.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:35.556+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:35.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:35.556+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:35.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:35.557+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:35.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:35.574+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:35.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:35.574+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:35.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:35.574+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:35.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:35.575+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:36.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:36.115+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:36.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:36.115+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:36.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:36.115+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:36.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:36.115+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:36.777 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:23:36.445+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 12232 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:23:36.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:36.523+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:36.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:36.523+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:36.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:36.523+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:36.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:36.524+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:36.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:36.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:36.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:36.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:36.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:36.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:36.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:36.599+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:37.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:37.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:37.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:37.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:37.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:37.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:37.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:37.067+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:37.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:37.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:37.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:37.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:37.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:37.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:37.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:37.629+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:37.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:37.492+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:37.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:37.492+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:37.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:37.492+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:37.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:37.493+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:38.516 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:38.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:38.517 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:38.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:38.517 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:38.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:38.517 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:38.049+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:38.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:38.583+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:38.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:38.583+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:38.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:38.583+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:38.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:38.583+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:38.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:38.508+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:38.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:38.508+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:38.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:38.508+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:38.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:38.508+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:39.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:39.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:39.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:39.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:39.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:39.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:39.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:39.018+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:39.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:39.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:39.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:39.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:39.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:39.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:39.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:39.623+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:39.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:39.471+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:39.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:39.471+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:39.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:39.471+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:39.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:39.472+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:40.495 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:40.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:40.495 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:40.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:40.495 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:40.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:40.495 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:40.019+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:40.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:40.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:40.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:40.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:40.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:40.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:40.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:40.487+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:40.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:40.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:40.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:40.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:40.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:40.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:40.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:40.594+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:41.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:40.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:41.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:40.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:41.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:40.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:41.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:40.998+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:41.776 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:23:41.446+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 12328 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:23:41.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:41.523+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:41.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:41.523+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:41.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:41.523+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:41.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:41.524+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:41.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:41.571+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:41.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:41.571+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:41.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:41.571+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:41.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:41.572+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:42.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:42.021+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:42.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:42.021+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:42.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:42.021+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:42.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:42.021+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:42.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:42.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:42.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:42.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:42.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:42.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:42.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:42.531+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:42.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:42.517+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:42.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:42.517+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:42.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:42.517+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:42.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:42.518+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:43.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:43.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:43.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:43.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:43.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:43.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:43.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:43.031+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:43.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:43.500+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:43.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:43.500+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:43.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:43.500+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:43.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:43.501+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:43.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:43.492+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:43.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:43.492+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:43.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:43.492+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:43.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:43.493+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:44.486 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:44.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:44.487 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:44.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:44.487 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:44.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:44.487 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:44.063+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:44.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:44.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:44.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:44.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:44.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:44.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:44.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:44.479+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:44.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:44.482+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:44.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:44.482+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:44.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:44.482+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:44.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:44.483+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:45.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:45.098+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:45.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:45.098+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:45.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:45.098+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:45.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:45.098+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:45.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:45.486+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:45.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:45.486+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:45.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:45.486+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:45.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:45.487+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:45.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:45.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:45.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:45.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:45.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:45.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:45.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:45.461+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:46.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:46.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:46.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:46.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:46.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:46.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:46.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:46.138+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:46.776 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:23:46.448+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 12425 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:23:46.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:46.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:46.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:46.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:46.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:46.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:46.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:46.511+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:46.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:46.480+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:46.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:46.480+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:46.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:46.480+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:46.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:46.481+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:47.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:47.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:47.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:47.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:47.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:47.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:47.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:47.120+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:47.765 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:47.496+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:47.766 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:47.496+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:47.766 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:47.496+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:47.766 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:47.497+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:47.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:47.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:47.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:47.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:47.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:47.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:47.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:47.489+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:48.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:48.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:48.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:48.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:48.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:48.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:48.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:48.104+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:48.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:48.495+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:48.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:48.495+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:48.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:48.495+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:48.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:48.496+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:48.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:48.546+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:48.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:48.546+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:48.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:48.546+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:48.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:48.547+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:49.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:49.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:49.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:49.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:49.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:49.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:49.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:49.089+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:49.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:49.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:49.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:49.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:49.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:49.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:49.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:49.507+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:49.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:49.498+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:49.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:49.498+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:49.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:49.498+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:49.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:49.498+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:50.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:50.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:50.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:50.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:50.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:50.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:50.475 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:50.076+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:50.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:50.476+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:50.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:50.476+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:50.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:50.476+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:50.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:50.477+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:50.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:50.465+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:50.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:50.465+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:50.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:50.465+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:50.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:50.467+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:51.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:51.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:51.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:51.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:51.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:51.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:51.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:51.089+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:51.776 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:23:51.449+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 12520 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:23:51.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:51.495+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:51.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:51.495+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:51.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:51.495+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:51.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:51.496+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:51.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:51.483+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:51.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:51.483+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:51.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:51.483+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:51.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:51.484+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:52.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:52.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:52.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:52.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:52.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:52.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:52.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:52.122+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:52.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:52.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:52.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:52.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:52.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:52.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:52.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:52.524+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:52.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:52.498+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:52.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:52.498+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:52.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:52.498+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:52.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:52.499+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:53.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:53.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:53.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:53.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:53.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:53.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:53.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:53.124+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:54.026 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:53.524+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:54.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:53.524+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:54.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:53.524+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:54.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:53.525+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:54.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:53.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:54.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:53.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:54.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:53.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:54.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:53.532+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:54.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:54.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:54.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:54.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:54.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:54.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:54.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:54.109+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:54.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:54.572+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:54.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:54.572+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:54.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:54.572+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:54.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:54.573+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:54.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:54.492+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:54.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:54.492+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:54.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:54.492+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:54.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:54.493+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:55.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:55.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:55.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:55.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:55.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:55.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:55.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:55.109+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:55.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:55.622+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:55.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:55.622+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:55.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:55.622+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:55.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:55.623+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:55.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:55.455+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:55.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:55.455+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:55.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:55.455+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:55.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:55.455+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:56.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:56.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:56.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:56.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:56.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:56.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:56.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:56.067+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:56.776 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:23:56.451+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 12618 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:23:56.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:56.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:56.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:56.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:56.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:56.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:56.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:56.574+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:56.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:56.406+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:56.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:56.406+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:56.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:56.406+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:56.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:56.407+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:57.447 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:57.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:57.447 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:57.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:57.447 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:57.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:57.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:57.033+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:57.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:57.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:57.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:57.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:57.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:57.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:57.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:57.561+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:57.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:57.439+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:57.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:57.439+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:57.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:57.439+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:57.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:57.440+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:58.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:58.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:58.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:58.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:58.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:58.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:58.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:58.069+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:58.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:58.520+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:58.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:58.520+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:58.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:58.520+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:58.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:58.520+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:58.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:58.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:58.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:58.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:58.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:58.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:58.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:58.420+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:59.452 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:59.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:59.452 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:59.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:59.452 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:59.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:23:59.453 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:23:59.089+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:23:59.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:59.444+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:59.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:59.444+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:59.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:59.444+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:23:59.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:23:59.444+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:23:59.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:59.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:59.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:59.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:59.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:59.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:23:59.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:23:59.560+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:00.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:00.057+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:00.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:00.057+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:00.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:00.057+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:00.408 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:00.057+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:00.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:00.399+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:00.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:00.399+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:00.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:00.399+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:00.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:00.400+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:00.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:00.557+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:00.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:00.557+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:00.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:00.557+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:00.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:00.558+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:01.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:01.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:01.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:01.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:01.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:01.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:01.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:01.088+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:01.776 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:24:01.452+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 12708 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:24:01.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:01.563+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:01.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:01.563+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:01.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:01.563+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:01.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:01.564+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:01.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:01.437+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:01.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:01.437+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:01.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:01.437+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:01.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:01.438+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:02.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:02.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:02.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:02.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:02.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:02.082+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:02.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:02.457+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:02.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:02.457+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:02.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:02.457+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:02.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:02.458+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:02.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:02.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:02.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:02.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:02.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:02.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:02.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:02.551+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:03.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:03.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:03.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:03.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:03.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:03.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:03.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:03.077+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:03.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:03.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:03.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:03.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:03.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:03.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:03.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:03.579+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:03.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:03.492+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:03.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:03.492+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:03.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:03.492+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:03.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:03.492+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:04.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:04.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:04.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:04.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:04.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:04.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:04.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:04.083+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:05.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:04.532+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:05.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:04.532+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:05.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:04.532+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:05.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:04.532+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:05.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:04.536+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:05.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:04.536+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:05.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:04.536+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:05.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:04.537+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:05.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:05.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:05.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:05.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:05.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:05.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:05.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:05.124+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:05.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:05.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:05.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:05.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:05.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:05.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:05.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:05.518+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:05.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:05.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:05.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:05.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:05.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:05.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:05.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:05.569+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:06.463 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:06.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:06.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:06.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:06.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:06.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:06.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:06.088+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:06.776 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:24:06.454+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 12803 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:24:06.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:06.576+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:06.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:06.576+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:06.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:06.576+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:06.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:06.577+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:06.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:06.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:06.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:06.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:06.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:06.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:06.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:06.544+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:07.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:07.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:07.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:07.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:07.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:07.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:07.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:07.063+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:07.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:07.497+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:07.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:07.497+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:07.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:07.497+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:07.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:07.498+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:07.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:07.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:07.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:07.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:07.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:07.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:07.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:07.535+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:08.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:08.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:08.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:08.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:08.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:08.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:08.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:08.112+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:08.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:08.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:08.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:08.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:08.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:08.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:08.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:08.553+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:08.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:08.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:08.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:08.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:08.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:08.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:08.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:08.514+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:09.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:09.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:09.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:09.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:09.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:09.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:09.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:09.072+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:10.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:09.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:10.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:09.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:10.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:09.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:10.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:09.524+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:10.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:09.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:10.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:09.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:10.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:09.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:10.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:09.554+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:10.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:10.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:10.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:10.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:10.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:10.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:10.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:10.041+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:11.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:10.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:11.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:10.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:11.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:10.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:11.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:10.571+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:11.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:10.542+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:11.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:10.542+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:11.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:10.542+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:11.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:10.543+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:11.463 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:11.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:11.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:11.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:11.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:11.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:11.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:11.083+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:11.776 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:24:11.455+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 12897 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:24:11.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:11.574+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:11.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:11.574+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:11.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:11.574+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:11.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:11.575+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:11.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:11.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:11.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:11.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:11.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:11.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:11.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:11.588+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:12.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:12.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:12.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:12.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:12.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:12.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:12.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:12.091+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:12.998 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:12.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:12.998 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:12.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:12.998 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:12.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:12.998 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:12.561+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:12.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:12.548+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:12.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:12.548+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:12.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:12.548+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:12.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:12.549+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:13.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:13.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:13.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:13.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:13.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:13.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:13.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:13.132+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:13.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:13.590+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:13.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:13.590+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:13.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:13.590+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:13.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:13.591+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:13.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:13.506+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:13.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:13.506+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:13.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:13.506+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:13.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:13.508+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:14.492 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:14.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:14.492 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:14.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:14.492 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:14.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:14.492 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:14.114+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:14.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:14.641+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:14.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:14.641+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:14.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:14.641+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:14.779 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:14.642+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:14.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:14.484+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:14.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:14.484+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:14.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:14.484+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:14.780 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:14.484+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:15.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:15.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:15.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:15.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:15.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:15.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:15.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:15.139+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:15.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:15.652+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:15.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:15.652+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:15.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:15.653+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:15.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:15.653+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:15.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:15.512+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:15.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:15.512+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:15.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:15.512+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:15.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:15.512+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:16.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:16.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:16.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:16.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:16.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:16.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:16.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:16.149+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:16.776 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:24:16.457+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 12993 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:24:16.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:16.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:16.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:16.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:16.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:16.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:16.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:16.626+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:16.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:16.496+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:16.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:16.496+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:16.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:16.496+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:16.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:16.497+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:17.476 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:17.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:17.477 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:17.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:17.477 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:17.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:17.477 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:17.149+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:17.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:17.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:17.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:17.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:17.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:17.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:17.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:17.636+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:17.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:17.469+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:17.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:17.469+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:17.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:17.469+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:17.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:17.469+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:18.481 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:18.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:18.481 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:18.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:18.481 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:18.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:18.481 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:18.157+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:18.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:18.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:18.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:18.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:18.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:18.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:18.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:18.617+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:18.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:18.473+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:18.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:18.473+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:18.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:18.473+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:18.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:18.474+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:19.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:19.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:19.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:19.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:19.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:19.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:19.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:19.153+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:19.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:19.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:19.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:19.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:19.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:19.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:19.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:19.590+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:19.778 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:19.481+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:19.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:19.481+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:19.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:19.481+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:19.779 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:19.482+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:20.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:20.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:20.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:20.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:20.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:20.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:20.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:20.121+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:21.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:20.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:21.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:20.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:21.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:20.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:21.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:20.620+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:21.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:20.529+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:21.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:20.529+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:21.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:20.529+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:21.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:20.530+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:21.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:21.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:21.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:21.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:21.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:21.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:21.085+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:21.776 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:24:21.458+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 13087 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:24:21.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:21.495+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:21.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:21.495+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:21.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:21.495+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:21.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:21.496+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:21.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:21.606+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:21.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:21.606+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:21.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:21.606+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:21.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:21.607+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:22.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:22.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:22.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:22.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:22.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:22.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:22.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:22.099+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:22.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:22.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:22.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:22.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:22.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:22.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:22.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:22.580+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:22.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:22.532+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:22.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:22.532+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:22.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:22.532+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:22.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:22.533+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:23.505 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:23.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:23.505 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:23.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:23.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:23.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:23.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:23.064+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:23.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:23.498+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:23.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:23.498+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:23.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:23.498+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:23.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:23.498+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:23.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:23.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:23.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:23.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:23.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:23.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:23.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:23.619+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:24.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:24.039+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:24.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:24.039+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:24.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:24.039+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:24.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:24.040+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:25.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:24.599+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:25.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:24.599+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:25.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:24.599+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:25.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:24.600+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:25.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:24.536+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:25.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:24.536+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:25.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:24.536+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:25.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:24.536+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:25.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:25.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:25.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:25.067+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:25.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:25.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:25.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:25.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:25.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:25.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:25.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:25.578+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:25.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:25.511+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:25.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:25.511+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:25.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:25.511+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:25.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:25.512+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:26.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:26.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:26.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:26.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:26.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:26.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:26.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:26.071+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:26.776 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:24:26.460+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 13183 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:24:26.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:26.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:26.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:26.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:26.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:26.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:26.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:26.535+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:26.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:26.528+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:26.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:26.528+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:26.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:26.528+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:26.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:26.529+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:27.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:27.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:27.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:27.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:27.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:27.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:27.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:27.051+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:28.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:27.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:28.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:27.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:28.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:27.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:28.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:27.576+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:28.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:27.558+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:28.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:27.559+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:28.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:27.559+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:28.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:27.560+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:28.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:28.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:28.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:28.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:28.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:28.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:28.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:28.054+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:28.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:28.572+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:28.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:28.572+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:28.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:28.572+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:28.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:28.573+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:28.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:28.512+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:28.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:28.512+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:28.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:28.512+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:28.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:28.513+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:29.511 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:29.040+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:29.511 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:29.040+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:29.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:29.040+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:29.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:29.041+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:29.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:29.530+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:29.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:29.530+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:29.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:29.530+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:29.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:29.530+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:29.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:29.503+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:29.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:29.503+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:29.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:29.503+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:29.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:29.504+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:30.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:30.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:30.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:30.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:30.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:30.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:30.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:30.033+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:30.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:30.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:30.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:30.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:30.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:30.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:30.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:30.527+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:30.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:30.505+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:30.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:30.505+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:30.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:30.505+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:30.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:30.506+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:31.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:31.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:31.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:31.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:31.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:31.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:31.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:31.015+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:31.738 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:24:31.462+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 13278 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:24:31.738 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:31.518+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:31.738 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:31.518+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:31.738 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:31.518+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:31.738 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:31.519+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:31.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:31.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:31.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:31.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:31.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:31.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:31.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:31.547+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:32.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:32.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:32.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:32.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:32.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:32.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:32.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:32.020+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:33.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:32.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:33.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:32.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:33.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:32.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:33.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:32.536+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:33.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:32.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:33.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:32.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:33.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:32.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:33.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:32.539+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:33.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:33.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:33.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:33.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:33.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:33.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:33.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:33.019+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:33.997 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:33.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:33.997 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:33.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:33.998 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:33.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:33.998 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:33.540+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:33.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:33.580+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:33.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:33.580+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:33.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:33.580+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:33.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:33.581+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:34.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:33.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:34.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:33.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:34.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:33.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:34.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:33.990+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:34.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:34.563+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:34.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:34.563+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:34.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:34.563+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:34.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:34.564+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:34.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:34.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:34.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:34.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:34.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:34.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:34.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:34.558+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:34.992 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:34.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:34.992 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:34.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:34.992 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:34.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:35.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:34.983+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:36.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:35.588+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:36.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:35.588+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:36.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:35.588+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:36.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:35.589+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:36.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:35.602+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:36.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:35.602+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:36.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:35.602+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:36.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:35.603+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:36.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:35.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:36.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:35.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:36.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:35.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:36.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:35.996+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:36.776 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:24:36.463+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 13375 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:24:36.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:36.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:36.776 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:36.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:36.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:36.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:36.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:36.637+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:36.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:36.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:36.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:36.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:36.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:36.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:36.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:36.606+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:37.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:37.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:37.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:37.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:37.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:37.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:37.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:37.033+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:37.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:37.676+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:37.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:37.676+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:37.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:37.676+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:37.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:37.677+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:37.996 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:37.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:37.997 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:37.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:37.997 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:37.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:37.997 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:37.590+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:38.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:37.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:38.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:37.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:38.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:37.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:38.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:37.988+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:39.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:38.710+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:39.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:38.710+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:39.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:38.710+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:39.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:38.711+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:39.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:38.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:39.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:38.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:39.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:38.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:39.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:38.609+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:39.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:39.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:39.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:39.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:39.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:39.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:39.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:39.033+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:40.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:39.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:40.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:39.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:40.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:39.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:40.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:39.750+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:40.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:39.576+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:40.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:39.576+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:40.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:39.576+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:40.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:39.576+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:40.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:40.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:40.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:40.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:40.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:40.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:40.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:40.028+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:40.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:40.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:40.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:40.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:40.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:40.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:40.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:40.706+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:40.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:40.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:40.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:40.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:40.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:40.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:40.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:40.601+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:41.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:40.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:41.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:40.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:41.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:40.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:41.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:40.996+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:41.743 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:24:41.465+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 13470 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:24:41.743 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:41.599+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:41.744 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:41.599+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:41.744 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:41.599+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:41.744 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:41.600+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:42.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:41.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:42.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:41.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:42.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:41.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:42.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:41.736+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:42.026 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:41.961+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:42.027 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:41.961+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:42.027 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:41.961+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:42.027 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:41.961+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:42.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:42.738+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:42.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:42.738+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:42.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:42.738+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:42.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:42.738+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:42.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:42.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:42.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:42.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:42.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:42.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:42.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:42.649+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:43.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:42.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:43.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:42.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:43.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:42.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:43.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:42.935+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:43.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:43.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:43.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:43.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:43.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:43.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:43.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:43.706+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:43.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:43.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:43.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:43.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:43.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:43.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:43.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:43.604+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:44.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:43.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:44.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:43.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:44.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:43.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:44.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:43.953+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:44.934 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:44.745+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:44.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:44.745+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:44.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:44.745+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:44.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:44.745+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:44.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:44.578+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:44.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:44.578+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:44.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:44.578+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:44.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:44.578+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:45.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:44.927+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:45.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:44.927+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:45.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:44.927+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:45.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:44.927+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:45.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:45.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:45.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:45.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:45.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:45.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:45.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:45.581+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:45.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:45.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:45.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:45.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:45.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:45.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:45.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:45.771+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:46.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:45.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:46.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:45.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:46.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:45.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:46.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:45.974+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:46.775 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:24:46.466+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 13566 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:24:46.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:46.597+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:46.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:46.597+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:46.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:46.597+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:46.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:46.598+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:47.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:46.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:47.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:46.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:47.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:46.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:47.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:46.804+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:47.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:46.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:47.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:46.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:47.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:46.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:47.278 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:46.958+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:47.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:47.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:47.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:47.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:47.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:47.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:47.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:47.819+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:47.969 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:47.585+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:47.969 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:47.585+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:47.969 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:47.585+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:47.969 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:47.586+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:48.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:47.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:48.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:47.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:48.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:47.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:48.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:47.961+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:48.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:48.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:48.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:48.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:48.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:48.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:48.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:48.582+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:49.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:48.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:49.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:48.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:49.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:48.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:49.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:48.921+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:49.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:48.857+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:49.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:48.857+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:49.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:48.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:49.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:48.858+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:49.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:49.840+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:49.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:49.840+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:49.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:49.840+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:49.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:49.841+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:49.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:49.599+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:49.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:49.599+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:49.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:49.599+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:49.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:49.600+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:50.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:49.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:50.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:49.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:50.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:49.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:50.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:49.932+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:50.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:50.555+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:50.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:50.555+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:50.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:50.555+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:50.867 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:50.555+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:51.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:50.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:51.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:50.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:51.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:50.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:51.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:50.963+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:51.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:50.859+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:51.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:50.859+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:51.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:50.859+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:51.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:50.859+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:51.776 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:24:51.468+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 13662 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:24:51.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:51.547+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:51.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:51.547+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:51.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:51.547+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:51.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:51.548+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:52.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:51.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:52.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:51.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:52.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:51.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:52.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:51.855+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:52.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:51.918+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:52.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:51.918+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:52.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:51.918+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:52.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:51.918+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:52.887 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:52.552+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:52.887 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:52.552+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:52.887 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:52.552+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:52.887 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:52.552+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:53.241 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:52.905+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:53.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:52.905+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:53.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:52.905+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:53.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:52.905+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:53.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:52.878+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:53.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:52.878+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:53.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:52.878+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:53.243 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:52.879+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:53.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:53.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:53.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:53.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:53.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:53.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:53.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:53.535+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:54.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:53.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:54.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:53.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:54.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:53.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:54.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:53.923+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:54.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:53.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:54.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:53.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:54.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:53.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:54.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:53.874+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:54.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:54.510+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:54.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:54.510+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:54.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:54.510+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:54.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:54.510+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:55.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:54.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:55.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:54.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:55.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:54.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:55.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:54.912+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:55.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:54.961+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:55.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:54.961+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:55.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:54.961+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:55.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:54.962+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:55.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:55.486+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:55.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:55.486+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:55.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:55.486+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:55.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:55.487+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:56.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:55.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:56.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:55.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:56.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:55.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:56.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:55.963+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:56.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:55.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:56.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:55.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:56.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:55.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:56.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:55.963+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:56.600 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:24:56.470+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 13758 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:24:56.601 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:56.465+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:56.601 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:56.465+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:56.601 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:56.465+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:56.601 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:56.466+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:57.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:56.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:57.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:56.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:57.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:56.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:57.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:56.960+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:57.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:56.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:57.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:56.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:57.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:56.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:57.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:56.996+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:57.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:57.447+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:57.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:57.447+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:57.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:57.447+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:57.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:57.448+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:58.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:57.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:58.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:57.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:58.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:57.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:58.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:57.999+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:58.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:57.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:58.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:57.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:58.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:57.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:58.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:57.960+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:58.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:58.433+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:58.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:58.433+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:58.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:58.433+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:58.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:58.434+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:24:59.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:59.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:59.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:59.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:59.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:59.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:24:59.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:24:59.015+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:59.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:58.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:59.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:58.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:59.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:58.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:24:59.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:58.942+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:24:59.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:59.387+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:59.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:59.387+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:59.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:59.388+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:24:59.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:24:59.388+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:00.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:59.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:00.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:59.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:00.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:59.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:00.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:24:59.907+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:00.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:00.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:00.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:00.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:00.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:00.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:00.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:00.020+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:00.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:00.420+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:00.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:00.420+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:00.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:00.420+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:00.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:00.420+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:01.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:01.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:01.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:01.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:01.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:01.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:01.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:01.038+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:01.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:00.920+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:01.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:00.920+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:01.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:00.920+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:01.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:00.921+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:01.776 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:25:01.471+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 13855 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:25:01.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:01.370+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:01.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:01.370+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:01.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:01.370+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:01.777 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:01.372+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:02.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:02.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:02.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:02.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:02.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:02.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:02.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:02.046+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:02.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:01.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:02.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:01.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:02.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:01.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:02.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:01.879+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:02.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:02.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:02.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:02.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:02.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:02.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:02.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:02.345+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:03.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:03.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:03.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:03.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:03.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:03.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:03.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:03.050+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:03.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:02.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:03.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:02.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:03.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:02.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:03.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:02.911+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:03.587 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:03.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:03.588 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:03.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:03.588 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:03.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:03.588 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:03.328+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:04.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:04.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:04.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:04.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:04.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:04.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:04.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:04.046+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:04.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:03.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:04.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:03.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:04.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:03.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:04.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:03.937+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:04.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:04.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:04.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:04.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:04.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:04.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:04.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:04.324+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:05.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:04.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:05.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:04.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:05.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:04.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:05.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:04.973+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:05.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:05.047+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:05.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:05.047+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:05.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:05.047+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:05.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:05.048+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:05.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:05.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:05.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:05.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:05.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:05.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:05.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:05.318+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:06.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:05.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:06.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:05.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:06.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:05.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:06.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:05.974+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:06.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:06.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:06.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:06.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:06.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:06.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:06.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:06.057+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:06.775 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:25:06.473+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 13951 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:25:06.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:06.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:06.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:06.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:06.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:06.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:06.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:06.349+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:07.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:06.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:07.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:06.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:07.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:06.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:07.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:06.970+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:07.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:07.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:07.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:07.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:07.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:07.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:07.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:07.084+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:07.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:07.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:07.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:07.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:07.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:07.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:07.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:07.323+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:08.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:08.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:08.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:08.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:08.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:08.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:08.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:08.091+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:08.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:07.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:08.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:07.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:08.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:07.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:08.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:07.926+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:08.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:08.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:08.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:08.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:08.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:08.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:08.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:08.367+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:09.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:09.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:09.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:09.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:09.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:09.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:09.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:09.044+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:09.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:08.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:09.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:08.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:09.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:08.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:09.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:08.957+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:09.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:09.405+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:09.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:09.405+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:09.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:09.405+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:09.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:09.405+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:10.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:10.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:10.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:10.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:10.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:10.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:10.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:10.068+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:10.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:09.947+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:10.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:09.947+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:10.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:09.947+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:10.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:09.948+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:10.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:10.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:10.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:10.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:10.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:10.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:10.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:10.433+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:11.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:10.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:11.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:10.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:11.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:10.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:11.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:10.992+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:11.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:11.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:11.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:11.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:11.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:11.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:11.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:11.100+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:11.673 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:25:11.475+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 14047 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:25:11.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:11.383+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:11.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:11.383+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:11.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:11.383+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:11.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:11.384+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:12.353 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:12.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:12.353 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:12.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:12.353 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:12.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:12.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:12.021+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:12.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:12.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:12.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:12.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:12.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:12.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:12.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:12.117+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:12.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:12.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:12.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:12.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:12.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:12.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:12.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:12.347+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:13.353 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:13.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:13.353 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:13.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:13.353 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:13.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:13.353 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:13.048+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:13.353 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:13.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:13.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:13.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:13.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:13.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:13.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:13.122+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:13.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:13.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:13.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:13.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:13.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:13.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:13.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:13.346+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:14.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:14.070+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:14.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:14.070+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:14.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:14.070+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:14.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:14.070+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:14.341 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:14.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:14.342 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:14.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:14.342 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:14.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:14.342 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:14.077+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:14.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:14.334+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:14.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:14.334+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:14.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:14.334+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:14.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:14.334+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:15.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:15.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:15.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:15.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:15.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:15.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:15.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:15.055+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:15.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:15.286+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:15.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:15.286+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:15.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:15.286+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:15.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:15.287+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:15.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:15.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:15.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:15.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:15.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:15.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:15.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:15.056+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:16.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:16.256+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:16.484 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:16.256+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:16.484 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:16.256+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:16.484 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:16.257+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:16.484 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:16.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:16.485 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:16.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:16.485 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:16.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:16.485 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:16.076+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:16.485 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:16.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:16.485 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:16.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:16.485 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:16.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:16.486 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:16.044+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:16.775 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:25:16.476+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 14143 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:25:17.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:17.267+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:17.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:17.267+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:17.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:17.267+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:17.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:17.267+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:17.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:17.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:17.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:17.122+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:17.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:17.122+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:17.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:17.123+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:17.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:17.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:17.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:17.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:17.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:17.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:17.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:17.078+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:18.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:18.226+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:18.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:18.226+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:18.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:18.226+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:18.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:18.227+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:18.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:18.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:18.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:18.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:18.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:18.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:18.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:18.033+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:18.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:18.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:18.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:18.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:18.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:18.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:18.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:18.163+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:19.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:19.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:19.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:19.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:19.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:19.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:19.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:19.259+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:19.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:19.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:19.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:19.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:19.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:19.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:19.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:19.035+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:19.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:19.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:19.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:19.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:19.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:19.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:19.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:19.180+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:20.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:19.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:20.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:19.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:20.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:19.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:20.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:19.996+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:20.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:20.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:20.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:20.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:20.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:20.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:20.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:20.219+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:20.277 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:20.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:20.277 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:20.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:20.277 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:20.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:20.277 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:20.225+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:21.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:20.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:21.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:20.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:21.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:20.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:21.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:20.985+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:21.486 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:21.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:21.486 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:21.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:21.486 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:21.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:21.486 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:21.175+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:21.486 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:21.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:21.487 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:21.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:21.487 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:21.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:21.487 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:21.176+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:21.775 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:25:21.478+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 14239 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:25:22.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:22.207+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:22.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:22.207+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:22.276 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:22.207+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:22.276 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:22.208+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:22.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:22.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:22.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:22.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:22.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:22.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:22.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:22.137+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:22.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:22.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:22.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:22.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:22.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:22.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:22.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:22.014+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:23.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:23.195+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:23.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:23.195+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:23.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:23.195+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:23.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:23.196+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:23.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:23.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:23.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:23.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:23.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:23.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:23.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:23.047+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:23.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:23.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:23.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:23.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:23.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:23.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:23.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:23.115+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:24.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:24.097+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:24.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:24.097+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:24.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:24.097+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:24.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:24.098+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:24.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:24.057+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:24.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:24.057+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:24.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:24.057+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:24.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:24.058+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:24.277 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:24.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:24.277 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:24.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:24.277 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:24.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:24.277 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:24.200+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:25.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:25.071+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:25.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:25.071+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:25.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:25.071+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:25.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:25.072+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:25.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:25.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:25.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:25.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:25.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:25.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:25.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:25.025+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:25.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:25.240+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:25.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:25.240+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:25.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:25.240+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:25.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:25.241+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:26.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:26.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:26.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:26.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:26.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:26.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:26.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:26.246+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:26.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:26.111+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:26.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:26.111+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:26.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:26.111+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:26.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:26.112+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:26.428 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:26.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:26.429 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:26.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:26.429 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:26.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:26.429 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:26.075+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:26.775 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:25:26.480+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 14335 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:25:27.296 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:27.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:27.296 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:27.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:27.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:27.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:27.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:27.078+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:27.297 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:27.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:27.297 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:27.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:27.297 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:27.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:27.297 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:27.038+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:27.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:27.289+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:27.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:27.289+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:27.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:27.289+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:27.776 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:27.290+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:28.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:28.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:28.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:28.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:28.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:28.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:28.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:28.113+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:28.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:28.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:28.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:28.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:28.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:28.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:28.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:28.252+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:28.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:28.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:28.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:28.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:28.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:28.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:28.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:28.083+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:29.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:29.225+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:29.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:29.225+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:29.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:29.225+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:29.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:29.226+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:29.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:29.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:29.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:29.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:29.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:29.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:29.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:29.118+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:29.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:29.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:29.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:29.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:29.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:29.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:29.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:29.044+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:30.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:30.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:30.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:30.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:30.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:30.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:30.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:30.069+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:30.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:30.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:30.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:30.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:30.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:30.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:30.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:30.149+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:30.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:30.246+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:30.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:30.246+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:30.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:30.246+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:30.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:30.246+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:31.489 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:31.284+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:31.489 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:31.284+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:31.489 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:31.284+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:31.489 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:31.284+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:31.489 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:31.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:31.489 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:31.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:31.490 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:31.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:31.490 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:31.173+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:31.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:31.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:31.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:31.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:31.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:31.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:31.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:31.082+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:31.775 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:25:31.481+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 14431 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:25:32.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:32.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:32.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:32.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:32.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:32.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:32.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:32.069+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:32.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:32.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:32.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:32.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:32.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:32.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:32.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:32.158+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:32.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:32.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:32.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:32.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:32.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:32.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:32.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:32.272+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:33.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:33.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:33.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:33.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:33.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:33.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:33.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:33.151+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:33.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:33.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:33.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:33.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:33.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:33.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:33.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:33.063+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:33.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:33.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:33.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:33.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:33.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:33.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:33.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:33.257+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:34.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:34.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:34.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:34.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:34.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:34.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:34.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:34.095+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:34.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:34.209+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:34.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:34.209+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:34.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:34.209+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:34.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:34.209+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:34.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:34.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:34.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:34.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:34.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:34.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:34.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:34.137+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:35.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:35.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:35.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:35.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:35.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:35.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:35.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:35.170+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:35.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:35.214+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:35.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:35.214+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:35.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:35.214+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:35.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:35.215+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:35.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:35.094+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:35.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:35.094+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:35.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:35.094+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:35.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:35.094+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:36.490 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:36.240+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:36.490 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:36.240+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:36.490 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:36.240+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:36.490 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:36.240+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:36.491 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:36.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:36.491 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:36.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:36.491 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:36.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:36.491 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:36.203+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:36.491 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:36.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:36.491 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:36.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:36.492 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:36.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:36.492 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:36.115+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:36.775 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:25:36.483+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 14526 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:25:37.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:37.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:37.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:37.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:37.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:37.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:37.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:37.089+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:37.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:37.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:37.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:37.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:37.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:37.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:37.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:37.210+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:37.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:37.265+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:37.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:37.265+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:37.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:37.265+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:37.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:37.266+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:38.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:38.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:38.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:38.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:38.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:38.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:38.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:38.065+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:38.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:38.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:38.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:38.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:38.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:38.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:38.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:38.252+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:38.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:38.183+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:38.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:38.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:38.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:38.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:38.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:38.184+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:39.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:39.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:39.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:39.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:39.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:39.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:39.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:39.109+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:39.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:39.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:39.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:39.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:39.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:39.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:39.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:39.206+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:39.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:39.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:39.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:39.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:39.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:39.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:39.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:39.297+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:40.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:40.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:40.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:40.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:40.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:40.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:40.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:40.128+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:40.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:40.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:40.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:40.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:40.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:40.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:40.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:40.222+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:40.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:40.270+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:40.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:40.270+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:40.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:40.270+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:40.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:40.271+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:41.492 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:41.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:41.492 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:41.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:41.492 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:41.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:41.492 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:41.237+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:41.493 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:41.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:41.493 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:41.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:41.493 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:41.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:41.493 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:41.317+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:41.493 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:41.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:41.493 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:41.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:41.494 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:41.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:41.494 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:41.140+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:41.775 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:25:41.485+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 14617 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:25:42.393 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:42.295+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:42.394 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:42.295+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:42.394 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:42.295+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:42.394 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:42.296+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:42.394 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:42.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:42.395 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:42.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:42.395 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:42.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:42.395 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:42.130+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:42.395 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:42.223+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:42.395 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:42.223+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:42.395 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:42.223+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:42.395 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:42.224+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:43.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:43.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:43.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:43.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:43.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:43.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:43.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:43.214+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:43.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:43.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:43.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:43.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:43.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:43.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:43.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:43.320+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:43.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:43.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:43.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:43.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:43.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:43.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:43.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:43.126+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:44.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:44.171+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:44.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:44.171+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:44.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:44.171+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:44.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:44.172+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:44.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:44.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:44.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:44.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:44.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:44.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:44.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:44.333+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:44.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:44.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:44.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:44.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:44.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:44.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:44.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:44.104+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:45.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:45.134+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:45.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:45.134+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:45.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:45.134+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:45.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:45.135+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:45.289+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:45.289+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:45.289+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:45.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:45.290+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:45.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:45.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:45.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:45.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:45.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:45.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:45.060+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:46.304 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:46.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:46.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:46.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:46.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:46.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:46.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:46.037+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:46.305 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:46.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:46.305 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:46.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:46.305 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:46.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:46.306 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:46.160+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:46.615 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:25:46.486+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 14707 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:25:46.615 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:46.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:46.615 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:46.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:46.616 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:46.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:46.616 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:46.298+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:47.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:47.207+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:47.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:47.207+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:47.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:47.207+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:47.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:47.208+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:47.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:47.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:47.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:47.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:47.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:47.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:47.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:47.264+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:47.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:47.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:47.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:47.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:47.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:47.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:47.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:47.026+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:48.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:48.248+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:48.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:48.248+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:48.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:48.248+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:48.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:48.248+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:48.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:48.242+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:48.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:48.242+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:48.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:48.242+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:48.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:48.243+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:48.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:48.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:48.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:48.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:48.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:48.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:48.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:48.029+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:49.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:49.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:49.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:49.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:49.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:49.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:49.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:49.072+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:49.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:49.274+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:49.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:49.274+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:49.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:49.274+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:49.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:49.275+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:49.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:49.261+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:49.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:49.261+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:49.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:49.261+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:49.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:49.262+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:50.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:50.233+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:50.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:50.233+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:50.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:50.233+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:50.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:50.234+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:50.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:50.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:50.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:50.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:50.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:50.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:50.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:50.053+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:50.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:50.238+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:50.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:50.238+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:50.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:50.238+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:50.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:50.239+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:51.495 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:51.278+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:51.496 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:51.278+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:51.496 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:51.278+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:51.496 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:51.279+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:51.496 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:51.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:51.496 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:51.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:51.497 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:51.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:51.497 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:51.214+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:51.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:51.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:51.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:51.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:51.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:51.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:51.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:51.051+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:51.775 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:25:51.488+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 14797 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:25:52.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:52.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:52.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:52.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:52.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:52.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:52.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:52.170+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:52.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:52.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:52.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:52.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:52.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:52.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:52.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:52.299+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:52.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:52.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:52.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:52.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:52.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:52.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:52.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:52.061+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:53.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:53.132+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:53.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:53.132+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:53.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:53.132+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:53.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:53.132+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:53.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:53.276+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:53.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:53.276+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:53.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:53.276+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:53.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:53.277+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:53.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:53.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:53.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:53.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:53.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:53.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:53.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:53.088+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:54.186 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:54.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:54.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:54.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:54.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:54.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:54.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:54.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:54.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:54.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:54.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:54.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:54.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:54.117+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:54.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:54.180+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:54.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:54.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:54.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:54.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:54.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:54.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:54.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:54.274+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:55.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:55.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:55.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:55.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:55.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:55.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:55.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:55.180+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:55.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:55.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:55.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:55.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:55.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:55.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:55.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:55.116+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:55.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:55.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:55.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:55.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:55.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:55.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:55.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:55.228+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:56.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:56.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:56.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:56.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:56.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:56.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:56.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:56.220+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:56.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:56.262+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:56.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:56.262+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:56.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:56.262+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:56.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:56.262+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:56.428 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:56.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:56.428 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:56.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:56.428 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:56.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:56.429 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:56.136+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:56.775 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:25:56.490+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 14887 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:25:57.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:57.243+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:57.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:57.243+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:57.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:57.243+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:57.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:57.244+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:57.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:57.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:57.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:57.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:57.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:57.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:57.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:57.257+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:57.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:57.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:57.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:57.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:57.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:57.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:57.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:57.139+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:58.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:58.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:58.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:58.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:58.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:58.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:58.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:58.146+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:58.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:58.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:58.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:58.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:58.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:58.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:58.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:58.226+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:58.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:58.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:58.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:58.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:58.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:58.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:58.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:58.278+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:25:59.241 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:59.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:59.241 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:59.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:59.241 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:59.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:25:59.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:25:59.185+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:59.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:59.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:59.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:59.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:59.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:59.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:25:59.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:25:59.190+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:25:59.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:59.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:59.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:59.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:59.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:59.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:25:59.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:25:59.236+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:00.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:00.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:00.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:00.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:00.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:00.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:00.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:00.235+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:00.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:00.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:00.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:00.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:00.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:00.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:00.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:00.180+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:00.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:00.217+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:00.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:00.217+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:00.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:00.217+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:00.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:00.217+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:01.498 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:01.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:01.498 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:01.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:01.498 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:01.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:01.499 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:01.146+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:01.499 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:01.246+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:01.499 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:01.246+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:01.499 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:01.246+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:01.499 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:01.247+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:01.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:01.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:01.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:01.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:01.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:01.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:01.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:01.192+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:01.775 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:26:01.491+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 14977 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:26:02.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:02.218+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:02.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:02.218+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:02.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:02.218+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:02.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:02.219+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:02.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:02.222+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:02.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:02.222+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:02.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:02.222+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:02.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:02.223+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:02.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:02.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:02.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:02.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:02.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:02.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:02.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:02.146+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:03.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:03.122+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:03.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:03.122+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:03.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:03.122+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:03.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:03.122+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:03.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:03.207+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:03.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:03.207+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:03.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:03.207+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:03.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:03.207+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:03.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:03.196+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:03.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:03.196+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:03.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:03.196+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:03.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:03.196+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:04.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:04.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:04.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:04.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:04.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:04.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:04.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:04.144+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:04.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:04.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:04.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:04.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:04.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:04.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:04.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:04.149+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:04.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:04.242+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:04.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:04.242+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:04.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:04.242+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:04.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:04.243+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:05.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:05.115+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:05.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:05.115+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:05.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:05.115+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:05.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:05.116+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:05.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:05.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:05.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:05.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:05.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:05.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:05.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:05.175+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:05.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:05.265+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:05.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:05.265+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:05.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:05.265+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:05.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:05.265+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:06.500 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:06.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:06.500 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:06.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:06.500 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:06.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:06.500 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:06.171+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:06.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:06.160+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:06.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:06.160+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:06.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:06.160+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:06.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:06.160+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:06.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:06.245+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:06.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:06.245+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:06.502 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:06.245+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:06.502 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:06.246+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:06.774 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:26:06.493+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 15066 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:26:07.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:07.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:07.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:07.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:07.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:07.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:07.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:07.136+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:07.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:07.218+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:07.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:07.218+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:07.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:07.218+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:07.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:07.219+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:07.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:07.287+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:07.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:07.287+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:07.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:07.287+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:07.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:07.287+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:08.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:08.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:08.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:08.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:08.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:08.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:08.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:08.173+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:08.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:08.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:08.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:08.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:08.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:08.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:08.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:08.217+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:08.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:08.258+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:08.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:08.258+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:08.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:08.258+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:08.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:08.259+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:09.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:09.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:09.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:09.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:09.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:09.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:09.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:09.155+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:09.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:09.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:09.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:09.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:09.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:09.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:09.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:09.212+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:09.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:09.230+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:09.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:09.230+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:09.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:09.230+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:09.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:09.230+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:10.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:10.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:10.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:10.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:10.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:10.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:10.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:10.173+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:10.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:10.207+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:10.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:10.207+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:10.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:10.207+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:10.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:10.207+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:10.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:10.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:10.528 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:10.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:10.528 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:10.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:10.528 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:10.260+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:11.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:11.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:11.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:11.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:11.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:11.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:11.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:11.151+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:11.502 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:11.244+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:11.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:11.244+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:11.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:11.244+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:11.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:11.244+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:11.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:11.186+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:11.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:11.186+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:11.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:11.186+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:11.504 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:11.186+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:11.775 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:26:11.495+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 15156 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:26:12.485 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:12.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:12.485 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:12.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:12.485 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:12.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:12.485 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:12.178+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:12.486 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:12.268+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:12.486 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:12.268+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:12.486 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:12.268+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:12.486 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:12.268+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:12.486 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:12.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:12.487 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:12.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:12.487 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:12.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:12.487 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:12.180+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:13.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:13.224+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:13.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:13.224+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:13.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:13.224+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:13.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:13.225+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:13.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:13.252+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:13.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:13.252+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:13.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:13.252+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:13.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:13.253+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:13.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:13.162+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:13.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:13.162+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:13.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:13.162+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:13.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:13.163+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:14.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:14.187+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:14.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:14.187+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:14.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:14.187+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:14.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:14.188+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:14.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:14.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:14.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:14.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:14.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:14.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:14.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:14.185+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:14.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:14.301+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:14.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:14.301+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:14.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:14.301+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:14.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:14.302+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:15.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:15.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:15.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:15.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:15.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:15.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:15.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:15.145+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:15.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:15.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:15.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:15.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:15.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:15.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:15.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:15.158+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:15.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:15.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:15.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:15.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:15.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:15.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:15.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:15.259+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:16.503 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:16.166+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:16.503 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:16.166+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:16.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:16.167+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:16.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:16.167+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:16.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:16.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:16.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:16.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:16.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:16.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:16.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:16.309+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:16.505 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:16.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:16.505 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:16.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:16.505 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:16.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:16.505 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:16.159+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:16.774 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:26:16.496+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 15246 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:26:17.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:17.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:17.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:17.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:17.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:17.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:17.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:17.142+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:17.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:17.132+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:17.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:17.132+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:17.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:17.132+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:17.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:17.133+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:17.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:17.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:17.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:17.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:17.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:17.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:17.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:17.275+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:18.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:18.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:18.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:18.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:18.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:18.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:18.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:18.150+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:18.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:18.246+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:18.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:18.246+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:18.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:18.246+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:18.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:18.247+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:18.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:18.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:18.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:18.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:18.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:18.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:18.528 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:18.125+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:19.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:19.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:19.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:19.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:19.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:19.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:19.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:19.080+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:19.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:19.129+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:19.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:19.129+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:19.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:19.129+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:19.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:19.129+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:19.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:19.212+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:19.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:19.212+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:19.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:19.212+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:19.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:19.213+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:20.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:20.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:20.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:20.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:20.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:20.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:20.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:20.065+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:20.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:20.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:20.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:20.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:20.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:20.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:20.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:20.178+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:20.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:20.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:20.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:20.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:20.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:20.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:20.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:20.115+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:21.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:21.094+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:21.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:21.094+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:21.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:21.094+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:21.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:21.095+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:21.506 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:21.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:21.506 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:21.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:21.506 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:21.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:21.506 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:21.142+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:21.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:21.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:21.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:21.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:21.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:21.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:21.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:21.096+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:21.774 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:26:21.497+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 15337 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:26:22.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:22.080+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:22.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:22.080+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:22.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:22.080+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:22.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:22.081+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:22.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:22.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:22.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:22.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:22.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:22.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:22.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:22.113+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:22.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:22.120+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:22.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:22.120+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:22.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:22.120+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:22.527 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:22.120+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:23.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:23.082+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:23.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:23.082+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:23.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:23.082+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:23.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:23.083+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:23.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:23.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:23.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:23.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:23.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:23.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:23.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:23.158+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:23.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:23.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:23.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:23.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:23.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:23.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:23.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:23.146+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:24.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:24.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:24.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:24.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:24.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:24.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:24.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:24.142+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:24.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:24.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:24.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:24.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:24.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:24.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:24.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:24.118+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:24.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:24.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:24.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:24.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:24.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:24.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:24.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:24.194+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:25.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:25.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:25.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:25.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:25.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:25.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:25.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:25.099+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:25.401 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:25.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:25.401 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:25.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:25.401 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:25.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:25.402 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:25.133+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:25.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:25.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:25.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:25.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:25.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:25.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:25.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:25.210+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:26.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:26.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:26.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:26.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:26.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:26.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:26.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:26.059+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:26.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:26.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:26.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:26.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:26.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:26.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:26.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:26.170+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:26.511 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:26.238+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:26.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:26.238+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:26.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:26.238+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:26.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:26.239+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:26.774 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:26:26.500+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 15427 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:26:27.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:27.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:27.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:27.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:27.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:27.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:27.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:27.199+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:27.276 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:27.017+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:27.276 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:27.017+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:27.276 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:27.017+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:27.276 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:27.018+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:27.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:27.217+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:27.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:27.217+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:27.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:27.217+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:27.277 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:27.217+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:28.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:28.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:28.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:28.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:28.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:28.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:28.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:28.206+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:28.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:28.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:28.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:28.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:28.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:28.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:28.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:28.209+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:28.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:28.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:28.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:28.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:28.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:28.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:28.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:28.064+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:29.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:29.176+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:29.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:29.176+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:29.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:29.176+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:29.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:29.178+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:29.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:29.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:29.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:29.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:29.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:29.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:29.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:29.036+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:29.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:29.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:29.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:29.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:29.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:29.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:29.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:29.211+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:30.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:30.183+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:30.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:30.183+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:30.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:30.183+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:30.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:30.183+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:30.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:30.053+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:30.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:30.053+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:30.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:30.053+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:30.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:30.054+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:30.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:30.183+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:30.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:30.183+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:30.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:30.183+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:30.527 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:30.184+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:31.508 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:31.081+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:31.508 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:31.081+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:31.508 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:31.081+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:31.509 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:31.082+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:31.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:31.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:31.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:31.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:31.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:31.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:31.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:31.198+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:31.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:31.169+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:31.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:31.169+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:31.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:31.169+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:31.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:31.169+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:31.774 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:26:31.502+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 15517 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:26:32.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:32.224+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:32.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:32.224+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:32.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:32.224+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:32.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:32.225+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:32.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:32.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:32.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:32.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:32.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:32.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:32.526 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:32.057+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:32.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:32.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:32.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:32.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:32.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:32.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:32.527 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:32.137+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:33.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:33.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:33.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:33.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:33.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:33.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:33.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:33.010+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:33.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:33.117+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:33.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:33.117+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:33.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:33.117+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:33.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:33.118+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:33.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:33.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:33.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:33.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:33.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:33.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:33.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:33.266+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:34.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:34.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:34.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:34.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:34.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:34.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:34.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:34.035+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:34.311 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:34.279+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:34.311 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:34.279+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:34.311 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:34.279+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:34.311 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:34.280+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:34.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:34.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:34.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:34.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:34.312 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:34.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:34.312 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:34.081+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:35.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:35.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:35.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:35.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:35.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:35.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:35.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:35.125+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:35.276 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:34.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:35.276 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:34.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:35.277 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:34.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:35.277 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:34.991+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:35.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:35.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:35.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:35.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:35.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:35.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:35.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:35.310+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:36.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:36.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:36.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:36.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:36.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:36.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:36.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:36.152+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:36.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:35.976+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:36.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:35.976+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:36.276 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:35.976+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:36.276 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:35.976+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:36.632 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:26:36.503+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 15638 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:26:36.632 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:36.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:36.632 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:36.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:36.633 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:36.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:36.633 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:36.350+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:37.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:37.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:37.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:37.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:37.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:37.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:37.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:37.164+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:37.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:36.971+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:37.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:36.971+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:37.276 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:36.971+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:37.276 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:36.972+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:37.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:37.368+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:37.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:37.368+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:37.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:37.368+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:37.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:37.368+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:38.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:38.169+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:38.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:38.169+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:38.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:38.169+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:38.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:38.170+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:38.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:37.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:38.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:37.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:38.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:37.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:38.276 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:37.962+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:38.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:38.348+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:38.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:38.348+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:38.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:38.348+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:38.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:38.349+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:39.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:39.173+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:39.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:39.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:39.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:39.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:39.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:39.174+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:39.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:38.942+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:39.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:38.942+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:39.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:38.942+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:39.276 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:38.943+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:39.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:39.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:39.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:39.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:39.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:39.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:39.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:39.350+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:40.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:39.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:40.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:39.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:40.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:39.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:40.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:39.894+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:40.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:40.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:40.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:40.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:40.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:40.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:40.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:40.208+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:40.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:40.399+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:40.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:40.399+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:40.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:40.399+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:40.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:40.399+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:41.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:40.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:41.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:40.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:41.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:40.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:41.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:40.902+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:41.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:41.380+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:41.512 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:41.380+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:41.512 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:41.380+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:41.512 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:41.381+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:41.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:41.228+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:41.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:41.228+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:41.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:41.228+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:41.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:41.229+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:41.774 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:26:41.505+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 15730 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:26:42.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:41.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:42.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:41.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:42.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:41.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:42.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:41.880+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:42.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:42.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:42.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:42.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:42.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:42.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:42.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:42.197+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:42.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:42.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:42.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:42.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:42.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:42.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:42.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:42.345+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:43.220 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:42.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:43.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:42.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:43.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:42.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:43.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:42.924+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:43.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:43.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:43.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:43.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:43.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:43.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:43.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:43.309+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:43.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:43.214+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:43.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:43.214+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:43.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:43.214+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:43.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:43.215+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:44.242 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:43.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:44.242 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:43.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:44.242 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:43.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:44.243 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:43.925+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:44.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:44.345+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:44.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:44.345+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:44.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:44.345+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:44.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:44.347+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:44.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:44.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:44.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:44.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:44.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:44.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:44.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:44.236+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:45.228 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:44.890+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:45.228 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:44.890+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:45.228 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:44.890+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:45.229 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:44.891+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:45.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:45.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:45.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:45.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:45.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:45.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:45.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:45.396+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:45.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:45.222+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:45.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:45.222+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:45.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:45.222+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:45.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:45.222+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:46.228 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:45.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:46.228 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:45.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:46.229 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:45.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:46.229 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:45.917+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:46.513 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:46.387+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:46.513 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:46.387+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:46.513 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:46.387+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:46.514 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:46.388+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:46.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:46.222+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:46.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:46.222+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:46.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:46.222+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:46.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:46.223+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:46.774 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:26:46.507+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 15820 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:26:47.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:46.928+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:47.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:46.928+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:47.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:46.928+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:47.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:46.929+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:47.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:47.391+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:47.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:47.391+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:47.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:47.391+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:47.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:47.392+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:47.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:47.199+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:47.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:47.199+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:47.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:47.199+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:47.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:47.199+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:48.197 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:47.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:48.197 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:47.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:48.197 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:47.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:48.197 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:47.881+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:48.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:48.389+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:48.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:48.389+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:48.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:48.389+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:48.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:48.390+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:48.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:48.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:48.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:48.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:48.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:48.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:48.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:48.191+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:49.230 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:48.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:49.230 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:48.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:49.231 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:48.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:49.231 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:48.870+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:49.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:49.402+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:49.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:49.402+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:49.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:49.402+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:49.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:49.403+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:49.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:49.224+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:49.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:49.224+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:49.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:49.224+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:49.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:49.224+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:50.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:49.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:50.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:49.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:50.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:49.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:50.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:49.833+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:50.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:50.362+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:50.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:50.362+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:50.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:50.362+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:50.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:50.363+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:50.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:50.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:50.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:50.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:50.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:50.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:50.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:50.216+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:51.251 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:50.865+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:51.252 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:50.865+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:51.252 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:50.865+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:51.252 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:50.866+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:51.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:51.327+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:51.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:51.327+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:51.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:51.327+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:51.516 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:51.328+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:51.516 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:51.245+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:51.516 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:51.245+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:51.516 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:51.245+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:51.516 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:51.246+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:51.774 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:26:51.509+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 15910 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:26:52.225 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:51.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:52.225 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:51.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:52.225 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:51.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:52.225 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:51.847+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:52.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:52.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:52.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:52.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:52.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:52.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:52.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:52.373+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:52.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:52.219+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:52.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:52.219+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:52.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:52.219+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:52.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:52.219+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:53.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:52.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:53.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:52.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:53.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:52.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:53.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:52.818+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:53.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:53.351+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:53.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:53.351+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:53.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:53.351+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:53.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:53.352+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:53.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:53.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:53.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:53.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:53.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:53.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:53.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:53.171+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:54.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:53.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:54.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:53.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:54.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:53.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:54.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:53.791+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:54.496 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:54.379+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:54.497 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:54.379+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:54.497 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:54.379+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:54.497 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:54.380+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:54.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:54.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:54.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:54.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:54.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:54.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:54.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:54.196+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:55.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:54.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:55.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:54.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:55.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:54.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:55.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:54.761+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:55.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:55.223+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:55.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:55.223+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:55.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:55.223+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:55.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:55.223+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:55.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:55.348+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:55.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:55.348+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:55.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:55.348+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:55.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:55.349+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:56.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:55.783+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:56.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:55.783+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:56.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:55.783+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:56.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:55.784+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:56.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:56.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:56.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:56.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:56.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:56.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:56.243 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:56.235+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:56.638 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:26:56.510+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 16000 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:26:56.639 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:56.379+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:56.639 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:56.379+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:56.639 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:56.379+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:56.639 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:56.380+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:57.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:56.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:57.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:56.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:57.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:56.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:57.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:56.766+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:57.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:57.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:57.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:57.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:57.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:57.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:57.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:57.193+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:57.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:57.412+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:57.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:57.412+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:57.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:57.412+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:57.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:57.413+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:58.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:57.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:58.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:57.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:58.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:57.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:58.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:57.767+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:58.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:58.414+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:58.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:58.414+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:58.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:58.414+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:58.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:58.415+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:58.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:58.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:58.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:58.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:58.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:58.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:58.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:58.216+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:59.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:58.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:59.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:58.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:59.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:58.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:26:59.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:58.760+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:26:59.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:59.250+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:59.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:59.250+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:59.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:59.250+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:26:59.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:26:59.251+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:26:59.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:59.438+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:59.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:59.438+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:59.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:59.438+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:26:59.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:26:59.438+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:00.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:59.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:00.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:59.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:00.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:59.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:00.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:26:59.784+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:00.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:00.223+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:00.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:00.223+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:00.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:00.223+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:00.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:00.223+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:00.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:00.444+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:00.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:00.444+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:00.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:00.444+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:00.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:00.445+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:01.189 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:00.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:01.190 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:00.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:01.190 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:00.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:01.190 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:00.821+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:01.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:01.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:01.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:01.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:01.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:01.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:01.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:01.184+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:01.774 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:27:01.512+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 16090 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:27:01.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:01.449+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:01.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:01.449+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:01.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:01.449+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:01.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:01.450+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:02.224 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:01.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:02.224 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:01.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:02.224 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:01.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:02.224 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:01.841+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:02.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:02.218+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:02.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:02.218+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:02.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:02.218+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:02.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:02.218+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:02.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:02.441+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:02.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:02.441+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:02.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:02.441+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:02.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:02.442+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:03.237 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:02.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:03.237 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:02.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:03.237 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:02.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:03.237 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:02.848+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:03.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:03.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:03.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:03.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:03.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:03.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:03.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:03.232+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:03.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:03.411+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:03.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:03.411+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:03.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:03.411+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:03.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:03.412+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:04.239 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:03.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:04.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:03.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:04.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:03.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:04.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:03.849+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:04.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:04.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:04.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:04.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:04.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:04.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:04.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:04.441+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:04.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:04.233+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:04.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:04.233+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:04.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:04.233+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:04.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:04.234+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:05.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:04.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:05.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:04.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:05.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:04.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:05.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:04.850+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:05.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:05.203+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:05.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:05.203+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:05.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:05.203+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:05.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:05.203+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:05.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:05.426+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:05.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:05.426+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:05.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:05.426+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:05.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:05.427+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:06.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:05.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:06.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:05.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:06.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:05.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:06.196 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:05.804+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:06.473 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:06.189+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:06.473 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:06.189+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:06.473 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:06.189+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:06.473 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:06.189+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:06.774 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:27:06.514+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 16178 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:27:06.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:06.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:06.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:06.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:06.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:06.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:06.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:06.468+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:07.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:06.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:07.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:06.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:07.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:06.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:07.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:06.771+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:07.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:07.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:07.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:07.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:07.508 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:07.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:07.508 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:07.153+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:07.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:07.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:07.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:07.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:07.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:07.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:07.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:07.502+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:08.138 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:07.820+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:08.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:07.820+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:08.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:07.820+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:08.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:07.820+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:08.471 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:08.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:08.472 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:08.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:08.472 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:08.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:08.472 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:08.133+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:08.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:08.466+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:08.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:08.466+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:08.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:08.466+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:08.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:08.466+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:09.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:08.808+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:09.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:08.808+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:09.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:08.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:09.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:08.809+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:09.430 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:09.120+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:09.431 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:09.120+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:09.431 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:09.120+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:09.431 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:09.121+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:09.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:09.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:09.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:09.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:09.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:09.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:09.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:09.426+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:10.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:09.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:10.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:09.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:10.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:09.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:10.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:09.836+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:10.475 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:10.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:10.475 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:10.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:10.475 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:10.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:10.476 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:10.144+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:10.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:10.469+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:10.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:10.469+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:10.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:10.469+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:10.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:10.470+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:11.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:10.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:11.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:10.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:11.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:10.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:11.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:10.872+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:11.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:11.165+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:11.441 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:11.165+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:11.441 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:11.165+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:11.441 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:11.166+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:11.774 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:27:11.516+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 16267 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:27:11.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:11.435+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:11.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:11.435+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:11.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:11.435+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:11.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:11.436+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:12.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:11.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:12.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:11.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:12.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:11.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:12.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:11.830+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:12.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:12.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:12.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:12.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:12.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:12.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:12.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:12.419+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:12.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:12.195+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:12.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:12.195+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:12.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:12.195+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:12.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:12.196+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:13.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:12.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:13.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:12.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:13.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:12.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:13.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:12.853+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:13.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:13.162+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:13.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:13.162+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:13.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:13.162+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:13.475 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:13.162+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:13.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:13.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:13.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:13.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:13.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:13.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:13.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:13.469+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:14.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:13.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:14.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:13.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:14.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:13.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:14.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:13.888+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:14.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:14.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:14.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:14.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:14.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:14.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:14.466 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:14.143+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:14.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:14.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:14.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:14.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:14.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:14.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:14.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:14.460+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:15.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:14.903+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:15.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:14.903+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:15.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:14.903+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:15.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:14.904+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:15.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:15.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:15.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:15.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:15.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:15.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:15.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:15.133+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:15.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:15.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:15.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:15.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:15.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:15.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:15.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:15.507+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:16.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:16.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:16.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:16.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:16.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:16.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:16.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:16.095+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:16.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:15.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:16.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:15.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:16.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:15.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:16.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:15.906+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:16.906 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:27:16.518+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 16356 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:27:16.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:16.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:16.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:16.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:16.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:16.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:16.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:16.545+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:17.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:17.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:17.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:17.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:17.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:17.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:17.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:17.056+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:17.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:16.900+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:17.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:16.900+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:17.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:16.900+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:17.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:16.901+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:17.857 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:17.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:17.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:17.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:17.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:17.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:17.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:17.592+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:18.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:18.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:18.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:18.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:18.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:18.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:18.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:18.028+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:18.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:17.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:18.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:17.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:18.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:17.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:18.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:17.852+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:18.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:18.603+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:18.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:18.603+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:18.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:18.603+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:18.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:18.604+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:19.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:18.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:19.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:18.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:19.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:18.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:19.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:18.982+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:19.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:18.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:19.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:18.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:19.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:18.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:19.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:18.876+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:19.898 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:19.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:19.898 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:19.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:19.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:19.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:19.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:19.610+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:20.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:19.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:20.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:19.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:20.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:19.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:20.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:19.894+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:20.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:19.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:20.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:19.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:20.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:19.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:20.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:19.976+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:20.913 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:20.640+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:20.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:20.641+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:20.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:20.641+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:20.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:20.641+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:21.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:20.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:21.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:20.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:21.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:20.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:21.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:20.909+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:21.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:21.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:21.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:21.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:21.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:21.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:21.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:21.011+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:21.929 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:27:21.520+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 16446 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:27:21.930 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:21.608+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:21.930 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:21.608+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:21.930 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:21.608+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:21.930 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:21.608+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:22.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:21.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:22.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:21.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:22.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:21.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:22.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:21.925+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:22.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:21.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:22.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:21.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:22.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:21.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:22.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:21.985+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:22.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:22.600+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:22.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:22.601+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:22.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:22.601+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:22.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:22.601+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:23.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:22.903+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:23.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:22.903+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:23.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:22.903+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:23.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:22.904+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:23.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:22.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:23.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:22.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:23.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:22.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:23.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:22.957+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:23.949 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:23.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:23.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:23.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:23.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:23.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:23.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:23.605+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:24.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:23.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:24.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:23.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:24.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:23.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:24.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:23.986+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:24.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:23.944+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:24.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:23.944+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:24.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:23.944+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:24.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:23.945+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:24.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:24.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:24.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:24.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:24.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:24.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:24.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:24.618+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:25.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:24.971+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:25.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:24.971+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:25.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:24.971+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:25.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:24.972+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:25.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:24.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:25.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:24.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:25.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:24.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:25.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:24.957+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:25.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:25.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:25.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:25.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:25.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:25.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:25.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:25.587+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:26.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:25.914+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:26.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:25.914+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:26.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:25.914+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:26.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:25.915+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:26.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:25.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:26.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:25.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:26.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:25.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:26.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:25.980+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:26.891 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:27:26.521+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 16536 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:27:26.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:26.612+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:26.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:26.612+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:26.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:26.612+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:26.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:26.612+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:27.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:26.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:27.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:26.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:27.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:26.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:27.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:26.887+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:27.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:26.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:27.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:26.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:27.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:26.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:27.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:26.994+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:28.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:27.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:28.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:27.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:28.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:27.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:28.025 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:27.636+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:28.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:27.856+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:28.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:27.856+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:28.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:27.856+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:28.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:27.856+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:28.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:28.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:28.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:28.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:28.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:28.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:28.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:28.036+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:29.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:28.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:29.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:28.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:29.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:28.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:29.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:28.659+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:29.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:28.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:29.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:28.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:29.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:28.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:29.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:28.822+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:29.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:29.080+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:29.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:29.080+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:29.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:29.080+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:29.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:29.081+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:30.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:29.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:30.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:29.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:30.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:29.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:30.025 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:29.687+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:30.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:29.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:30.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:29.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:30.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:29.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:30.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:29.863+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:30.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:30.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:30.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:30.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:30.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:30.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:30.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:30.037+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:30.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:30.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:30.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:30.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:30.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:30.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:30.912 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:30.651+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:31.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:31.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:31.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:31.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:31.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:31.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:31.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:31.004+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:31.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:30.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:31.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:30.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:31.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:30.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:31.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:30.906+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:31.913 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:27:31.523+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 16626 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:27:31.913 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:31.655+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:31.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:31.655+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:31.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:31.655+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:31.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:31.656+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:32.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:31.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:32.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:31.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:32.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:31.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:32.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:31.963+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:32.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:31.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:32.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:31.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:32.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:31.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:32.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:31.908+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:32.912 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:32.641+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:32.912 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:32.641+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:32.912 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:32.641+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:32.912 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:32.642+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:33.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:32.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:33.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:32.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:33.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:32.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:33.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:32.907+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:33.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:32.945+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:33.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:32.945+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:33.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:32.945+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:33.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:32.945+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:33.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:33.676+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:33.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:33.676+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:33.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:33.676+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:33.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:33.677+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:34.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:33.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:34.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:33.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:34.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:33.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:34.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:33.921+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:34.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:33.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:34.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:33.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:34.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:33.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:34.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:33.894+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:35.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:34.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:35.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:34.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:35.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:34.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:35.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:34.727+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:35.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:34.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:35.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:34.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:35.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:34.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:35.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:34.942+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:35.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:34.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:35.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:34.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:35.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:34.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:35.026 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:34.958+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:36.055 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:35.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:36.056 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:35.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:36.056 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:35.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:36.056 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:35.902+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:36.056 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:35.776+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:36.057 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:35.776+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:36.057 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:35.776+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:36.057 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:35.777+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:36.057 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:35.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:36.057 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:35.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:36.057 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:35.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:36.057 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:35.964+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:36.815 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:27:36.525+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 16715 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:27:37.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:36.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:37.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:36.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:37.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:36.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:37.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:36.869+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:37.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:36.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:37.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:36.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:37.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:36.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:37.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:36.810+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:37.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:36.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:37.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:36.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:37.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:36.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:37.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:36.982+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:38.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:37.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:38.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:37.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:38.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:37.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:38.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:37.910+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:38.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:37.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:38.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:37.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:38.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:37.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:38.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:37.794+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:38.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:38.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:38.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:38.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:38.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:38.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:38.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:38.018+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:39.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:38.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:39.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:38.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:39.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:38.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:39.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:38.923+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:39.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:38.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:39.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:38.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:39.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:38.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:39.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:38.809+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:39.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:39.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:39.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:39.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:39.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:39.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:39.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:39.033+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:40.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:39.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:40.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:39.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:40.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:39.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:40.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:39.897+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:40.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:40.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:40.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:40.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:40.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:40.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:40.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:40.080+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:40.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:39.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:40.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:39.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:40.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:39.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:40.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:39.837+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:41.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:40.891+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:41.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:40.891+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:41.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:40.891+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:41.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:40.891+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:41.071 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:40.805+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:41.071 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:40.805+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:41.071 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:40.805+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:41.071 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:40.806+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:41.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:41.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:41.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:41.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:41.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:41.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:41.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:41.065+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:41.807 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:27:41.527+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 16804 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:27:42.199 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:41.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:42.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:41.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:42.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:41.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:42.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:41.861+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:42.200 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:41.801+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:42.200 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:41.801+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:42.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:41.801+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:42.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:41.803+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:42.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:42.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:42.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:42.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:42.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:42.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:42.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:42.044+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:43.072 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:42.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:43.072 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:42.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:43.072 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:42.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:43.072 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:42.833+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:43.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:42.807+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:43.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:42.807+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:43.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:42.807+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:43.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:42.808+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:43.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:43.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:43.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:43.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:43.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:43.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:43.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:43.067+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:43.836 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:43.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:43.837 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:43.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:43.837 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:43.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:43.837 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:43.784+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:44.097 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:43.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:44.097 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:43.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:44.097 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:43.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:44.098 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:43.832+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:44.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:44.092+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:44.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:44.092+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:44.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:44.092+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:44.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:44.092+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:45.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:44.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:45.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:44.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:45.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:44.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:45.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:44.760+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:45.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:44.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:45.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:44.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:45.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:44.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:45.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:44.834+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:45.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:45.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:45.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:45.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:45.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:45.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:45.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:45.080+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:46.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:45.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:46.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:45.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:46.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:45.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:46.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:45.768+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:46.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:45.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:46.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:45.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:46.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:45.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:46.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:45.852+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:46.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:46.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:46.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:46.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:46.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:46.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:46.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:46.097+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:46.796 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:27:46.529+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 16894 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:27:47.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:46.790+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:47.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:46.790+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:47.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:46.790+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:47.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:46.791+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:47.117 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:46.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:47.117 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:46.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:47.117 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:46.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:47.117 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:46.863+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:47.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:47.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:47.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:47.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:47.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:47.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:47.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:47.111+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:48.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:47.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:48.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:47.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:48.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:47.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:48.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:47.844+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:48.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:47.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:48.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:47.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:48.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:47.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:48.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:47.833+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:48.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:48.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:48.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:48.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:48.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:48.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:48.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:48.148+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:49.115 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:48.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:49.116 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:48.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:49.116 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:48.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:49.116 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:48.804+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:49.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:48.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:49.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:48.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:49.117 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:48.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:49.117 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:48.838+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:49.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:49.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:49.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:49.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:49.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:49.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:49.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:49.111+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:50.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:49.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:50.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:49.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:50.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:49.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:50.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:49.787+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:50.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:49.880+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:50.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:49.880+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:50.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:49.880+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:50.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:49.881+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:50.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:50.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:50.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:50.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:50.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:50.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:50.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:50.156+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:51.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:50.929+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:51.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:50.929+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:51.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:50.929+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:51.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:50.929+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:51.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:50.745+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:51.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:50.745+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:51.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:50.745+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:51.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:50.746+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:51.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:51.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:51.416 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:51.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:51.416 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:51.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:51.416 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:51.153+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:51.772 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:27:51.531+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 16982 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:27:52.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:51.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:52.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:51.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:52.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:51.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:52.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:51.951+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:52.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:51.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:52.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:51.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:52.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:51.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:52.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:51.768+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:52.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:52.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:52.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:52.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:52.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:52.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:52.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:52.133+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:53.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:53.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:53.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:53.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:53.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:53.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:53.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:53.001+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:53.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:52.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:53.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:52.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:53.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:52.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:53.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:52.753+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:53.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:53.154+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:53.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:53.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:53.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:53.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:53.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:53.155+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:54.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:53.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:54.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:53.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:54.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:53.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:54.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:53.779+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:54.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:54.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:54.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:54.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:54.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:54.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:54.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:54.007+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:54.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:54.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:54.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:54.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:54.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:54.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:54.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:54.194+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:55.055 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:55.004+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:55.055 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:55.004+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:55.055 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:55.004+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:55.055 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:55.005+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:55.056 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:54.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:55.056 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:54.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:55.056 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:54.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:55.056 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:54.787+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:55.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:55.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:55.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:55.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:55.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:55.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:55.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:55.153+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:56.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:55.993+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:56.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:55.993+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:56.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:55.993+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:56.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:55.993+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:56.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:55.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:56.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:55.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:56.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:55.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:56.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:55.799+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:56.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:56.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:56.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:56.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:56.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:56.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:56.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:56.135+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:56.800 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:27:56.533+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 17072 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:27:57.101 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:56.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:57.101 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:56.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:57.101 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:56.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:57.101 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:56.796+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:57.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:57.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:57.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:57.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:57.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:57.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:57.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:57.036+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:57.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:57.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:57.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:57.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:57.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:57.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:57.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:57.096+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:58.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:58.037+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:58.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:58.037+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:58.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:58.037+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:58.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:58.038+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:58.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:57.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:58.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:57.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:58.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:57.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:58.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:57.822+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:58.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:58.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:58.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:58.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:58.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:58.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:58.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:58.066+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:59.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:58.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:59.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:58.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:59.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:58.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:27:59.114 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:58.791+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:27:59.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:58.994+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:59.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:58.994+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:59.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:58.994+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:27:59.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:27:58.995+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:27:59.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:59.108+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:59.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:59.108+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:59.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:59.108+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:27:59.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:27:59.108+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:00.106 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:59.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:00.106 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:59.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:00.107 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:59.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:00.107 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:27:59.840+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:00.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:00.008+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:00.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:00.008+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:00.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:00.008+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:00.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:00.009+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:00.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:00.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:00.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:00.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:00.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:00.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:00.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:00.132+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:01.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:00.857+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:01.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:00.857+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:01.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:00.857+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:01.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:00.857+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:01.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:01.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:01.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:01.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:01.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:01.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:01.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:01.130+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:01.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:01.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:01.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:01.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:01.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:01.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:01.526 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:01.051+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:01.859 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:28:01.535+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 17161 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:28:02.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:01.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:02.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:01.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:02.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:01.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:02.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:01.855+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:02.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:02.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:02.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:02.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:02.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:02.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:02.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:02.020+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:02.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:02.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:02.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:02.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:02.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:02.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:02.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:02.177+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:03.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:02.981+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:03.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:02.981+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:03.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:02.981+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:03.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:02.981+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:03.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:03.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:03.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:03.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:03.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:03.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:03.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:03.133+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:03.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:02.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:03.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:02.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:03.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:02.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:03.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:02.884+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:04.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:03.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:04.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:03.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:04.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:03.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:04.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:03.844+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:04.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:04.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:04.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:04.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:04.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:04.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:04.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:04.029+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:04.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:04.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:04.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:04.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:04.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:04.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:04.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:04.114+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:05.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:05.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:05.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:05.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:05.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:05.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:05.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:05.077+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:05.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:05.076+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:05.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:05.076+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:05.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:05.076+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:05.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:05.077+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:05.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:04.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:05.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:04.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:05.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:04.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:05.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:04.874+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:06.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:06.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:06.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:06.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:06.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:06.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:06.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:06.089+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:06.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:06.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:06.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:06.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:06.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:06.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:06.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:06.046+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:06.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:05.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:06.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:05.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:06.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:05.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:06.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:05.872+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:06.920 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:28:06.537+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 17252 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:28:07.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:07.083+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:07.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:07.083+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:07.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:07.083+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:07.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:07.084+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:07.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:06.911+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:07.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:06.911+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:07.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:06.911+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:07.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:06.912+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:07.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:07.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:07.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:07.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:07.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:07.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:07.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:07.026+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:08.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:08.069+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:08.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:08.069+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:08.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:08.069+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:08.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:08.070+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:08.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:07.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:08.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:07.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:08.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:07.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:08.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:07.886+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:08.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:08.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:08.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:08.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:08.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:08.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:08.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:08.019+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:09.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:09.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:09.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:09.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:09.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:09.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:09.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:09.030+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:09.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:08.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:09.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:08.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:09.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:08.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:09.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:08.896+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:09.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:09.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:09.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:09.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:09.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:09.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:09.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:09.024+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:10.203 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:09.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:10.203 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:09.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:10.203 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:09.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:10.203 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:09.996+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:10.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:09.934+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:10.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:09.934+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:10.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:09.934+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:10.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:09.935+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:10.204 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:10.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:10.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:10.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:10.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:10.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:10.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:10.002+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:11.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:11.046+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:11.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:11.046+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:11.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:11.046+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:11.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:11.046+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:11.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:10.895+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:11.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:10.895+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:11.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:10.895+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:11.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:10.896+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:11.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:11.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:11.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:11.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:11.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:11.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:11.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:11.017+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:11.902 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:28:11.539+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 17341 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:28:12.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:12.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:12.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:12.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:12.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:12.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:12.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:12.091+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:12.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:11.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:12.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:11.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:12.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:11.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:12.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:11.898+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:12.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:12.000+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:12.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:12.000+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:12.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:12.000+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:12.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:12.001+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:13.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:13.128+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:13.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:13.128+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:13.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:13.128+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:13.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:13.130+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:13.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:12.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:13.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:12.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:13.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:12.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:13.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:12.913+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:13.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:13.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:13.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:13.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:13.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:13.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:13.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:13.051+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:14.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:14.142+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:14.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:14.142+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:14.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:14.142+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:14.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:14.142+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:14.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:13.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:14.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:13.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:14.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:13.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:14.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:13.921+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:14.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:14.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:14.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:14.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:14.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:14.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:14.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:14.084+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:15.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:15.061+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:15.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:15.061+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:15.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:15.061+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:15.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:15.062+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:15.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:14.917+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:15.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:14.917+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:15.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:14.917+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:15.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:14.917+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:15.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:15.176+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:15.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:15.176+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:15.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:15.176+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:15.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:15.177+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:16.230 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:15.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:16.231 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:15.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:16.231 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:15.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:16.231 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:15.907+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:16.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:16.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:16.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:16.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:16.232 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:16.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:16.232 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:16.018+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:16.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:16.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:16.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:16.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:16.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:16.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:16.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:16.226+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:16.906 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:28:16.540+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 17431 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:28:17.236 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:16.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:17.237 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:16.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:17.237 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:16.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:17.237 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:16.901+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:17.237 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:17.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:17.237 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:17.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:17.238 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:17.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:17.238 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:17.055+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:17.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:17.231+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:17.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:17.231+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:17.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:17.231+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:17.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:17.232+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:18.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:17.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:18.222 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:17.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:18.222 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:17.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:18.222 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:17.894+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:18.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:18.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:18.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:18.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:18.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:18.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:18.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:18.026+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:18.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:18.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:18.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:18.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:18.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:18.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:18.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:18.217+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:19.209 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:19.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:19.209 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:19.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:19.209 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:19.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:19.209 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:19.037+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:19.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:18.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:19.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:18.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:19.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:18.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:19.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:18.853+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:19.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:19.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:19.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:19.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:19.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:19.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:19.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:19.204+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:20.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:20.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:20.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:20.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:20.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:20.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:20.089 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:19.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:20.089 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:19.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:20.089 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:19.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:20.089 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:19.834+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:20.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:20.083+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:20.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:20.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:20.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:20.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:20.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:20.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:20.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:20.162+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:21.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:20.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:21.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:20.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:21.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:20.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:21.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:20.872+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:21.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:21.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:21.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:21.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:21.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:21.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:21.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:21.115+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:21.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:21.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:21.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:21.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:21.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:21.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:21.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:21.170+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:21.899 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:28:21.542+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 17521 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:28:22.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:22.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:22.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:22.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:22.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:22.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:22.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:22.072+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:22.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:21.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:22.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:21.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:22.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:21.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:22.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:21.894+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:22.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:22.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:22.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:22.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:22.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:22.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:22.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:22.126+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:23.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:22.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:23.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:22.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:23.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:22.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:23.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:22.885+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:23.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:23.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:23.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:23.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:23.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:23.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:23.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:23.065+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:23.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:23.135+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:23.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:23.135+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:23.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:23.135+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:23.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:23.136+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:24.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:24.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:24.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:24.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:24.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:24.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:24.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:24.082+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:24.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:24.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:24.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:24.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:24.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:24.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:24.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:24.159+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:24.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:23.927+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:24.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:23.927+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:24.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:23.927+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:24.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:23.927+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:25.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:25.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:25.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:25.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:25.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:25.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:25.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:25.057+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:25.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:24.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:25.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:24.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:25.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:24.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:25.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:24.954+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:25.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:25.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:25.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:25.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:25.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:25.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:25.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:25.134+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:26.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:26.007+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:26.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:26.007+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:26.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:26.007+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:26.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:26.007+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:26.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:26.115+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:26.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:26.115+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:26.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:26.115+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:26.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:26.116+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:26.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:25.971+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:26.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:25.971+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:26.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:25.971+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:26.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:25.972+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:27.000 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:28:26.544+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 17612 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:28:27.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:27.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:27.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:27.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:27.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:27.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:27.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:27.029+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:27.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:27.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:27.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:27.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:27.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:27.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:27.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:27.074+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:27.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:26.995+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:27.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:26.995+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:27.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:26.995+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:27.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:26.996+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:28.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:28.098+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:28.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:28.098+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:28.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:28.098+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:28.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:28.099+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:28.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:28.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:28.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:28.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:28.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:28.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:28.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:28.060+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:28.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:28.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:28.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:28.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:28.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:28.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:28.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:28.013+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:29.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:29.128+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:29.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:29.128+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:29.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:29.128+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:29.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:29.129+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:29.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:29.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:29.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:29.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:29.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:29.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:29.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:29.029+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:29.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:29.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:29.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:29.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:29.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:29.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:29.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:29.078+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:30.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:30.103+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:30.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:30.103+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:30.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:30.103+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:30.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:30.104+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:30.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:30.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:30.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:30.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:30.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:30.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:30.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:30.011+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:30.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:30.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:30.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:30.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:30.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:30.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:30.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:30.063+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:31.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:31.100+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:31.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:31.100+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:31.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:31.100+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:31.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:31.101+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:31.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:31.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:31.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:31.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:31.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:31.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:31.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:31.011+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:31.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:31.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:31.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:31.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:31.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:31.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:31.276 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:31.075+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:31.675 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:28:31.547+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 17699 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:28:32.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:32.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:32.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:32.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:32.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:32.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:32.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:32.032+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:32.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:32.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:32.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:32.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:32.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:32.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:32.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:32.109+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:32.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:32.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:32.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:32.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:32.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:32.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:32.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:32.109+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:33.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:33.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:33.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:33.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:33.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:33.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:33.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:33.074+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:33.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:33.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:33.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:33.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:33.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:33.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:33.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:33.019+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:33.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:33.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:33.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:33.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:33.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:33.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:33.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:33.159+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:34.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:34.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:34.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:34.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:34.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:34.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:34.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:34.120+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:34.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:34.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:34.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:34.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:34.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:34.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:34.425 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:34.114+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:34.425 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:34.055+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:34.425 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:34.055+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:34.425 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:34.055+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:34.425 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:34.056+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:35.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:35.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:35.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:35.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:35.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:35.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:35.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:35.078+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:35.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:35.140+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:35.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:35.140+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:35.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:35.140+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:35.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:35.140+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:35.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:35.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:35.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:35.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:35.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:35.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:35.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:35.022+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:36.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:36.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:36.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:36.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:36.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:36.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:36.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:36.121+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:36.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:36.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:36.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:36.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:36.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:36.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:36.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:36.183+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:36.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:36.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:36.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:36.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:36.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:36.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:36.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:36.010+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:36.679 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:28:36.549+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 17789 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:28:37.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:37.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:37.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:37.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:37.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:37.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:37.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:37.118+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:37.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:37.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:37.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:37.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:37.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:37.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:37.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:37.002+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:37.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:37.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:37.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:37.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:37.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:37.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:37.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:37.182+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:38.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:38.085+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:38.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:38.085+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:38.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:38.085+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:38.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:38.086+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:38.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:38.017+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:38.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:38.017+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:38.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:38.017+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:38.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:38.017+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:38.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:38.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:38.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:38.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:38.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:38.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:38.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:38.143+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:39.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:39.086+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:39.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:39.086+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:39.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:39.086+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:39.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:39.087+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:39.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:39.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:39.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:39.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:39.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:39.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:39.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:39.065+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:39.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:39.108+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:39.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:39.108+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:39.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:39.108+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:39.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:39.108+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:40.487 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:40.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:40.487 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:40.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:40.487 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:40.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:40.487 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:40.078+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:40.488 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:40.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:40.488 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:40.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:40.488 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:40.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:40.488 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:40.078+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:40.488 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:40.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:40.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:40.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:40.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:40.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:40.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:40.105+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:41.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:41.118+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:41.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:41.118+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:41.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:41.118+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:41.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:41.119+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:41.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:41.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:41.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:41.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:41.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:41.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:41.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:41.076+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:41.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:41.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:41.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:41.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:41.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:41.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:41.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:41.062+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:42.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:28:41.551+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 17878 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:28:42.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:42.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:42.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:42.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:42.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:42.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:42.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:42.114+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:42.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:42.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:42.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:42.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:42.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:42.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:42.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:42.109+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:42.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:42.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:42.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:42.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:42.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:42.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:42.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:42.088+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:43.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:43.095+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:43.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:43.095+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:43.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:43.095+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:43.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:43.096+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:43.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:43.099+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:43.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:43.099+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:43.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:43.099+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:43.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:43.099+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:43.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:43.120+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:43.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:43.120+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:43.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:43.120+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:43.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:43.120+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:44.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:44.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:44.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:44.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:44.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:44.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:44.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:44.096+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:44.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:44.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:44.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:44.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:44.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:44.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:44.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:44.085+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:44.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:44.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:44.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:44.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:44.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:44.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:44.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:44.055+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:45.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:45.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:45.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:45.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:45.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:45.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:45.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:45.143+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:45.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:45.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:45.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:45.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:45.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:45.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:45.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:45.037+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:45.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:45.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:45.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:45.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:45.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:45.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:45.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:45.060+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:46.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:46.023+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:46.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:46.023+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:46.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:46.023+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:46.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:46.024+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:46.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:46.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:46.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:46.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:46.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:46.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:46.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:46.011+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:46.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:46.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:46.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:46.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:46.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:46.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:46.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:46.191+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:46.983 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:28:46.552+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 17968 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:28:47.243 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:46.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:47.243 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:46.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:47.243 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:46.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:47.243 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:46.979+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:47.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:47.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:47.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:47.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:47.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:47.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:47.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:47.017+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:47.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:47.238+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:47.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:47.238+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:47.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:47.238+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:47.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:47.238+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:48.047 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:48.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:48.047 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:48.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:48.047 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:48.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:48.047 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:48.010+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:48.048 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:47.993+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:48.048 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:47.993+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:48.048 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:47.993+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:48.048 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:47.994+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:48.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:48.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:48.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:48.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:48.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:48.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:48.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:48.231+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:49.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:49.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:49.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:49.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:49.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:49.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:49.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:49.017+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:49.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:49.220+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:49.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:49.220+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:49.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:49.220+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:49.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:49.220+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:49.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:49.017+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:49.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:49.017+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:49.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:49.017+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:49.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:49.017+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:50.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:49.988+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:50.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:49.988+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:50.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:49.988+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:50.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:49.989+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:50.245 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:50.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:50.245 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:50.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:50.245 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:50.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:50.245 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:50.004+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:50.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:50.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:50.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:50.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:50.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:50.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:50.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:50.239+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:51.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:50.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:51.276 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:50.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:51.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:50.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:51.277 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:50.964+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:51.277 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:51.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:51.277 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:51.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:51.277 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:51.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:51.277 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:51.027+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:51.560 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:51.271+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:51.560 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:51.271+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:51.560 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:51.271+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:51.560 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:51.271+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:51.944 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:28:51.554+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 18056 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:28:52.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:52.069+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:52.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:52.069+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:52.069+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:52.070+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:51.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:52.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:51.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:52.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:51.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:52.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:51.940+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:52.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:52.264+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:52.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:52.264+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:52.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:52.264+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:52.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:52.264+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:53.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:52.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:53.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:52.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:53.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:52.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:53.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:52.945+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:53.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:53.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:53.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:53.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:53.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:53.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:53.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:53.060+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:53.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:53.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:53.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:53.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:53.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:53.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:53.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:53.284+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:54.251 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:53.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:54.251 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:53.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:54.251 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:53.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:54.251 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:53.907+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:54.251 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:54.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:54.252 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:54.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:54.252 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:54.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:54.252 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:54.032+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:54.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:54.246+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:54.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:54.246+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:54.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:54.246+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:54.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:54.246+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:55.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:55.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:55.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:55.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:55.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:55.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:55.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:55.069+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:55.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:54.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:55.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:54.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:55.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:54.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:55.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:54.939+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:55.626 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:55.278+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:55.626 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:55.278+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:55.626 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:55.278+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:55.626 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:55.279+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:56.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:55.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:56.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:55.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:56.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:55.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:56.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:55.942+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:56.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:56.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:56.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:56.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:56.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:56.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:56.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:56.079+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:56.563 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:56.319+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:56.563 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:56.319+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:56.563 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:56.319+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:56.564 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:56.320+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:56.992 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:28:56.556+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 18146 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:28:57.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:56.988+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:57.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:56.988+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:57.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:56.988+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:57.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:56.989+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:57.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:57.127+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:57.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:57.127+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:57.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:57.127+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:57.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:57.128+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:57.773 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:57.295+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:57.773 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:57.295+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:57.773 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:57.295+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:57.773 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:57.296+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:58.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:58.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:58.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:58.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:58.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:58.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:58.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:58.011+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:58.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:58.097+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:58.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:58.097+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:58.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:58.097+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:58.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:58.098+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:58.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:58.263+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:58.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:58.263+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:58.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:58.263+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:58.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:58.264+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:59.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:59.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:59.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:59.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:59.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:59.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:28:59.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:28:59.050+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:28:59.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:59.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:59.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:59.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:59.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:59.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:28:59.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:28:59.077+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:28:59.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:59.285+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:59.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:59.285+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:59.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:59.285+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:28:59.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:28:59.285+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:00.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:00.059+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:00.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:00.059+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:00.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:00.059+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:00.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:00.060+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:00.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:00.242+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:00.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:00.242+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:00.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:00.242+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:00.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:00.242+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:00.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:00.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:00.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:00.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:00.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:00.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:00.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:00.120+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:01.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:01.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:01.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:01.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:01.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:01.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:01.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:01.151+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:01.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:01.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:01.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:01.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:01.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:01.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:01.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:01.081+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:01.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:01.288+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:01.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:01.288+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:01.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:01.288+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:01.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:01.289+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:02.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:29:01.558+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 18235 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:29:02.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:02.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:02.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:02.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:02.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:02.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:02.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:02.113+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:02.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:02.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:02.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:02.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:02.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:02.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:02.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:02.178+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:02.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:02.243+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:02.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:02.243+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:02.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:02.243+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:02.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:02.243+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:03.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:03.200+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:03.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:03.200+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:03.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:03.200+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:03.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:03.201+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:03.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:03.218+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:03.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:03.218+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:03.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:03.218+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:03.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:03.218+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:03.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:03.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:03.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:03.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:03.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:03.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:03.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:03.156+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:04.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:04.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:04.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:04.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:04.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:04.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:04.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:04.217+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:04.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:04.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:04.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:04.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:04.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:04.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:04.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:04.159+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:04.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:04.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:04.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:04.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:04.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:04.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:04.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:04.234+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:05.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:05.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:05.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:05.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:05.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:05.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:05.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:05.113+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:05.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:05.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:05.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:05.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:05.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:05.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:05.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:05.171+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:05.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:05.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:05.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:05.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:05.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:05.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:05.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:05.282+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:06.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:06.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:06.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:06.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:06.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:06.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:06.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:06.320+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:06.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:06.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:06.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:06.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:06.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:06.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:06.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:06.161+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:06.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:06.137+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:06.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:06.137+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:06.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:06.137+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:06.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:06.139+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:07.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:29:06.560+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 18325 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:29:07.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:07.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:07.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:07.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:07.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:07.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:07.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:07.315+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:07.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:07.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:07.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:07.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:07.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:07.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:07.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:07.199+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:07.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:07.188+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:07.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:07.188+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:07.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:07.188+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:07.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:07.188+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:08.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:08.243+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:08.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:08.243+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:08.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:08.243+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:08.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:08.243+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:08.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:08.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:08.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:08.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:08.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:08.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:08.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:08.221+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:08.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:08.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:08.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:08.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:08.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:08.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:08.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:08.335+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:09.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:09.365+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:09.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:09.365+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:09.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:09.365+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:09.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:09.366+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:09.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:09.288+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:09.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:09.288+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:09.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:09.288+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:09.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:09.288+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:09.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:09.207+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:09.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:09.207+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:09.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:09.207+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:09.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:09.208+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:10.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:10.416+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:10.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:10.416+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:10.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:10.416+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:10.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:10.417+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:10.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:10.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:10.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:10.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:10.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:10.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:10.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:10.205+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:10.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:10.267+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:10.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:10.267+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:10.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:10.267+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:10.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:10.268+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:11.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:11.368+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:11.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:11.368+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:11.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:11.368+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:11.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:11.368+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:11.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:11.245+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:11.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:11.245+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:11.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:11.245+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:11.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:11.245+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:11.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:11.233+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:11.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:11.233+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:11.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:11.233+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:11.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:11.234+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:12.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:29:11.562+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 18413 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:29:12.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:12.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:12.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:12.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:12.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:12.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:12.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:12.204+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:12.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:12.352+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:12.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:12.352+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:12.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:12.352+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:12.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:12.352+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:12.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:12.212+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:12.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:12.212+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:12.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:12.212+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:12.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:12.213+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:13.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:13.188+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:13.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:13.188+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:13.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:13.188+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:13.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:13.189+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:13.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:13.351+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:13.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:13.351+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:13.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:13.351+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:13.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:13.351+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:13.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:13.225+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:13.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:13.225+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:13.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:13.225+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:13.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:13.225+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:14.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:14.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:14.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:14.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:14.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:14.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:14.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:14.178+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:14.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:14.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:14.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:14.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:14.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:14.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:14.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:14.340+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:14.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:14.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:14.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:14.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:14.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:14.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:14.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:14.231+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:15.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:15.339+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:15.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:15.339+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:15.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:15.339+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:15.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:15.339+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:15.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:15.262+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:15.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:15.262+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:15.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:15.262+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:15.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:15.262+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:15.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:15.207+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:15.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:15.207+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:15.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:15.207+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:15.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:15.208+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:16.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:16.242+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:16.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:16.242+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:16.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:16.242+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:16.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:16.242+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:16.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:16.212+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:16.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:16.212+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:16.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:16.212+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:16.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:16.213+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:16.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:16.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:16.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:16.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:16.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:16.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:16.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:16.323+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:17.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:29:16.565+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 18503 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:29:17.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:17.222+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:17.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:17.222+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:17.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:17.222+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:17.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:17.222+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:17.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:17.223+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:17.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:17.223+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:17.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:17.223+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:17.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:17.223+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:17.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:17.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:17.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:17.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:17.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:17.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:17.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:17.347+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:18.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:18.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:18.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:18.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:18.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:18.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:18.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:18.274+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:18.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:18.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:18.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:18.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:18.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:18.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:18.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:18.367+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:18.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:18.188+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:18.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:18.188+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:18.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:18.188+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:18.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:18.189+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:19.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:19.378+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:19.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:19.378+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:19.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:19.378+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:19.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:19.378+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:19.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:19.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:19.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:19.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:19.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:19.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:19.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:19.140+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:19.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:19.285+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:19.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:19.285+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:19.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:19.285+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:19.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:19.285+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:20.502 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:20.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:20.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:20.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:20.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:20.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:20.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:20.146+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:20.503 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:20.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:20.503 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:20.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:20.503 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:20.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:20.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:20.263+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:20.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:20.394+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:20.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:20.394+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:20.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:20.395+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:20.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:20.395+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:21.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:21.387+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:21.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:21.387+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:21.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:21.387+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:21.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:21.388+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:21.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:21.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:21.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:21.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:21.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:21.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:21.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:21.265+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:21.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:21.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:21.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:21.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:21.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:21.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:21.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:21.176+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:22.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:29:21.567+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 18593 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:29:22.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:22.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:22.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:22.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:22.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:22.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:22.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:22.393+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:22.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:22.189+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:22.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:22.189+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:22.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:22.189+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:22.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:22.190+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:22.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:22.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:22.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:22.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:22.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:22.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:22.525 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:22.222+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:23.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:23.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:23.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:23.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:23.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:23.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:23.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:23.180+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:23.440 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:23.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:23.440 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:23.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:23.440 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:23.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:23.440 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:23.209+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:23.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:23.434+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:23.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:23.434+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:23.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:23.434+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:23.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:23.435+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:24.406 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:24.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:24.406 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:24.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:24.406 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:24.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:24.406 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:24.228+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:24.406 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:24.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:24.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:24.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:24.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:24.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:24.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:24.150+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:24.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:24.401+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:24.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:24.401+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:24.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:24.401+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:24.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:24.402+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:25.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:25.164+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:25.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:25.164+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:25.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:25.164+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:25.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:25.164+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:25.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:25.277+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:25.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:25.277+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:25.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:25.277+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:25.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:25.277+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:25.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:25.397+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:25.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:25.397+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:25.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:25.397+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:25.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:25.398+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:26.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:26.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:26.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:26.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:26.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:26.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:26.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:26.118+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:26.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:26.263+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:26.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:26.263+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:26.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:26.263+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:26.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:26.264+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:26.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:26.428+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:26.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:26.428+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:26.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:26.428+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:26.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:26.429+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:27.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:29:26.569+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 18684 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:29:27.405 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:27.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:27.406 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:27.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:27.406 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:27.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:27.406 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:27.130+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:27.406 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:27.218+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:27.406 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:27.218+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:27.407 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:27.218+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:27.407 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:27.219+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:27.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:27.401+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:27.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:27.401+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:27.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:27.401+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:27.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:27.402+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:28.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:28.215+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:28.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:28.215+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:28.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:28.215+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:28.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:28.215+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:28.425 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:28.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:28.425 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:28.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:28.425 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:28.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:28.425 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:28.091+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:28.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:28.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:28.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:28.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:28.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:28.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:28.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:28.420+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:29.457 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:29.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:29.457 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:29.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:29.457 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:29.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:29.458 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:29.262+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:29.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:29.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:29.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:29.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:29.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:29.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:29.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:29.067+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:29.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:29.452+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:29.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:29.452+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:29.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:29.452+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:29.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:29.453+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:30.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:30.245+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:30.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:30.245+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:30.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:30.245+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:30.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:30.246+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:30.509 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:30.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:30.509 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:30.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:30.509 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:30.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:30.509 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:30.030+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:30.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:30.503+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:30.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:30.503+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:30.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:30.503+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:30.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:30.504+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:31.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:31.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:31.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:31.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:31.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:31.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:31.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:31.078+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:31.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:31.235+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:31.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:31.235+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:31.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:31.235+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:31.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:31.236+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:32.023 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:29:31.571+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 18773 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:29:32.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:31.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:32.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:31.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:32.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:31.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:32.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:31.539+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:32.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:32.272+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:32.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:32.272+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:32.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:32.272+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:32.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:32.273+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:32.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:32.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:32.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:32.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:32.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:32.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:32.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:32.103+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:33.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:32.572+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:33.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:32.572+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:33.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:32.572+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:33.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:32.573+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:33.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:33.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:33.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:33.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:33.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:33.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:33.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:33.227+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:33.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:33.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:33.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:33.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:33.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:33.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:33.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:33.140+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:34.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:33.535+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:34.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:33.535+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:34.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:33.535+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:34.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:33.536+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:34.495 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:34.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:34.496 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:34.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:34.496 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:34.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:34.496 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:34.253+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:34.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:34.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:34.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:34.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:34.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:34.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:34.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:34.104+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:34.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:34.490+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:34.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:34.490+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:34.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:34.490+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:34.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:34.491+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:35.500 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:35.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:35.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:35.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:35.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:35.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:35.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:35.205+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:35.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:35.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:35.501 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:35.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:35.502 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:35.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:35.502 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:35.102+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:35.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:35.496+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:35.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:35.496+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:35.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:35.496+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:35.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:35.497+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:36.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:36.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:36.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:36.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:36.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:36.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:03.153883+0000 (oldest deadline 2024-02-19T19:19:23.153883+0000) 2024-02-19T19:29:36.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:36.084+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:36.503 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:36.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:36.503 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:36.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:36.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:36.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:36.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:36.211+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:36.773 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:29:36.573+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 18864 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:29:36.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:36.499+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:36.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:36.499+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:36.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:36.499+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:36.774 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:36.500+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:37.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:37.091+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:37.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:37.248+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:37.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:37.248+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:37.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:37.248+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:37.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:37.248+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:38.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:37.525+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:38.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:37.525+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:38.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:37.525+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:38.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:37.526+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:38.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:38.092+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:38.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:38.244+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:38.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:38.244+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:38.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:38.244+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:38.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:38.244+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:39.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:38.518+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:39.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:38.518+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:39.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:38.518+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:39.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:38.519+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:39.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:39.107+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:39.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:39.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:39.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:39.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:39.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:39.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:39.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:39.258+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:40.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:39.549+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:40.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:39.549+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:40.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:39.549+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:40.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:39.550+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:40.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:40.088+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:40.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:40.235+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:40.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:40.235+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:40.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:40.235+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:40.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:40.236+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:41.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:40.565+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:41.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:40.565+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:41.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:40.565+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:41.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:40.566+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:41.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:41.123+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:41.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:41.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:41.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:41.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:41.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:41.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:41.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:41.206+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:42.006 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:29:41.575+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 18954 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:29:42.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:41.562+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:42.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:41.562+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:42.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:41.562+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:42.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:41.563+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:42.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:42.120+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:42.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:42.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:42.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:42.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:42.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:42.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:42.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:42.206+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:42.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:42.513+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:42.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:42.513+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:42.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:42.513+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:42.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:42.514+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:43.508 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:43.077+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:43.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:43.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:43.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:43.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:43.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:43.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:43.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:43.237+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:43.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:43.503+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:43.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:43.503+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:43.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:43.503+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:43.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:43.505+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:44.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:44.041+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:44.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:44.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:44.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:44.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:44.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:44.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:44.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:44.198+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:45.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:44.522+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:45.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:44.522+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:45.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:44.522+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:45.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:44.523+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:45.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:45.034+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:45.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:45.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:45.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:45.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:45.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:45.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:45.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:45.217+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:46.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:45.525+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:46.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:45.525+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:46.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:45.525+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:46.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:45.526+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:46.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:46.041+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:46.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:46.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:46.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:46.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:46.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:46.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:46.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:46.252+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:46.772 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:29:46.577+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 19045 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:29:46.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:46.502+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:46.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:46.502+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:46.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:46.502+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:46.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:46.503+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:47.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:47.050+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:47.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:47.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:47.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:47.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:47.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:47.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:13.883669+0000 (oldest deadline 2024-02-19T19:19:33.883669+0000) 2024-02-19T19:29:47.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:47.206+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:47.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:47.515+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:47.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:47.515+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:47.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:47.515+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:19:10.766945+0000 (oldest deadline 2024-02-19T19:19:30.766945+0000) 2024-02-19T19:29:47.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:47.516+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:48.489 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:48.199+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:48.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:48.082+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:48.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:48.485+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:49.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:49.226+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:49.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:49.060+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:49.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:49.459+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:50.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:50.074+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:50.469 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:50.248+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:50.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:50.464+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:51.471 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:51.114+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:51.471 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:51.239+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:51.772 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:29:51.580+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 19134 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:29:51.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:51.466+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:52.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:52.195+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:52.430 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:52.126+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:52.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:52.425+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:53.432 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:53.200+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:53.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:53.112+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:53.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:53.428+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:54.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:54.093+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:54.471 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:54.178+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:54.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:54.466+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:55.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:55.227+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:55.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:55.135+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:55.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:55.446+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:56.486 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:56.263+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:56.487 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:56.119+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:56.772 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:29:56.582+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 19225 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:29:56.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:56.482+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:57.441 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:57.306+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:57.442 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:57.149+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:57.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:57.437+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:58.469 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:58.267+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:58.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:58.156+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:58.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:58.464+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:29:59.480 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:29:59.278+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:59.481 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:29:59.141+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:29:59.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:29:59.476+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:00.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:00.250+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:00.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:00.179+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:00.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:00.179+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:00.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:00.179+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:00.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:00.179+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:00.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:00.488+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:01.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:01.236+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:01.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:01.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:01.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:01.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:01.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:01.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:01.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:01.158+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:02.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:30:01.584+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 19315 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:30:02.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:01.524+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:02.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:02.192+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:02.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:02.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:02.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:02.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:02.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:02.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:02.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:02.174+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:03.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:02.569+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:03.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:03.238+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:03.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:03.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:03.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:03.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:03.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:03.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:03.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:03.137+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:04.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:03.539+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:04.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:04.199+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:04.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:04.173+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:04.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:04.173+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:04.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:04.173+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:04.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:04.175+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:05.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:04.545+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:05.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:05.157+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:05.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:05.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:05.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:05.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:05.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:05.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:05.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:05.156+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:06.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:05.539+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:06.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:06.165+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:06.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:06.154+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:06.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:06.154+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:06.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:06.154+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:06.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:06.154+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:07.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:30:06.586+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 19406 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:30:07.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:06.527+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:07.514 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:07.124+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:07.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:07.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:07.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:07.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:07.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:07.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:07.515 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:07.122+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:07.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:07.510+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:08.498 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:08.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:08.499 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:08.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:08.499 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:08.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:08.499 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:08.082+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:08.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:08.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:08.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:08.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:08.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:08.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:08.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:08.091+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:08.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:08.494+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:09.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:09.087+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:09.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:09.087+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:09.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:09.087+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:09.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:09.088+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:09.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:09.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:09.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:09.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:09.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:09.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:09.524 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:09.074+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:10.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:09.524+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:10.490 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:10.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:10.490 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:10.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:10.490 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:10.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:10.490 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:10.074+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:10.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:10.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:10.491 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:10.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:10.491 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:10.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:10.491 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:10.077+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:10.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:10.485+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:11.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:11.037+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:11.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:11.037+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:11.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:11.037+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:11.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:11.038+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:11.327 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:11.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:11.327 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:11.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:11.327 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:11.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:11.327 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:11.099+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:11.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:11.456+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:12.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:30:11.588+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 19495 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:30:12.412 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:12.045+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:12.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:12.045+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:12.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:12.045+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:12.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:12.045+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:12.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:12.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:12.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:12.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:12.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:12.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:12.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:12.105+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:12.726 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:12.408+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:13.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:13.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:13.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:13.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:13.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:13.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:13.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:13.060+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:13.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:13.014+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:13.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:13.014+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:13.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:13.014+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:13.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:13.015+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:13.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:13.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:13.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:13.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:13.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:13.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:13.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:13.419+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:14.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:14.004+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:14.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:14.004+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:14.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:14.004+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:14.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:14.005+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:14.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:14.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:14.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:14.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:14.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:14.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:14.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:14.070+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:14.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:14.398+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:14.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:14.398+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:14.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:14.398+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:14.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:14.399+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:15.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:14.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:15.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:14.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:15.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:14.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:15.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:14.960+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:15.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:15.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:15.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:15.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:15.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:15.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:15.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:15.072+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:15.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:15.404+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:15.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:15.404+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:15.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:15.404+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:15.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:15.405+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:16.115 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:15.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:16.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:15.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:16.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:15.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:16.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:15.995+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:16.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:16.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:16.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:16.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:16.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:16.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:16.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:16.112+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:16.717 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:30:16.591+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 19585 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:30:16.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:16.407+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:16.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:16.407+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:16.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:16.407+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:16.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:16.408+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:17.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:16.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:17.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:16.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:17.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:16.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:17.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:16.957+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:17.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:17.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:17.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:17.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:17.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:17.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:17.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:17.156+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:17.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:17.443+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:17.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:17.443+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:17.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:17.443+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:17.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:17.444+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:18.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:18.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:18.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:18.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:18.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:18.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:18.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:18.138+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:18.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:17.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:18.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:17.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:18.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:17.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:18.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:17.937+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:18.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:18.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:18.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:18.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:18.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:18.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:18.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:18.462+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:19.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:18.917+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:19.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:18.917+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:19.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:18.917+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:19.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:18.917+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:19.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:19.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:19.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:19.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:19.464 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:19.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:19.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:19.173+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:19.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:19.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:19.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:19.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:19.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:19.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:19.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:19.460+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:20.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:19.899+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:20.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:19.899+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:20.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:19.899+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:20.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:19.900+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:20.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:20.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:20.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:20.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:20.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:20.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:20.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:20.138+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:20.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:20.437+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:20.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:20.437+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:20.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:20.437+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:20.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:20.438+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:21.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:20.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:21.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:20.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:21.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:20.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:21.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:20.937+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:21.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:21.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:21.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:21.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:21.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:21.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:21.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:21.163+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:21.720 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:30:21.593+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 19674 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:30:21.720 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:21.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:21.721 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:21.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:21.721 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:21.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:21.721 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:21.416+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:22.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:21.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:22.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:21.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:22.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:21.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:22.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:21.889+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:22.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:22.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:22.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:22.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:22.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:22.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:22.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:22.146+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:22.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:22.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:22.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:22.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:22.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:22.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:22.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:22.409+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:23.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:23.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:23.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:23.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:23.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:23.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:23.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:23.122+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:23.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:22.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:23.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:22.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:23.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:22.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:23.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:22.923+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:23.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:23.425+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:23.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:23.425+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:23.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:23.425+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:23.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:23.426+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:24.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:23.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:24.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:23.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:24.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:23.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:24.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:23.939+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:24.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:24.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:24.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:24.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:24.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:24.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:24.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:24.138+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:24.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:24.441+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:24.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:24.441+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:24.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:24.441+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:24.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:24.442+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:25.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:25.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:25.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:25.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:25.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:25.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:25.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:25.113+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:25.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:24.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:25.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:24.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:25.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:24.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:25.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:24.964+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:25.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:25.463+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:25.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:25.463+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:25.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:25.463+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:25.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:25.464+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:26.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:25.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:26.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:25.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:26.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:25.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:26.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:25.929+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:26.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:26.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:26.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:26.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:26.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:26.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:26.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:26.082+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:26.599 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:26.484+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:26.600 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:26.484+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:26.600 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:26.484+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:26.600 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:26.484+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:26.918 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:30:26.595+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 19765 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:30:27.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:26.913+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:27.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:26.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:27.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:26.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:27.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:26.914+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:27.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:27.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:27.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:27.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:27.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:27.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:27.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:27.111+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:27.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:27.494+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:27.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:27.494+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:27.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:27.494+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:27.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:27.495+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:28.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:28.098+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:28.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:28.098+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:28.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:28.098+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:28.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:28.099+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:28.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:27.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:28.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:27.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:28.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:27.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:28.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:27.919+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:28.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:28.516+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:28.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:28.516+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:28.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:28.516+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:28.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:28.517+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:29.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:29.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:29.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:29.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:29.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:29.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:29.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:29.113+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:29.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:28.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:29.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:28.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:29.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:28.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:29.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:28.929+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:29.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:29.475+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:29.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:29.475+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:29.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:29.475+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:29.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:29.476+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:30.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:29.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:30.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:29.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:30.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:29.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:30.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:29.903+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:30.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:30.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:30.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:30.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:30.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:30.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:30.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:30.134+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:30.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:30.441+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:30.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:30.441+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:30.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:30.441+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:30.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:30.442+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:31.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:30.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:31.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:30.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:31.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:30.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:31.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:30.862+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:31.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:31.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:31.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:31.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:31.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:31.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:31.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:31.107+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:31.772 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:30:31.598+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 19854 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:30:31.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:31.471+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:31.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:31.471+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:31.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:31.471+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:31.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:31.472+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:32.113 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:31.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:32.113 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:31.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:32.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:31.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:32.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:31.813+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:32.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:32.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:32.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:32.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:32.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:32.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:32.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:32.110+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:32.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:32.501+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:32.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:32.501+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:32.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:32.501+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:32.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:32.502+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:33.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:33.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:33.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:33.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:33.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:33.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:33.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:33.071+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:33.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:32.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:33.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:32.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:33.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:32.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:33.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:32.837+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:33.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:33.477+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:33.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:33.477+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:33.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:33.477+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:33.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:33.478+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:34.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:33.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:34.115 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:33.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:34.115 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:33.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:34.115 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:33.826+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:34.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:34.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:34.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:34.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:34.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:34.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:34.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:34.111+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:34.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:34.523+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:34.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:34.523+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:34.790 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:34.523+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:34.791 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:34.524+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:35.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:34.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:35.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:34.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:35.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:34.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:35.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:34.787+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:35.488 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:35.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:35.488 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:35.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:35.488 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:35.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:35.488 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:35.081+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:35.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:35.484+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:35.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:35.484+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:35.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:35.484+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:35.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:35.484+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:36.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:35.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:36.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:35.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:36.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:35.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:36.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:35.819+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:36.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:36.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:36.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:36.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:36.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:36.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:36.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:36.127+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:36.772 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:30:36.600+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 19945 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:30:36.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:36.509+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:36.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:36.509+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:36.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:36.509+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:36.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:36.510+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:37.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:36.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:37.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:36.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:37.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:36.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:37.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:36.850+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:37.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:37.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:37.504 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:37.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:37.504 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:37.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:37.504 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:37.153+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:37.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:37.499+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:37.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:37.499+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:37.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:37.499+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:37.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:37.500+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:38.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:37.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:38.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:37.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:38.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:37.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:38.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:37.859+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:38.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:38.186+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:38.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:38.186+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:38.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:38.186+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:38.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:38.187+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:38.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:38.455+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:38.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:38.455+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:38.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:38.455+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:38.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:38.455+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:39.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:38.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:39.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:38.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:39.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:38.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:39.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:38.855+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:39.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:39.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:39.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:39.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:39.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:39.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:39.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:39.415+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:39.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:39.189+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:39.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:39.189+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:39.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:39.189+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:39.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:39.189+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:40.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:39.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:40.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:39.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:40.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:39.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:40.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:39.825+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:40.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:40.418+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:40.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:40.418+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:40.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:40.418+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:40.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:40.419+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:40.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:40.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:40.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:40.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:40.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:40.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:40.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:40.213+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:41.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:40.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:41.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:40.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:41.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:40.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:41.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:40.866+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:41.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:41.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:41.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:41.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:41.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:41.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:41.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:41.171+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:41.730 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:30:41.602+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 20035 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:30:41.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:41.455+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:41.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:41.455+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:41.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:41.455+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:41.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:41.456+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:42.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:41.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:42.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:41.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:42.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:41.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:42.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:41.873+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:42.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:42.189+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:42.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:42.189+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:42.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:42.189+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:42.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:42.189+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:42.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:42.411+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:42.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:42.411+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:42.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:42.411+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:42.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:42.411+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:43.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:42.901+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:43.219 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:42.901+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:43.219 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:42.901+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:43.219 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:42.902+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:43.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:43.372+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:43.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:43.372+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:43.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:43.372+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:43.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:43.372+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:43.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:43.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:43.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:43.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:43.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:43.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:43.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:43.215+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:44.186 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:43.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:44.186 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:43.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:44.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:43.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:44.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:43.865+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:44.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:44.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:44.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:44.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:44.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:44.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:44.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:44.183+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:44.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:44.404+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:44.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:44.404+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:44.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:44.404+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:44.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:44.404+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:45.225 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:44.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:45.225 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:44.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:45.225 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:44.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:45.225 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:44.847+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:45.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:45.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:45.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:45.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:45.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:45.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:45.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:45.415+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:45.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:45.221+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:45.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:45.221+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:45.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:45.221+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:45.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:45.221+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:46.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:45.882+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:46.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:45.882+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:46.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:45.882+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:46.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:45.883+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:46.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:46.203+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:46.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:46.203+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:46.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:46.203+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:46.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:46.203+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:46.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:46.448+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:46.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:46.448+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:46.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:46.448+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:46.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:46.449+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:46.772 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:30:46.604+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 20125 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:30:47.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:46.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:47.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:46.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:47.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:46.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:47.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:46.930+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:47.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:47.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:47.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:47.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:47.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:47.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:47.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:47.156+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:47.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:47.499+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:47.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:47.499+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:47.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:47.499+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:47.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:47.499+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:48.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:47.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:48.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:47.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:48.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:47.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:48.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:47.909+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:48.488 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:48.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:48.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:48.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:48.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:48.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:48.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:48.174+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:48.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:48.483+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:48.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:48.483+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:48.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:48.483+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:48.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:48.484+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:49.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:48.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:49.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:48.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:49.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:48.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:49.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:48.861+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:49.473 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:49.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:49.473 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:49.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:49.473 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:49.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:49.473 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:49.170+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:49.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:49.469+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:49.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:49.469+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:49.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:49.469+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:49.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:49.470+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:50.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:50.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:50.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:50.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:50.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:50.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:50.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:50.136+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:50.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:49.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:50.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:49.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:50.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:49.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:50.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:49.907+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:50.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:50.476+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:50.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:50.476+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:50.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:50.476+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:50.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:50.477+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:51.126 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:50.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:51.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:50.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:51.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:50.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:51.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:50.862+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:51.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:51.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:51.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:51.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:51.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:51.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:51.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:51.123+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:51.772 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:30:51.607+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 20215 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:30:51.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:51.493+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:51.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:51.493+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:51.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:51.493+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:51.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:51.494+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:52.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:51.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:52.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:51.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:52.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:51.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:52.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:51.831+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:52.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:52.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:52.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:52.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:52.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:52.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:52.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:52.099+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:53.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:52.543+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:53.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:52.543+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:53.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:52.543+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:53.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:52.543+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:53.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:52.784+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:53.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:52.784+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:53.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:52.784+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:53.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:52.784+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:53.505 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:53.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:53.505 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:53.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:53.505 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:53.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:53.505 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:53.148+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:53.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:53.501+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:53.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:53.501+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:53.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:53.501+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:53.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:53.502+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:54.150 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:53.817+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:54.150 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:53.817+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:54.150 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:53.817+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:54.150 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:53.818+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:54.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:54.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:54.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:54.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:54.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:54.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:54.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:54.147+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:54.808 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:54.550+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:54.808 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:54.550+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:54.808 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:54.550+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:54.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:54.550+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:55.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:54.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:55.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:54.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:55.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:54.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:55.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:54.805+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:55.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:55.162+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:55.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:55.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:55.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:55.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:55.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:55.163+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:55.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:55.582+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:55.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:55.582+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:55.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:55.582+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:55.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:55.583+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:56.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:55.840+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:56.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:55.840+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:56.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:55.840+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:56.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:55.841+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:56.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:56.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:56.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:56.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:56.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:56.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:56.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:56.137+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:57.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:30:56.609+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 20305 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:30:57.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:56.614+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:57.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:56.614+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:57.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:56.614+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:57.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:56.615+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:57.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:56.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:57.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:56.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:57.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:56.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:57.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:56.804+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:57.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:57.154+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:57.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:57.154+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:57.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:57.154+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:57.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:57.154+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:58.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:57.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:58.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:57.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:58.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:57.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:58.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:57.593+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:58.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:57.779+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:58.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:57.779+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:58.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:57.779+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:58.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:57.780+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:58.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:58.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:58.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:58.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:58.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:58.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:58.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:58.128+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:59.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:58.601+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:59.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:58.601+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:59.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:58.601+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:30:59.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:58.602+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:30:59.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:58.814+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:59.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:58.814+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:59.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:58.814+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:30:59.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:58.815+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:30:59.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:59.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:59.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:59.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:59.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:59.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:30:59.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:30:59.171+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:00.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:59.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:00.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:59.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:00.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:59.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:00.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:30:59.591+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:00.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:59.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:00.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:59.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:00.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:59.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:00.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:30:59.771+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:00.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:00.199+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:00.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:00.199+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:00.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:00.199+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:00.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:00.199+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:01.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:00.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:01.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:00.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:01.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:00.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:01.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:00.795+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:01.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:00.601+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:01.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:00.601+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:01.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:00.601+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:01.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:00.602+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:01.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:01.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:01.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:01.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:01.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:01.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:01.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:01.176+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:01.878 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:31:01.611+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 20396 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:31:01.878 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:01.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:01.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:01.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:01.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:01.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:01.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:01.820+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:01.879 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:01.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:01.879 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:01.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:01.879 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:01.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:01.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:01.592+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:02.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:02.224+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:02.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:02.224+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:02.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:02.224+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:02.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:02.224+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:03.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:02.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:03.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:02.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:03.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:02.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:03.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:02.817+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:03.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:02.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:03.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:02.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:03.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:02.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:03.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:02.641+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:03.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:03.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:03.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:03.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:03.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:03.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:03.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:03.214+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:04.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:03.801+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:04.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:03.801+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:04.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:03.801+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:04.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:03.802+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:04.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:03.625+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:04.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:03.625+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:04.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:03.625+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:04.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:03.625+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:04.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:04.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:04.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:04.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:04.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:04.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:04.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:04.214+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:04.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:04.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:04.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:04.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:04.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:04.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:04.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:04.665+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:04.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:04.832+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:04.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:04.832+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:04.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:04.832+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:04.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:04.834+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:05.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:05.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:05.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:05.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:05.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:05.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:05.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:05.239+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:06.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:05.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:06.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:05.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:06.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:05.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:06.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:05.857+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:06.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:05.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:06.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:05.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:06.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:05.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:06.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:05.713+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:06.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:06.262+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:06.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:06.262+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:06.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:06.262+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:06.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:06.263+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:06.885 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:31:06.613+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 20486 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:31:06.886 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:06.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:06.886 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:06.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:06.886 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:06.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:06.886 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:06.688+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:06.882+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:06.882+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:06.882+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:06.882+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:07.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:07.266+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:07.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:07.266+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:07.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:07.266+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:07.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:07.266+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:08.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:07.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:08.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:07.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:08.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:07.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:08.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:07.889+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:08.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:07.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:08.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:07.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:08.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:07.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:08.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:07.676+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:08.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:08.219+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:08.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:08.219+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:08.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:08.219+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:08.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:08.219+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:09.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:08.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:09.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:08.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:09.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:08.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:09.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:08.710+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:09.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:08.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:09.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:08.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:09.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:08.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:09.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:08.935+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:09.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:09.247+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:09.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:09.247+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:09.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:09.247+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:09.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:09.248+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:10.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:09.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:10.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:09.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:10.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:09.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:10.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:09.919+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:10.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:09.681+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:10.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:09.681+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:10.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:09.681+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:10.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:09.682+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:10.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:10.205+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:10.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:10.205+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:10.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:10.205+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:10.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:10.206+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:11.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:10.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:11.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:10.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:11.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:10.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:11.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:10.671+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:11.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:10.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:11.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:10.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:11.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:10.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:11.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:10.897+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:11.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:11.248+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:11.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:11.248+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:11.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:11.248+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:11.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:11.248+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:11.895 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:31:11.615+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 20573 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:31:11.896 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:11.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:11.897 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:11.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:11.897 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:11.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:11.897 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:11.671+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:12.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:11.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:12.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:11.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:12.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:11.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:12.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:11.919+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:12.691 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:12.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:12.691 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:12.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:12.691 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:12.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:12.691 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:12.285+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:12.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:12.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:12.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:12.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:12.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:12.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:12.948 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:12.688+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:13.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:12.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:13.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:12.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:13.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:12.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:13.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:12.944+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:13.584 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:13.280+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:13.584 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:13.280+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:13.584 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:13.280+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:13.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:13.281+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:13.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:13.716+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:13.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:13.716+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:13.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:13.716+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:13.934 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:13.716+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:14.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:13.929+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:14.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:13.929+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:14.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:13.929+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:14.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:13.930+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:14.733 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:14.280+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:14.733 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:14.280+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:14.734 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:14.280+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:14.734 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:14.280+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:15.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:14.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:15.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:14.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:15.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:14.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:15.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:14.896+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:15.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:14.729+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:15.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:14.729+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:15.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:14.729+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:15.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:14.730+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:15.691 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:15.299+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:15.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:15.299+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:15.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:15.299+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:15.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:15.299+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:16.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:15.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:16.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:15.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:16.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:15.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:16.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:15.918+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:16.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:15.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:16.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:15.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:16.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:15.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:16.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:15.688+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:16.622 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:16.269+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:16.622 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:16.269+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:16.622 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:16.269+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:16.622 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:16.269+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:16.918 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:31:16.618+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 20663 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:31:16.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:16.665+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:16.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:16.665+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:16.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:16.665+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:16.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:16.666+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:17.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:16.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:17.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:16.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:17.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:16.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:17.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:16.916+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:17.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:17.255+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:17.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:17.255+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:17.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:17.255+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:17.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:17.255+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:18.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:17.929+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:18.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:17.929+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:18.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:17.929+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:18.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:17.929+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:18.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:17.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:18.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:17.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:18.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:17.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:18.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:17.692+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:18.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:18.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:18.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:18.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:18.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:18.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:18.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:18.213+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:19.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:18.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:19.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:18.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:19.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:18.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:19.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:18.726+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:19.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:18.955+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:19.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:18.955+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:19.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:18.955+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:19.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:18.955+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:19.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:19.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:19.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:19.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:19.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:19.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:19.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:19.180+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:20.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:19.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:20.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:19.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:20.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:19.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:20.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:19.751+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:20.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:19.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:20.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:19.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:20.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:19.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:20.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:19.915+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:20.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:20.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:20.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:20.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:20.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:20.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:20.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:20.160+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:21.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:20.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:21.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:20.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:21.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:20.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:21.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:20.751+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:21.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:20.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:21.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:20.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:21.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:20.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:21.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:20.874+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:21.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:21.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:21.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:21.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:21.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:21.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:21.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:21.191+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:21.990 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:31:21.620+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 20753 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:31:21.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:21.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:21.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:21.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:21.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:21.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:21.991 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:21.753+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:21.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:21.863+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:21.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:21.863+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:21.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:21.863+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:21.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:21.864+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:22.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:22.173+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:22.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:22.173+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:22.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:22.173+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:22.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:22.173+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:23.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:22.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:23.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:22.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:23.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:22.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:23.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:22.896+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:23.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:22.772+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:23.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:22.772+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:23.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:22.772+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:23.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:22.773+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:23.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:23.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:23.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:23.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:23.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:23.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:23.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:23.175+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:23.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:23.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:23.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:23.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:23.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:23.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:23.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:23.769+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:24.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:23.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:24.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:23.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:24.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:23.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:24.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:23.869+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:24.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:24.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:24.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:24.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:24.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:24.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:24.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:24.133+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:25.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:24.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:25.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:24.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:25.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:24.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:25.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:24.754+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:25.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:24.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:25.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:24.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:25.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:24.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:25.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:24.847+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:25.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:25.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:25.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:25.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:25.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:25.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:25.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:25.127+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:26.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:25.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:26.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:25.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:26.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:25.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:26.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:25.760+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:26.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:25.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:26.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:25.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:26.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:25.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:26.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:25.863+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:26.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:26.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:26.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:26.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:26.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:26.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:26.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:26.104+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:26.749 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:31:26.622+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 20842 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:31:27.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:26.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:27.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:26.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:27.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:26.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:27.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:26.848+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:27.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:26.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:27.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:26.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:27.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:26.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:27.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:26.761+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:27.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:27.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:27.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:27.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:27.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:27.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:27.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:27.119+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:28.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:27.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:28.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:27.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:28.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:27.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:28.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:27.853+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:28.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:27.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:28.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:27.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:28.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:27.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:28.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:27.761+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:28.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:28.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:28.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:28.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:28.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:28.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:28.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:28.075+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:29.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:28.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:29.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:28.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:29.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:28.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:29.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:28.869+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:29.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:28.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:29.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:28.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:29.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:28.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:29.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:28.751+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:29.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:29.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:29.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:29.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:29.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:29.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:29.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:29.048+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:30.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:29.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:30.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:29.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:30.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:29.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:30.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:29.917+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:30.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:29.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:30.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:29.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:30.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:29.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:30.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:29.763+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:30.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:30.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:30.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:30.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:30.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:30.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:30.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:30.015+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:31.064 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:30.929+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:31.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:30.929+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:31.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:30.929+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:31.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:30.930+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:31.065 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:30.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:31.065 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:30.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:31.065 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:30.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:31.065 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:30.795+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:31.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:31.061+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:31.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:31.061+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:31.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:31.061+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:31.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:31.061+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:31.976 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:31:31.625+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 20931 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:31:31.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:31.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:31.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:31.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:31.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:31.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:31.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:31.779+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:32.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:32.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:32.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:32.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:32.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:32.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:32.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:32.012+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:32.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:31.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:32.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:31.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:32.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:31.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:32.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:31.973+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:32.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:32.775+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:32.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:32.775+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:32.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:32.775+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:32.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:32.775+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:33.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:32.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:33.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:32.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:33.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:32.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:33.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:32.959+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:33.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:33.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:33.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:33.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:33.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:33.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:33.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:33.010+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:33.928 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:33.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:33.928 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:33.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:33.928 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:33.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:33.928 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:33.779+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:33.924+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:33.924+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:33.924+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:34.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:33.925+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:34.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:34.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:34.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:34.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:34.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:34.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:34.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:34.052+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:35.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:34.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:35.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:34.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:35.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:34.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:35.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:34.917+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:35.075 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:34.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:35.075 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:34.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:35.075 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:34.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:35.075 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:34.780+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:35.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:35.070+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:35.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:35.070+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:35.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:35.070+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:35.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:35.070+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:36.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:35.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:36.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:35.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:36.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:35.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:36.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:35.915+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:36.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:35.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:36.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:35.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:36.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:35.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:36.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:35.759+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:36.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:36.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:36.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:36.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:36.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:36.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:36.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:36.075+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:36.919 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:31:36.627+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 21021 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:31:36.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:36.802+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:36.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:36.802+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:36.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:36.802+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:36.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:36.803+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:37.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:37.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:37.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:37.054+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:37.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:36.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:37.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:36.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:37.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:36.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:37.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:36.916+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:38.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:37.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:38.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:37.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:38.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:37.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:38.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:37.786+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:38.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:38.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:38.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:38.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:38.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:38.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:38.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:38.014+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:38.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:37.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:38.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:37.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:38.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:37.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:38.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:37.946+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:39.034 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:38.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:39.034 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:38.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:39.034 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:38.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:39.034 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:38.778+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:39.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:38.966+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:39.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:38.966+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:39.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:38.966+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:39.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:38.966+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:39.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:39.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:39.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:39.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:39.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:39.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:39.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:39.030+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:40.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:39.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:40.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:39.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:40.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:39.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:40.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:39.936+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:40.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:39.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:40.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:39.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:40.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:39.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:40.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:39.739+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:40.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:40.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:40.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:40.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:40.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:40.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:40.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:40.064+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:41.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:40.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:41.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:40.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:41.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:40.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:41.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:40.975+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:41.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:40.729+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:41.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:40.729+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:41.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:40.729+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:41.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:40.729+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:41.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:41.080+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:41.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:41.080+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:41.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:41.080+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:41.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:41.080+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:41.945 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:31:41.630+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 21109 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:31:41.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:41.703+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:41.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:41.703+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:41.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:41.703+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:41.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:41.704+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:42.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:41.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:42.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:41.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:42.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:41.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:42.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:41.943+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:42.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:42.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:42.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:42.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:42.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:42.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:42.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:42.130+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:43.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:42.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:43.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:42.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:43.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:42.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:43.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:42.709+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:43.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:42.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:43.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:42.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:43.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:42.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:43.025 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:42.905+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:43.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:43.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:43.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:43.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:43.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:43.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:43.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:43.134+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:43.864 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:43.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:43.864 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:43.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:43.864 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:43.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:43.864 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:43.727+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:44.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:43.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:44.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:43.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:44.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:43.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:44.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:43.861+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:44.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:44.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:44.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:44.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:44.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:44.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:44.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:44.151+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:45.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:44.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:45.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:44.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:45.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:44.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:45.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:44.763+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:45.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:44.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:45.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:44.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:45.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:44.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:45.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:44.884+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:45.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:45.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:45.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:45.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:45.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:45.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:45.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:45.154+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:46.109 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:45.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:46.109 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:45.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:46.109 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:45.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:46.109 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:45.787+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:46.110 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:45.863+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:46.110 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:45.863+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:46.110 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:45.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:46.110 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:45.865+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:46.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:46.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:46.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:46.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:46.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:46.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:46.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:46.106+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:47.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:31:46.632+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 21194 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:31:47.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:46.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:47.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:46.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:47.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:46.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:47.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:46.874+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:47.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:46.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:47.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:46.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:47.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:46.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:47.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:46.747+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:47.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:47.061+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:47.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:47.061+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:47.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:47.061+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:47.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:47.062+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:48.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:47.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:48.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:47.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:48.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:47.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:48.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:47.743+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:48.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:47.866+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:48.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:47.866+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:48.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:47.866+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:48.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:47.866+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:48.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:48.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:48.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:48.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:48.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:48.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:48.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:48.019+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:49.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:48.844+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:49.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:48.844+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:49.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:48.844+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:49.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:48.845+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:49.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:48.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:49.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:48.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:49.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:48.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:49.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:48.750+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:49.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:49.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:49.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:49.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:49.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:49.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:49.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:49.003+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:50.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:49.817+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:50.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:49.817+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:50.195 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:49.817+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:50.195 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:49.818+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:50.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:49.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:50.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:49.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:50.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:49.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:50.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:49.800+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:50.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:50.006+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:50.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:50.006+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:50.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:50.006+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:50.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:50.007+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:51.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:50.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:51.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:50.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:51.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:50.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:51.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:50.999+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:51.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:50.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:51.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:50.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:51.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:50.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:51.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:50.834+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:51.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:50.830+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:51.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:50.830+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:51.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:50.830+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:51.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:50.831+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:52.004 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:31:51.634+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 21274 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:31:52.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:51.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:52.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:51.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:52.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:51.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:52.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:51.799+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:52.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:51.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:52.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:51.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:52.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:51.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:52.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:51.813+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:52.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:52.000+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:52.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:52.000+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:52.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:52.000+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:52.001+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:53.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:52.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:53.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:52.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:53.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:52.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:53.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:52.843+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:53.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:53.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:53.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:53.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:53.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:53.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:53.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:53.017+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:53.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:52.777+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:53.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:52.777+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:53.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:52.777+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:53.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:52.777+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:53.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:53.790+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:53.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:53.790+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:53.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:53.790+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:53.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:53.791+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:53.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:53.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:53.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:53.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:53.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:53.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:53.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:53.814+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:54.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:53.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:54.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:53.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:54.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:53.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:54.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:53.981+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:55.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:54.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:55.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:54.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:55.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:54.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:55.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:54.798+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:55.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:54.857+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:55.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:54.857+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:55.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:54.857+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:55.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:54.857+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:55.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:55.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:55.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:55.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:55.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:55.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:55.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:55.019+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:56.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:55.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:56.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:55.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:56.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:55.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:56.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:55.775+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:56.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:55.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:56.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:55.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:56.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:55.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:56.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:55.818+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:56.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:56.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:56.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:56.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:56.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:56.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:56.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:56.010+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:56.991 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:31:56.637+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 21354 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:31:56.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:56.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:56.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:56.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:56.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:56.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:56.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:56.811+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:56.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:56.792+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:56.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:56.792+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:56.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:56.792+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:56.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:56.793+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:57.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:56.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:57.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:56.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:57.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:56.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:57.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:56.987+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:58.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:57.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:58.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:57.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:58.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:57.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:58.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:57.779+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:58.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:57.795+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:58.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:57.795+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:58.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:57.795+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:58.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:57.796+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:58.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:57.956+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:58.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:57.956+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:58.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:57.956+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:58.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:57.956+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:59.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:58.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:59.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:58.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:59.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:58.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:59.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:58.768+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:31:59.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:58.757+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:59.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:58.757+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:59.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:58.757+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:59.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:58.758+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:59.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:58.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:59.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:58.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:59.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:58.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:59.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:58.948+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:59.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:59.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:59.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:59.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:59.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:59.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:31:59.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:31:59.792+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:59.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:59.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:59.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:59.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:59.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:59.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:31:59.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:31:59.982+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:31:59.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:59.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:59.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:59.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:59.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:59.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:31:59.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:31:59.733+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:00.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:00.790+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:00.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:00.790+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:00.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:00.790+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:00.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:00.791+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:00.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:00.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:00.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:00.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:00.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:00.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:00.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:00.720+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:01.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:00.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:01.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:00.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:01.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:00.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:01.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:00.974+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:02.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:32:01.639+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 21439 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:32:02.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:01.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:02.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:01.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:02.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:01.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:02.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:01.743+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:02.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:01.731+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:02.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:01.731+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:02.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:01.731+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:02.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:01.732+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:02.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:02.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:02.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:02.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:02.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:02.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:02.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:02.021+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:03.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:02.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:03.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:02.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:03.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:02.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:03.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:02.696+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:03.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:02.690+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:03.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:02.690+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:03.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:02.690+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:03.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:02.690+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:03.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:03.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:03.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:03.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:03.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:03.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:03.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:03.038+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:04.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:03.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:04.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:03.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:04.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:03.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:04.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:03.702+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:04.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:03.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:04.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:03.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:04.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:03.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:04.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:03.717+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:04.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:04.039+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:04.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:04.039+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:04.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:04.039+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:04.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:04.040+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:05.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:04.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:05.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:04.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:05.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:04.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:05.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:04.705+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:05.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:04.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:05.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:04.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:05.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:04.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:05.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:04.759+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:05.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:05.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:05.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:05.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:05.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:05.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:05.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:05.004+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:06.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:05.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:06.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:05.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:06.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:05.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:06.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:05.762+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:06.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:05.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:06.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:05.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:06.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:05.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:06.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:05.689+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:06.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:06.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:06.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:06.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:06.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:06.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:06.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:06.043+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:07.005 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:32:06.641+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 21522 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:32:07.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:06.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:07.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:06.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:07.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:06.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:07.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:06.699+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:07.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:06.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:07.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:06.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:07.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:06.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:07.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:06.804+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:07.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:07.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:07.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:07.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:07.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:07.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:07.002+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:08.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:07.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:08.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:07.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:08.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:07.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:08.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:07.707+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:08.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:07.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:08.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:07.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:08.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:07.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:08.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:07.777+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:08.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:08.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:08.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:08.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:08.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:08.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:08.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:08.031+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:09.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:08.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:09.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:08.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:09.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:08.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:09.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:08.758+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:09.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:08.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:09.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:08.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:09.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:08.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:09.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:08.669+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:09.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:09.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:09.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:09.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:09.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:09.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:09.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:09.048+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:09.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:09.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:09.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:09.669+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:10.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:09.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:09.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:09.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:09.788+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:10.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:10.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:10.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:10.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:10.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:10.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:10.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:10.013+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:11.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:10.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:11.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:10.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:11.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:10.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:11.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:10.836+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:11.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:10.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:11.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:10.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:11.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:10.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:11.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:10.625+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:11.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:11.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:11.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:11.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:11.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:11.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:11.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:11.048+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:11.851 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:32:11.644+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 21609 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:32:11.852 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:11.583+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:11.852 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:11.583+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:11.852 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:11.583+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:11.852 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:11.584+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:12.101 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:11.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:12.102 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:11.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:12.102 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:11.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:12.102 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:11.849+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:12.453 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:12.098+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:12.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:12.098+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:12.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:12.098+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:12.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:12.099+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:12.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:12.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:12.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:12.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:12.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:12.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:12.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:12.595+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:13.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:12.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:13.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:12.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:13.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:12.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:13.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:12.875+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:13.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:13.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:13.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:13.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:13.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:13.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:13.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:13.128+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:13.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:13.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:13.885 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:13.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:13.885 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:13.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:13.885 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:13.561+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:14.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:13.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:14.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:13.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:14.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:13.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:14.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:13.882+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:14.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:14.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:14.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:14.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:14.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:14.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:14.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:14.119+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:14.862 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:14.519+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:14.862 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:14.519+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:14.862 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:14.519+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:14.863 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:14.520+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:15.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:14.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:15.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:14.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:15.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:14.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:15.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:14.860+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:15.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:15.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:15.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:15.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:15.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:15.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:15.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:15.071+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:16.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:15.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:16.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:15.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:16.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:15.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:16.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:15.569+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:16.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:15.816+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:16.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:15.816+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:16.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:15.816+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:16.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:15.816+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:16.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:16.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:16.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:16.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:16.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:16.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:16.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:16.114+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:17.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:32:16.646+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 21698 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:32:17.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:16.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:17.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:16.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:17.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:16.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:17.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:16.592+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:17.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:16.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:17.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:16.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:17.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:16.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:17.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:16.771+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:17.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:17.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:17.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:17.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:17.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:17.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:17.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:17.111+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:17.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:17.596+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:17.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:17.596+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:17.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:17.596+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:17.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:17.596+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:18.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:17.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:18.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:17.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:18.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:17.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:18.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:17.812+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:18.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:18.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:18.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:18.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:18.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:18.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:18.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:18.138+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:19.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:18.638+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:19.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:18.638+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:19.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:18.638+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:19.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:18.639+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:19.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:18.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:19.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:18.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:19.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:18.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:19.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:18.833+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:19.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:19.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:19.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:19.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:19.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:19.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:19.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:19.095+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:20.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:19.622+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:20.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:19.622+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:20.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:19.622+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:20.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:19.623+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:20.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:19.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:20.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:19.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:20.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:19.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:20.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:19.786+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:20.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:20.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:20.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:20.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:20.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:20.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:20.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:20.126+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:21.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:20.662+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:21.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:20.662+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:21.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:20.662+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:21.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:20.663+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:21.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:20.827+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:21.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:20.827+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:21.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:20.827+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:21.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:20.827+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:21.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:21.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:21.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:21.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:21.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:21.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:21.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:21.121+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:22.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:32:21.649+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 21789 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:32:22.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:21.822+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:22.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:21.822+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:22.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:21.822+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:22.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:21.823+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:22.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:21.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:22.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:21.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:22.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:21.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:22.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:21.667+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:22.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:22.120+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:22.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:22.120+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:22.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:22.120+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:22.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:22.120+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:23.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:22.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:23.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:22.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:23.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:22.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:23.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:22.778+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:23.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:22.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:23.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:22.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:23.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:22.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:23.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:22.651+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:23.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:23.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:23.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:23.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:23.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:23.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:23.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:23.161+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:24.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:23.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:24.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:23.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:24.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:23.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:24.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:23.803+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:24.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:23.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:24.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:23.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:24.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:23.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:24.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:23.627+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:24.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:24.140+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:24.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:24.140+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:24.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:24.140+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:24.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:24.140+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:24.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:25.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:24.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:25.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:24.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:25.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:24.818+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:25.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:24.654+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:25.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:24.654+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:25.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:24.654+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:25.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:24.655+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:25.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:25.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:25.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:25.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:25.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:25.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:25.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:25.139+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:26.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:25.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:26.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:25.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:26.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:25.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:26.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:25.812+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:26.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:25.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:26.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:25.662+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:26.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:25.662+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:26.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:25.663+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:26.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:26.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:26.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:26.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:26.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:26.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:26.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:26.143+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:27.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:32:26.651+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 21879 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:32:27.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:26.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:27.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:26.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:27.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:26.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:27.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:26.616+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:27.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:26.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:27.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:26.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:27.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:26.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:27.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:26.829+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:27.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:27.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:27.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:27.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:27.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:27.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:27.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:27.133+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:27.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:27.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:27.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:27.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:27.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:27.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:27.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:27.652+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:28.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:27.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:28.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:27.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:28.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:27.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:28.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:27.869+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:28.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:28.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:28.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:28.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:28.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:28.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:28.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:28.176+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:29.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:28.891+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:29.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:28.891+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:29.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:28.891+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:29.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:28.892+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:29.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:28.646+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:29.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:28.646+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:29.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:28.646+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:29.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:28.646+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:29.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:29.157+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:29.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:29.157+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:29.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:29.157+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:29.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:29.157+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:30.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:29.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:30.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:29.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:30.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:29.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:30.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:29.885+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:30.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:29.653+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:30.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:29.653+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:30.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:29.653+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:30.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:29.653+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:30.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:30.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:30.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:30.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:30.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:30.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:30.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:30.161+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:30.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:30.685+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:30.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:30.685+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:30.685+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:30.686+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:30.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:30.851+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:30.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:30.851+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:30.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:30.851+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:30.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:30.852+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:31.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:31.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:31.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:31.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:31.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:31.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:31.127+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:32.022 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:32:31.654+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 21969 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:32:32.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:31.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:32.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:31.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:32.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:31.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:32.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:31.709+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:32.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:31.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:32.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:31.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:32.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:31.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:32.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:31.850+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:32.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:32.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:32.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:32.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:32.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:32.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:32.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:32.087+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:33.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:32.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:33.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:32.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:33.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:32.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:33.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:32.736+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:33.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:32.800+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:33.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:32.800+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:33.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:32.800+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:33.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:32.801+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:33.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:33.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:33.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:33.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:33.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:33.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:33.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:33.131+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:34.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:33.800+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:34.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:33.800+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:34.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:33.800+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:34.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:33.801+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:34.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:33.769+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:34.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:33.769+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:34.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:33.769+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:34.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:33.769+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:34.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:34.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:34.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:34.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:34.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:34.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:34.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:34.100+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:35.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:34.818+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:35.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:34.818+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:35.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:34.818+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:35.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:34.819+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:35.135 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:34.800+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:35.135 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:34.800+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:35.135 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:34.800+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:35.136 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:34.801+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:35.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:35.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:35.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:35.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:35.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:35.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:35.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:35.132+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:36.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:35.781+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:36.108 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:35.781+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:36.108 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:35.781+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:36.108 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:35.781+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:36.108 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:35.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:36.108 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:35.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:36.108 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:35.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:36.109 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:35.819+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:36.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:36.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:36.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:36.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:36.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:36.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:36.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:36.105+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:37.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:32:36.656+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 22059 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:32:37.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:36.743+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:37.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:36.743+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:37.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:36.743+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:37.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:36.744+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:37.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:36.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:37.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:36.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:37.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:36.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:37.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:36.805+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:37.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:37.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:37.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:37.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:37.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:37.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:37.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:37.129+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:38.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:37.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:38.025 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:37.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:38.025 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:37.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:38.025 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:37.704+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:38.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:37.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:38.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:37.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:38.026 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:37.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:38.026 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:37.768+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:38.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:38.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:38.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:38.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:38.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:38.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:38.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:38.121+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:39.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:38.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:39.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:38.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:39.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:38.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:39.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:38.729+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:39.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:38.750+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:39.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:38.750+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:39.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:38.750+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:39.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:38.751+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:39.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:39.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:39.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:39.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:39.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:39.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:39.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:39.107+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:40.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:39.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:40.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:39.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:40.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:39.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:40.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:39.728+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:40.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:39.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:40.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:39.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:40.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:39.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:40.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:39.700+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:40.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:40.106+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:40.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:40.106+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:40.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:40.106+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:40.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:40.107+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:41.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:40.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:41.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:40.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:41.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:40.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:41.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:40.708+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:41.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:40.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:41.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:40.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:41.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:40.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:41.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:40.727+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:41.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:41.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:41.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:41.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:41.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:41.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:41.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:41.075+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:42.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:32:41.659+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 22150 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:32:42.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:41.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:42.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:41.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:42.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:41.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:42.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:41.695+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:42.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:41.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:42.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:41.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:42.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:41.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:42.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:41.708+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:42.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:42.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:42.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:42.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:42.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:42.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:42.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:42.109+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:43.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:42.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:43.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:42.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:43.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:42.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:43.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:42.687+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:43.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:42.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:43.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:42.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:43.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:42.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:43.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:42.671+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:43.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:43.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:43.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:43.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:43.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:43.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:43.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:43.078+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:44.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:43.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:44.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:43.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:44.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:43.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:44.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:43.712+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:44.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:43.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:44.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:43.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:44.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:43.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:44.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:43.712+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:44.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:44.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:44.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:44.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:44.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:44.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:44.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:44.051+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:45.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:44.737+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:45.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:44.737+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:45.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:44.737+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:45.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:44.738+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:45.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:44.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:45.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:44.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:45.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:44.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:45.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:44.755+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:45.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:45.057+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:45.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:45.057+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:45.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:45.057+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:45.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:45.057+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:46.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:45.775+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:46.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:45.775+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:46.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:45.775+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:46.016 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:45.776+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:46.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:45.736+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:46.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:45.736+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:46.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:45.736+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:46.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:45.737+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:46.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:46.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:46.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:46.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:46.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:46.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:46.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:46.012+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:47.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:32:46.661+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 22239 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:32:47.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:46.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:47.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:46.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:47.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:46.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:47.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:46.726+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:47.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:46.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:47.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:46.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:47.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:46.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:47.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:46.762+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:47.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:47.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:47.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:47.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:47.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:47.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:47.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:47.053+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:48.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:47.730+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:48.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:47.730+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:48.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:47.730+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:48.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:47.732+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:48.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:47.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:48.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:47.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:48.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:47.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:48.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:47.735+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:48.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:48.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:48.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:48.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:48.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:48.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:48.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:48.055+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:49.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:48.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:49.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:48.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:49.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:48.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:49.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:48.781+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:49.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:48.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:49.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:48.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:49.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:48.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:49.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:48.700+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:49.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:49.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:49.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:49.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:49.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:49.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:49.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:49.104+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:50.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:49.734+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:50.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:49.734+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:50.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:49.734+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:50.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:49.734+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:50.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:49.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:50.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:49.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:50.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:49.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:50.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:49.669+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:50.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:50.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:50.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:50.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:50.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:50.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:50.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:50.119+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:51.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:50.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:51.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:50.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:51.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:50.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:51.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:50.765+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:51.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:50.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:51.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:50.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:51.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:50.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:51.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:50.701+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:51.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:51.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:51.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:51.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:51.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:51.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:51.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:51.131+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:52.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:32:51.664+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 22329 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:32:52.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:51.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:52.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:51.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:52.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:51.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:52.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:51.753+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:52.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:51.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:52.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:51.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:52.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:51.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:52.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:51.720+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:52.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:52.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:52.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:52.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:52.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:52.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:52.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:52.172+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:52.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:52.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:52.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:52.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:52.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:52.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:52.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:52.672+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:53.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:52.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:53.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:52.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:53.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:52.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:53.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:52.783+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:53.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:53.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:53.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:53.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:53.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:53.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:53.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:53.142+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:54.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:53.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:54.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:53.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:54.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:53.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:54.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:53.807+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:54.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:53.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:54.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:53.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:54.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:53.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:54.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:53.684+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:54.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:54.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:54.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:54.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:54.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:54.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:54.134+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:55.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:54.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:55.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:54.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:55.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:54.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:55.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:54.810+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:55.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:54.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:55.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:54.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:55.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:54.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:55.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:54.693+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:55.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:55.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:55.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:55.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:55.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:55.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:55.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:55.139+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:55.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:55.674+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:55.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:55.674+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:55.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:55.674+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:55.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:55.675+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:56.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:55.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:56.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:55.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:56.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:55.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:56.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:55.828+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:56.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:56.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:56.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:56.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:56.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:56.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:56.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:56.125+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:56.794 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:32:56.666+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 22419 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:32:56.794 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:56.642+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:56.794 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:56.642+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:56.794 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:56.642+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:56.795 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:56.643+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:57.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:56.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:57.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:56.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:57.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:56.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:57.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:56.850+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:57.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:57.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:57.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:57.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:57.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:57.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:57.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:57.143+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:58.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:57.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:58.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:57.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:58.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:57.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:58.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:57.693+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:58.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:57.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:58.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:57.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:58.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:57.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:58.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:57.895+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:58.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:58.157+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:58.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:58.157+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:58.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:58.157+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:58.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:58.158+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:59.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:58.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:59.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:58.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:59.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:58.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:32:59.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:58.696+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:32:59.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:58.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:59.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:58.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:59.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:58.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:32:59.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:58.940+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:32:59.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:59.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:59.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:59.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:59.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:59.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:32:59.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:32:59.177+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:00.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:59.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:00.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:59.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:00.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:59.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:00.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:32:59.676+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:00.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:59.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:00.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:59.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:00.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:59.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:00.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:32:59.914+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:00.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:00.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:00.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:00.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:00.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:00.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:00.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:00.153+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:01.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:00.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:01.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:00.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:01.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:00.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:01.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:00.693+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:01.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:00.884+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:01.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:00.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:01.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:00.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:01.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:00.885+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:01.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:01.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:01.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:01.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:01.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:01.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:01.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:01.203+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:02.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:33:01.669+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 22510 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:33:02.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:01.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:02.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:01.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:02.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:01.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:02.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:01.878+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:02.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:01.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:02.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:01.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:02.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:01.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:02.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:01.740+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:02.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:02.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:02.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:02.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:02.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:02.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:02.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:02.196+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:03.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:02.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:03.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:02.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:03.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:02.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:03.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:02.900+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:03.151 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:02.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:03.151 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:02.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:03.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:02.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:03.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:02.787+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:03.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:03.147+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:03.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:03.147+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:03.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:03.147+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:03.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:03.148+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:03.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:03.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:03.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:03.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:03.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:03.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:03.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:03.778+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:04.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:03.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:04.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:03.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:04.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:03.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:04.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:03.890+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:04.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:04.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:04.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:04.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:04.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:04.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:04.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:04.127+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:05.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:04.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:05.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:04.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:05.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:04.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:05.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:04.938+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:05.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:04.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:05.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:04.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:05.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:04.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:05.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:04.750+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:05.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:05.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:05.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:05.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:05.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:05.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:05.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:05.164+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:06.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:05.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:06.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:05.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:06.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:05.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:06.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:05.742+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:06.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:05.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:06.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:05.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:06.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:05.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:06.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:05.919+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:06.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:06.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:06.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:06.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:06.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:06.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:06.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:06.124+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:07.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:33:06.671+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 22600 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:33:07.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:06.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:07.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:06.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:07.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:06.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:07.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:06.895+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:07.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:06.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:07.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:06.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:07.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:06.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:07.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:06.702+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:07.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:07.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:07.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:07.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:07.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:07.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:07.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:07.143+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:08.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:07.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:08.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:07.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:08.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:07.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:08.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:07.719+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:08.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:07.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:08.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:07.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:08.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:07.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:08.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:07.912+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:08.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:08.128+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:08.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:08.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:08.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:08.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:08.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:08.129+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:09.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:08.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:09.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:08.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:09.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:08.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:09.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:08.910+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:09.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:08.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:09.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:08.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:09.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:08.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:09.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:08.712+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:09.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:09.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:09.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:09.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:09.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:09.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:09.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:09.110+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:10.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:09.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:10.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:09.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:10.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:09.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:10.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:09.861+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:10.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:09.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:10.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:09.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:10.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:09.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:10.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:09.692+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:10.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:10.101+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:10.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:10.101+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:10.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:10.101+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:10.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:10.101+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:11.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:10.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:11.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:10.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:11.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:10.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:11.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:10.834+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:11.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:10.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:11.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:10.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:11.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:10.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:11.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:10.734+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:11.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:11.086+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:11.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:11.086+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:11.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:11.086+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:11.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:11.086+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:12.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:33:11.674+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 22690 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:33:12.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:11.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:12.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:11.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:12.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:11.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:12.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:11.721+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:12.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:11.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:12.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:11.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:12.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:11.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:12.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:11.876+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:12.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:12.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:12.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:12.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:12.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:12.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:12.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:12.049+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:13.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:12.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:13.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:12.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:13.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:12.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:13.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:12.927+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:13.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:12.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:13.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:12.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:13.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:12.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:13.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:12.751+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:13.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:13.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:13.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:13.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:13.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:13.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:13.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:13.066+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:13.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:13.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:13.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:13.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:13.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:13.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:13.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:13.765+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:14.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:13.965+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:14.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:13.965+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:14.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:13.965+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:14.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:13.965+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:14.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:14.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:14.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:14.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:14.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:14.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:14.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:14.053+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:15.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:14.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:15.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:14.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:15.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:14.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:15.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:14.935+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:15.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:14.722+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:15.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:14.722+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:15.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:14.722+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:15.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:14.723+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:15.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:15.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:15.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:15.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:15.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:15.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:15.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:15.026+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:16.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:15.940+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:16.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:15.940+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:16.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:15.940+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:16.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:15.941+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:16.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:15.743+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:16.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:15.743+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:16.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:15.743+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:16.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:15.743+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:16.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:16.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:16.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:16.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:16.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:16.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:16.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:16.033+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:16.975 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:33:16.676+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 22780 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:33:16.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:16.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:16.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:16.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:16.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:16.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:16.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:16.771+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:17.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:16.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:17.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:16.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:17.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:16.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:17.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:16.973+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:17.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:17.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:17.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:17.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:17.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:17.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:17.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:17.072+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:18.048 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:18.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:18.048 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:18.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:18.048 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:18.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:18.049 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:18.019+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:18.049 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:17.788+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:18.049 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:17.788+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:18.049 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:17.788+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:18.049 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:17.789+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:18.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:18.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:18.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:18.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:18.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:18.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:18.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:18.055+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:19.105 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:18.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:19.105 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:18.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:19.106 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:18.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:19.106 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:18.983+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:19.106 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:18.771+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:19.106 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:18.771+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:19.106 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:18.771+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:19.107 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:18.772+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:19.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:19.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:19.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:19.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:19.102+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:20.092 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:19.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:20.092 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:19.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:20.092 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:19.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:20.092 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:19.808+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:20.093 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:19.954+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:20.093 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:19.954+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:20.093 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:19.954+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:20.093 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:19.955+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:20.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:20.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:20.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:20.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:20.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:20.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:20.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:20.089+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:21.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:20.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:21.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:20.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:21.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:20.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:21.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:20.976+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:21.103 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:20.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:21.104 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:20.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:21.104 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:20.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:21.104 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:20.811+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:21.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:21.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:21.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:21.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:21.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:21.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:21.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:21.100+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:22.009 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:33:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:33:21.679+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 22871 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:33:22.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:21.815+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:22.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:21.815+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:22.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:21.815+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:22.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:21.816+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:22.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:22.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:22.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:22.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:22.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:22.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:22.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:22.006+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:22.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:22.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:22.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:22.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:22.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:22.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:22.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:22.079+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:23.189 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:23.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:23.189 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:23.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:23.189 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:23.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:23.189 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:23.030+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:23.190 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:22.858+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:23.190 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:22.858+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:23.190 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:22.858+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:23.190 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:22.859+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:23.190 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:23.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:23.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:23.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:23.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:23.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:23.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:23.078+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:24.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:23.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:24.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:23.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:24.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:23.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:24.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:23.993+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:24.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:24.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:24.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:24.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:24.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:24.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:24.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:24.060+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:24.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:23.865+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:24.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:23.865+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:24.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:23.865+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:24.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:23.866+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:25.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:25.012+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:25.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:25.012+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:25.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:25.012+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:25.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:25.013+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:25.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:24.899+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:25.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:24.899+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:25.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:24.899+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:25.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:24.899+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:25.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:25.034+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:25.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:25.034+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:25.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:25.034+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:25.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:25.034+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:26.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:26.044+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:26.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:26.044+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:26.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:26.044+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:26.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:26.044+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:26.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:25.911+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:26.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:25.911+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:26.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:25.911+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:26.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:25.912+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:26.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:25.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:26.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:25.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:26.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:25.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:26.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:25.986+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:26.945 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:33:26.682+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 22961 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:33:26.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:26.922+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:26.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:26.922+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:26.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:26.922+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:26.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:26.923+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:26.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:27.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:26.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:27.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:26.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:27.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:26.942+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:27.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:27.067+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:27.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:27.067+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:27.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:27.067+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:27.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:27.068+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:28.110+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:28.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:28.110+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:28.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:28.110+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:28.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:28.111+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:28.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:27.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:28.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:27.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:28.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:27.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:28.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:27.949+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:28.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:27.902+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:28.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:27.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:28.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:27.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:28.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:27.903+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:29.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:28.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:29.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:28.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:29.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:28.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:29.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:28.854+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:29.114 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:28.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:29.114 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:28.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:29.114 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:28.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:29.114 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:28.942+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:29.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:29.110+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:29.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:29.110+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:29.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:29.110+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:29.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:29.111+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:30.077 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:29.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:30.078 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:29.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:30.078 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:29.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:30.078 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:29.818+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:30.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:29.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:30.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:29.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:30.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:29.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:30.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:29.909+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:30.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:30.074+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:30.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:30.074+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:30.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:30.074+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:30.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:30.075+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:31.094 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:30.951+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:31.094 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:30.951+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:31.095 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:30.951+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:31.095 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:30.952+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:31.095 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:30.781+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:31.095 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:30.781+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:31.095 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:30.781+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:31.095 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:30.781+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:31.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:31.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:31.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:31.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:31.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:31.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:31.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:31.092+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:32.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:33:31.684+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 23050 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:33:32.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:31.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:32.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:31.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:32.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:31.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:32.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:31.906+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:32.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:31.734+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:32.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:31.734+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:32.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:31.734+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:32.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:31.734+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:32.405 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:32.100+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:32.405 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:32.100+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:32.405 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:32.100+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:32.405 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:32.100+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:33.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:32.696+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:33.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:32.696+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:33.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:32.696+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:33.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:32.697+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:33.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:32.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:33.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:32.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:33.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:32.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:33.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:32.883+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:33.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:33.122+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:33.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:33.122+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:33.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:33.122+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:33.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:33.123+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:33.848 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:33.706+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:33.849 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:33.706+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:33.849 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:33.706+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:33.849 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:33.706+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:34.172 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:33.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:34.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:33.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:34.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:33.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:34.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:33.847+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:34.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:34.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:34.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:34.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:34.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:34.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:34.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:34.170+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:35.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:34.710+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:35.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:34.710+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:35.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:34.710+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:35.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:34.710+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:35.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:34.876+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:35.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:34.876+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:35.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:34.876+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:35.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:34.877+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:35.477 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:35.207+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:35.477 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:35.207+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:35.477 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:35.207+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:35.477 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:35.208+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:35.855+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:36.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:35.855+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:36.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:35.855+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:36.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:35.855+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:36.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:35.729+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:36.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:35.729+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:36.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:35.729+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:36.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:35.729+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:36.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:36.234+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:36.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:36.234+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:36.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:36.234+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:36.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:36.234+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:37.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:33:36.687+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 23140 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:33:37.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:36.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:37.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:36.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:37.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:36.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:37.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:36.863+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:37.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:36.773+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:37.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:36.773+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:37.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:36.773+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:37.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:36.774+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:37.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:37.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:37.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:37.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:37.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:37.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:37.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:37.201+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:38.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:37.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:38.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:37.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:38.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:37.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:38.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:37.822+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:38.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:37.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:38.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:37.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:38.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:37.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:38.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:37.889+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:38.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:38.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:38.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:38.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:38.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:38.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:38.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:38.158+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:39.122 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:38.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:39.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:38.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:39.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:38.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:39.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:38.920+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:39.123 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:38.837+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:39.123 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:38.837+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:39.123 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:38.837+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:39.123 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:38.837+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:39.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:39.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:39.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:39.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:39.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:39.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:39.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:39.120+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:40.125 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:39.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:40.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:39.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:40.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:39.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:40.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:39.969+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:40.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:39.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:40.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:39.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:40.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:39.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:40.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:39.816+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:40.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:40.122+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:40.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:40.122+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:40.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:40.122+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:40.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:40.123+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:41.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:40.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:41.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:40.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:41.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:40.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:41.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:40.779+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:41.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:40.973+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:41.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:40.973+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:41.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:40.973+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:41.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:40.973+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:41.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:41.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:41.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:41.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:41.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:41.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:41.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:41.173+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:42.012 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:33:41.690+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 23231 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:33:42.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:41.815+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:42.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:41.815+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:42.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:41.815+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:42.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:41.816+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:42.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:42.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:42.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:42.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:42.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:42.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:42.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:42.126+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:42.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:42.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:42.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:42.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:42.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:42.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:42.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:42.010+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:43.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:43.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:43.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:43.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:43.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:43.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:43.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:43.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:43.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:43.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:43.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:43.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:43.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:43.031+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:43.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:42.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:43.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:42.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:43.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:42.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:43.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:42.859+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:43.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:43.110+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:44.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:44.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:44.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:44.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:44.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:44.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:44.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:44.140+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:44.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:43.902+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:44.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:43.902+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:44.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:43.902+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:44.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:43.903+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:44.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:44.034+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:44.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:44.034+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:44.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:44.034+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:44.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:44.035+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:45.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:44.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:45.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:44.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:45.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:44.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:45.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:44.921+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:45.129+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:45.129+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:45.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:45.129+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:45.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:45.130+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:45.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:45.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:45.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:45.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:45.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:45.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:45.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:45.008+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:46.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:46.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:46.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:46.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:46.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:46.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:46.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:46.022+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:46.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:45.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:46.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:45.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:46.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:45.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:46.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:45.938+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:46.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:46.118+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:46.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:46.118+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:46.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:46.118+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:46.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:46.119+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:47.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:33:46.691+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 23321 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:33:47.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:46.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:47.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:46.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:47.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:46.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:47.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:46.908+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:47.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:47.116+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:47.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:47.116+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:47.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:47.116+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:47.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:47.116+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:47.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:47.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:47.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:47.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:47.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:47.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:47.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:47.018+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:48.150 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:48.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:48.151 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:48.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:48.151 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:48.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:48.151 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:48.069+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:48.151 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:47.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:48.151 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:47.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:48.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:47.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:48.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:47.942+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:48.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:48.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:48.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:48.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:48.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:48.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:48.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:48.148+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:48.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:48.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:48.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:49.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:48.964+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:49.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:49.179+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:49.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:49.179+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:49.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:49.179+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:49.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:49.179+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:49.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:49.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:49.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:49.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:49.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:49.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:49.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:49.118+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:50.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:50.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:50.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:50.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:50.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:50.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:50.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:50.173+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:50.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:50.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:50.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:50.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:50.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:50.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:50.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:50.114+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:50.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:49.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:50.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:49.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:50.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:49.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:50.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:49.931+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:51.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:51.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:51.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:51.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:51.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:51.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:51.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:51.069+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:51.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:50.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:51.204 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:50.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:51.204 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:50.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:51.204 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:50.900+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:51.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:51.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:51.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:51.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:51.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:51.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:51.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:51.201+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:52.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:33:51.695+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 23410 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:33:52.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:51.853+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:52.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:51.853+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:52.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:51.853+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:52.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:51.854+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:52.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:52.190+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:52.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:52.190+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:52.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:52.190+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:52.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:52.191+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:52.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:52.102+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:52.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:52.102+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:52.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:52.102+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:52.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:52.103+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:53.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:53.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:53.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:53.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:53.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:53.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:53.172 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:53.138+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:53.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:52.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:53.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:52.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:53.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:52.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:53.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:52.904+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:53.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:53.168+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:53.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:53.168+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:53.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:53.168+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:53.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:53.169+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:54.199 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:54.099+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:54.199 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:54.099+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:54.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:54.099+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:54.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:54.100+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:54.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:53.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:54.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:53.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:54.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:53.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:54.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:53.888+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:54.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:54.196+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:54.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:54.196+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:54.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:54.196+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:54.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:54.197+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:55.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:54.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:55.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:54.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:55.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:54.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:55.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:54.891+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:55.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:55.075+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:55.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:55.075+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:55.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:55.075+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:55.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:55.076+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:55.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:55.167+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:55.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:55.167+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:55.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:55.167+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:55.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:55.167+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:55.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:55.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:56.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:55.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:56.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:55.906+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:56.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:56.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:56.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:56.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:56.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:56.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:56.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:56.141+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:56.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:56.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:56.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:56.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:56.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:56.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:56.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:56.046+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:57.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:33:56.698+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 23501 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:33:57.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:56.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:57.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:56.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:57.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:56.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:57.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:56.896+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:57.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:57.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:57.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:57.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:57.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:57.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:57.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:57.141+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:57.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:57.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:57.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:57.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:57.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:57.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:57.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:57.060+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:58.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:57.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:58.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:57.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:58.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:57.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:58.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:57.883+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:58.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:58.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:58.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:58.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:58.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:58.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:58.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:58.040+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:58.411 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:58.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:58.411 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:58.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:58.412 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:58.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:58.412 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:58.152+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:59.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:59.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:59.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:59.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:59.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:59.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:33:59.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:59.004+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:33:59.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:58.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:59.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:58.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:59.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:58.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:59.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:58.839+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:59.402 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:59.139+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:59.403 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:59.139+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:59.403 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:59.139+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:33:59.403 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:33:59.140+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:33:59.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:59.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:59.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:59.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:59.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:59.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:33:59.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:33:59.863+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:00.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:00.167+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:00.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:00.167+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:00.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:00.167+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:00.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:00.168+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:00.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:59.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:00.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:59.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:00.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:59.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:00.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:33:59.980+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:00.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:00.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:00.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:00.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:00.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:00.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:00.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:00.845+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:01.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:01.135+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:01.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:01.135+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:01.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:01.135+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:01.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:01.136+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:01.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:00.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:01.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:00.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:01.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:00.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:01.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:00.966+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:01.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:34:01.701+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 23591 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:34:01.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:01.827+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:01.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:01.827+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:01.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:01.827+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:01.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:01.828+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:02.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:02.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:02.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:02.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:02.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:02.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:02.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:02.153+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:02.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:01.964+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:02.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:01.964+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:02.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:01.964+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:02.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:01.965+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:03.125 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:02.972+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:03.125 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:02.972+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:03.125 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:02.972+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:03.125 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:02.973+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:03.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:02.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:03.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:02.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:03.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:02.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:03.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:02.796+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:03.458 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:03.185+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:03.458 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:03.185+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:03.458 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:03.185+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:03.458 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:03.186+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:04.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:03.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:04.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:03.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:04.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:03.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:04.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:03.957+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:04.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:03.837+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:04.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:03.837+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:04.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:03.837+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:04.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:03.838+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:04.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:04.152+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:04.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:04.152+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:04.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:04.152+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:04.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:04.154+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:05.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:04.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:05.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:04.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:05.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:04.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:05.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:04.871+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:05.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:05.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:05.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:05.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:05.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:05.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:05.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:05.007+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:05.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:05.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:05.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:05.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:05.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:05.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:05.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:05.156+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:06.198 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:06.014+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:06.198 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:06.014+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:06.198 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:06.014+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:06.198 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:06.015+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:06.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:05.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:06.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:05.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:06.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:05.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:06.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:05.872+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:06.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:06.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:06.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:06.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:06.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:06.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:06.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:06.202+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:07.009 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:34:06.704+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 23682 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:34:07.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:06.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:07.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:06.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:07.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:06.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:07.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:06.909+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:07.223+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:07.223+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:07.223+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:07.224+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:07.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:07.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:07.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:07.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:07.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:07.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:07.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:07.006+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:08.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:08.015+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:08.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:08.015+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:08.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:08.015+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:08.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:08.016+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:08.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:07.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:08.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:07.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:08.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:07.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:08.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:07.936+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:08.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:08.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:08.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:08.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:08.510 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:08.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:08.510 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:08.212+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:09.239 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:09.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:09.239 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:09.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:09.239 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:09.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:09.239 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:09.044+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:09.239 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:08.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:09.239 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:08.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:09.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:08.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:09.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:08.966+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:09.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:09.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:09.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:09.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:09.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:09.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:09.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:09.236+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:10.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:10.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:10.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:10.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:10.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:10.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:10.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:10.032+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:10.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:09.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:10.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:09.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:10.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:09.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:10.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:09.955+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:10.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:10.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:10.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:10.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:10.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:10.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:10.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:10.262+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:10.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:10.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:10.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:10.950+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:11.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:11.025+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:11.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:11.025+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:11.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:11.025+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:11.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:11.026+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:11.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:11.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:11.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:11.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:11.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:11.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:11.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:11.270+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:11.990 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:34:11.706+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 23771 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:34:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:12.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:12.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:12.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:12.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:12.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:12.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:12.076+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:12.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:11.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:12.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:11.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:12.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:11.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:12.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:11.987+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:12.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:12.317+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:12.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:12.317+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:12.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:12.317+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:12.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:12.318+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:13.353 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:13.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:13.354 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:13.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:13.354 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:13.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:13.354 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:13.113+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:13.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:13.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:13.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:13.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:13.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:13.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:13.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:13.034+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:13.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:13.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:13.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:13.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:13.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:13.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:13.629 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:13.352+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:14.072+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:14.072+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:14.072+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:14.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:14.072+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:14.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:13.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:14.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:13.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:14.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:13.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:14.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:13.991+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:14.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:14.325+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:14.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:14.325+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:14.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:14.325+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:14.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:14.326+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:14.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:15.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:14.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:15.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:14.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:15.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:14.981+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:15.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:15.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:15.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:15.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:15.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:15.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:15.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:15.089+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:15.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:15.360+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:15.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:15.360+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:15.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:15.360+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:15.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:15.361+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:16.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:16.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:16.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:16.091+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:16.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:16.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:16.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:16.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:16.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:16.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:16.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:16.005+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:16.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:16.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:16.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:16.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:16.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:16.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:16.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:16.359+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:17.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:34:16.709+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 23862 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:34:17.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:17.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:17.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:17.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:17.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:17.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:17.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:17.107+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:17.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:17.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:17.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:17.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:17.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:17.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:17.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:17.025+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:17.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:17.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:17.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:17.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:17.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:17.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:17.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:17.357+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:18.313 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:18.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:18.313 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:18.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:18.313 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:18.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:18.313 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:18.154+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:18.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:18.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:18.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:18.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:18.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:18.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:18.314 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:18.055+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:18.603 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:18.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:18.603 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:18.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:18.603 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:18.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:18.603 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:18.311+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:19.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:19.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:19.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:19.066+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:19.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:19.186+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:19.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:19.186+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:19.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:19.186+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:19.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:19.187+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:19.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:19.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:19.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:19.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:19.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:19.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:19.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:19.301+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:20.338 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:20.179+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:20.339 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:20.179+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:20.339 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:20.179+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:20.339 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:20.179+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:20.339 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:20.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:20.339 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:20.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:20.339 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:20.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:20.339 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:20.028+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:20.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:20.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:20.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:20.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:20.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:20.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:20.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:20.337+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:21.333 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:21.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:21.333 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:21.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:21.333 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:21.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:21.333 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:21.130+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:21.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:21.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:21.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:21.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:21.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:21.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:21.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:21.049+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:21.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:21.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:21.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:21.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:21.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:21.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:21.331+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:22.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:34:21.712+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 23952 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:34:22.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:22.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:22.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:22.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:22.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:22.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:22.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:22.098+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:22.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:22.031+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:22.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:22.031+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:22.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:22.031+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:22.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:22.032+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:22.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:22.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:22.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:22.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:22.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:22.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:22.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:22.359+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:23.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:22.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:23.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:22.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:23.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:22.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:23.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:22.991+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:23.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:23.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:23.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:23.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:23.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:23.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:23.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:23.066+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:23.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:23.374+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:23.629 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:23.374+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:23.629 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:23.374+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:23.629 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:23.375+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:24.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:24.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:24.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:24.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:24.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:24.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:24.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:24.079+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:24.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:23.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:24.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:23.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:24.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:23.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:24.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:23.962+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:24.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:24.394+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:24.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:24.394+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:24.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:24.394+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:24.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:24.394+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:25.107+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:25.107+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:25.107+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:25.108+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:25.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:24.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:25.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:24.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:25.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:24.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:25.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:24.966+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:25.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:25.433+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:25.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:25.433+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:25.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:25.433+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:25.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:25.433+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:26.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:26.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:26.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:26.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:26.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:26.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:26.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:26.078+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:26.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:25.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:26.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:25.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:26.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:25.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:26.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:25.990+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:26.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:26.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:26.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:26.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:26.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:26.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:26.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:26.441+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:27.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:34:26.715+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 24043 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:34:27.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:26.944+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:27.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:26.944+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:27.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:26.944+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:27.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:26.945+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:27.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:27.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:27.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:27.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:27.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:27.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:27.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:27.093+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:27.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:27.465+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:27.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:27.465+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:27.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:27.465+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:27.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:27.466+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:28.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:28.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:28.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:28.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:28.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:28.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:28.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:28.094+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:28.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:27.911+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:28.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:27.911+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:28.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:27.911+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:28.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:27.912+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:28.494+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:28.494+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:28.494+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:28.495+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:29.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:29.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:29.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:29.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:29.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:29.070+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:29.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:28.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:29.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:28.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:29.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:28.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:29.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:28.961+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:29.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:29.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:29.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:29.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:29.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:29.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:29.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:29.460+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:30.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:30.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:30.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:30.048+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:30.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:29.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:30.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:29.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:30.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:29.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:30.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:30.000+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:30.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:30.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:30.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:30.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:30.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:30.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:30.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:30.440+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:31.045 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:30.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:31.046 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:30.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:31.046 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:30.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:31.046 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:30.998+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:31.469 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:31.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:31.470 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:31.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:31.470 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:31.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:31.470 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:31.043+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:31.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:31.466+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:31.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:31.466+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:31.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:31.466+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:31.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:31.467+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:32.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:34:31.717+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 24132 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:34:32.487 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:32.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:32.487 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:32.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:32.488 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:32.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:32.488 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:32.043+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:32.488 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:32.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:32.488 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:32.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:32.488 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:32.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:32.488 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:32.033+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:32.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:32.484+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:32.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:32.484+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:32.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:32.484+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:32.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:32.485+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:33.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:33.071+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:33.508 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:33.071+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:33.508 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:33.072+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:33.508 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:33.072+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:33.508 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:33.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:33.508 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:33.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:33.509 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:33.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:33.509 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:33.082+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:33.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:33.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:33.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:33.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:33.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:33.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:33.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:33.505+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:34.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:34.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:34.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:34.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:34.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:34.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:34.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:34.057+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:34.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:34.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:34.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:34.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:34.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:34.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:34.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:34.125+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:34.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:34.504+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:34.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:34.504+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:34.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:34.504+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:34.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:34.504+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:35.479 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:35.097+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:35.480 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:35.097+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:35.480 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:35.097+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:35.480 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:35.098+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:35.480 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:35.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:35.480 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:35.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:35.480 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:35.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:35.481 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:35.084+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:35.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:35.477+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:35.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:35.477+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:35.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:35.477+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:35.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:35.478+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:36.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:36.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:36.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:36.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:36.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:36.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:36.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:36.061+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:36.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:36.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:36.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:36.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:36.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:36.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:36.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:36.120+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:36.734 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:34:36.721+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 24223 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:34:36.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:36.479+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:36.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:36.479+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:36.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:36.479+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:36.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:36.480+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:37.490 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:37.155+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:37.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:37.155+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:37.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:37.155+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:37.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:37.156+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:37.492 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:37.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:37.492 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:37.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:37.492 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:37.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:37.493 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:37.023+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:37.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:37.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:37.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:37.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:37.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:37.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:37.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:37.488+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:38.502 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:38.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:38.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:38.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:38.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:38.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:38.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:38.134+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:38.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:38.070+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:38.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:38.070+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:38.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:38.070+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:38.504 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:38.071+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:38.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:38.500+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:38.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:38.500+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:38.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:38.500+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:38.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:38.500+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:39.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:39.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:39.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:39.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:39.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:39.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:39.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:39.113+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:39.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:39.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:39.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:39.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:39.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:39.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:39.465 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:39.048+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:39.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:39.461+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:39.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:39.461+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:39.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:39.461+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:39.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:39.462+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:40.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:40.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:40.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:40.090+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:40.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:40.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:40.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:40.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:40.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:40.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:40.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:40.012+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:40.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:40.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:40.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:40.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:40.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:40.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:40.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:40.492+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:41.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:41.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:41.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:41.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:41.460 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:41.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:41.460 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:41.050+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:41.460 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:41.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:41.460 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:41.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:41.461 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:41.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:41.461 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:41.123+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:41.726 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:41.456+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:41.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:41.456+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:41.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:41.456+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:41.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:41.457+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:42.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:34:41.723+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 24313 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:34:42.443 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:42.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:42.443 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:42.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:42.444 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:42.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:42.444 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:42.058+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:42.444 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:42.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:42.444 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:42.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:42.444 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:42.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:42.444 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:42.157+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:42.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:42.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:42.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:42.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:42.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:42.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:42.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:42.442+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:43.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:43.160+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:43.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:43.160+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:43.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:43.160+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:43.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:43.161+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:43.428 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:43.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:43.428 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:43.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:43.429 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:43.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:43.429 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:43.056+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:43.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:43.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:43.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:43.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:43.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:43.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:43.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:43.426+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:44.423 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:44.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:44.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:44.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:44.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:44.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:44.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:44.086+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:44.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:44.137+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:44.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:44.137+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:44.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:44.137+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:44.425 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:44.138+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:44.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:44.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:44.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:44.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:44.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:44.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:44.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:44.422+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:45.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:45.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:45.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:45.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:45.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:45.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:45.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:45.072+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:45.459 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:45.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:45.459 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:45.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:45.459 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:45.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:45.459 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:45.174+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:45.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:45.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:45.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:45.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:45.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:45.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:45.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:45.456+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:46.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:46.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:46.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:46.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:46.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:46.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:46.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:46.206+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:46.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:46.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:46.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:46.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:46.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:46.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:46.411 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:46.105+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:46.729 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:46.407+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:46.729 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:46.407+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:46.729 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:46.407+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:46.729 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:46.407+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:47.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:34:46.726+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 24404 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:34:47.363 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:47.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:47.364 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:47.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:47.364 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:47.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:47.364 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:47.251+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:47.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:47.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:47.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:47.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:47.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:47.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:47.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:47.063+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:47.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:47.361+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:47.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:47.361+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:47.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:47.361+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:47.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:47.362+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:48.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:48.101+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:48.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:48.101+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:48.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:48.101+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:48.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:48.102+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:48.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:48.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:48.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:48.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:48.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:48.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:48.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:48.336+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:48.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:48.270+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:48.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:48.270+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:48.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:48.270+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:48.523 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:48.270+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:49.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:49.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:49.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:49.089+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:49.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:49.326+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:49.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:49.326+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:49.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:49.326+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:49.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:49.327+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:49.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:49.310+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:49.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:49.310+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:49.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:49.310+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:49.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:49.311+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:50.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:50.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:50.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:50.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:50.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:50.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:50.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:50.356+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:50.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:50.351+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:50.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:50.351+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:50.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:50.351+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:50.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:50.352+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:50.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:50.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:50.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:50.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:50.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:50.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:50.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:50.122+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:51.368 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:51.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:51.369 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:51.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:51.369 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:51.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:51.369 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:51.088+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:51.732 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:51.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:51.732 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:51.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:51.732 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:51.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:51.732 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:51.366+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:51.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:51.375+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:51.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:51.375+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:51.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:51.375+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:51.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:51.375+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:52.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:34:51.729+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 24493 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:34:52.395 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:52.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:52.395 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:52.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:52.395 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:52.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:52.395 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:52.336+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:52.395 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:52.120+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:52.396 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:52.120+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:52.396 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:52.120+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:52.396 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:52.121+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:52.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:52.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:52.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:52.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:52.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:52.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:52.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:52.393+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:53.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:53.086+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:53.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:53.086+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:53.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:53.086+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:53.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:53.087+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:53.630 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:53.357+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:53.630 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:53.357+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:53.630 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:53.357+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:53.630 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:53.357+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:53.630 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:53.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:53.630 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:53.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:53.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:53.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:53.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:53.355+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:54.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:54.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:54.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:54.112+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:54.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:54.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:54.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:54.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:54.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:54.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:54.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:54.346+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:54.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:54.404+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:54.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:54.404+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:54.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:54.404+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:54.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:54.405+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:55.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:55.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:55.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:55.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:55.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:55.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:55.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:55.090+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:55.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:55.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:55.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:55.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:55.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:55.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:55.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:55.397+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:55.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:55.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:55.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:55.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:55.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:55.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:55.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:55.360+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:56.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:56.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:56.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:56.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:56.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:56.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:56.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:56.052+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:56.734 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:56.407+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:56.734 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:56.407+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:56.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:56.407+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:56.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:56.408+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:56.735 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:56.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:56.735 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:56.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:56.735 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:56.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:56.735 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:56.319+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:57.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:34:56.732+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 24584 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:34:57.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:57.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:57.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:57.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:57.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:57.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:57.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:57.096+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:57.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:57.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:57.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:57.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:57.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:57.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:57.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:57.292+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:57.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:57.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:57.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:57.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:57.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:57.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:57.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:57.359+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:58.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:58.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:58.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:58.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:58.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:58.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:58.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:58.379+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:58.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:58.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:58.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:58.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:58.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:58.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:58.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:58.332+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:34:58.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:58.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:58.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:58.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:58.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:58.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:58.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:58.146+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:59.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:59.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:59.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:59.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:59.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:59.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:34:59.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:34:59.098+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:59.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:59.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:59.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:34:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:34:59.365+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:34:59.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:59.362+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:59.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:59.362+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:59.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:59.362+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:34:59.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:34:59.362+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:00.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:00.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:00.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:00.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:00.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:00.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:00.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:00.060+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:00.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:00.400+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:00.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:00.400+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:00.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:00.400+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:00.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:00.401+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:00.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:00.321+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:00.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:00.321+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:00.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:00.321+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:00.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:00.322+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:01.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:01.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:01.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:01.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:01.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:01.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:01.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:01.090+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:01.737 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:01.381+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:01.738 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:01.381+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:01.738 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:01.381+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:01.738 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:01.382+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:01.738 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:01.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:01.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:01.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:01.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:01.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:01.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:01.359+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:02.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:35:01.735+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 24674 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:35:02.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:02.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:02.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:02.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:02.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:02.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:02.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:02.342+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:02.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:02.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:02.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:02.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:02.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:02.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:02.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:02.135+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:02.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:02.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:02.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:02.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:02.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:02.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:02.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:02.368+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:03.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:03.310+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:03.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:03.310+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:03.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:03.310+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:03.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:03.311+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:03.378 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:03.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:03.379 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:03.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:03.379 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:03.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:03.379 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:03.110+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:03.630 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:03.375+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:03.630 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:03.375+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:03.630 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:03.375+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:03.630 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:03.376+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:04.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:04.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:04.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:04.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:04.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:04.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:04.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:04.264+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:04.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:04.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:04.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:04.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:04.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:04.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:04.106+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:04.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:04.407+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:04.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:04.407+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:04.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:04.407+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:04.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:04.407+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:05.457 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:05.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:05.458 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:05.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:05.458 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:05.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:05.458 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:05.265+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:05.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:05.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:05.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:05.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:05.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:05.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:05.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:05.091+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:05.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:05.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:05.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:05.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:05.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:05.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:05.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:05.456+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:06.485 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:06.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:06.485 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:06.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:06.485 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:06.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:06.485 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:06.044+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:06.485 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:06.255+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:06.486 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:06.255+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:06.486 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:06.255+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:06.486 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:06.256+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:06.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:06.482+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:06.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:06.482+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:06.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:06.482+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:06.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:06.483+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:07.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:35:06.737+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 24765 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:35:07.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:07.211+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:07.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:07.211+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:07.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:07.211+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:07.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:07.212+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:07.504 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:07.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:07.505 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:07.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:07.505 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:07.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:07.505 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:07.028+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:07.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:07.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:07.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:07.502+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:08.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:08.181+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:08.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:08.181+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:08.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:08.181+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:08.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:08.181+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:08.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:08.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:08.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:08.048+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:08.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:08.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:08.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:08.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:08.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:08.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:08.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:08.517+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:09.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:09.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:09.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:09.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:09.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:09.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:09.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:09.229+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:09.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:08.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:09.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:08.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:09.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:08.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:09.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:08.999+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:09.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:09.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:09.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:10.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:09.561+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:10.209+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:10.209+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:10.209+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:10.210+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:10.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:10.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:10.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:10.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:10.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:10.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:10.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:10.018+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:11.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:10.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:11.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:10.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:11.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:10.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:11.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:10.577+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:11.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:11.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:11.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:11.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:11.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:11.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:11.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:11.026+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:11.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:11.207+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:11.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:11.207+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:11.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:11.207+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:11.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:11.207+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:12.014 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:35:11.740+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 24854 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:35:12.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:11.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:12.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:11.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:12.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:11.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:12.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:11.622+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:12.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:12.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:12.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:12.013+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:12.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:12.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:12.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:12.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:12.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:12.167+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:12.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:12.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:12.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:12.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:12.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:12.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:12.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:12.611+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:13.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:13.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:13.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:13.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:13.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:13.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:13.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:13.159+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:13.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:12.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:13.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:12.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:13.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:12.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:13.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:12.992+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:13.998 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:13.655+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:13.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:13.655+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:13.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:13.655+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:13.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:13.656+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:13.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:13.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:13.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:13.996+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:14.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:14.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:14.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:14.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:14.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:14.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:14.177+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:14.982 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:14.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:14.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:14.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:14.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:14.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:14.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:14.661+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:14.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:14.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:14.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:14.980+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:15.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:15.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:15.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:15.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:15.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:15.189+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:16.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:15.710+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:16.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:15.710+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:16.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:15.710+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:16.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:15.710+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:16.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:16.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:16.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:16.221+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:16.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:16.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:16.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:16.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:16.017+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:17.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:35:16.742+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 24945 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:35:17.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:16.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:17.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:16.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:17.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:16.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:17.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:16.750+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:17.057+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:17.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:17.057+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:17.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:17.057+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:17.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:17.057+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:17.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:17.202+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:17.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:17.202+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:17.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:17.202+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:17.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:17.203+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:17.755+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:17.755+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:17.755+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:17.755+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:18.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:18.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:18.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:18.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:18.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:18.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:18.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:18.073+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:18.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:18.236+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:18.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:18.236+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:18.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:18.236+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:18.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:18.237+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:19.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:18.788+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:19.115 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:18.788+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:19.115 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:18.788+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:19.115 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:18.789+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:19.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:19.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:19.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:19.112+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:19.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:19.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:19.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:19.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:19.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:19.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:19.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:19.205+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:20.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:19.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:20.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:19.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:20.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:19.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:20.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:19.820+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:20.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:20.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:20.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:20.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:20.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:20.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:20.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:20.066+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:20.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:20.155+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:20.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:20.155+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:20.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:20.155+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:20.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:20.156+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:21.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:20.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:21.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:20.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:21.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:20.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:21.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:20.804+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:21.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:21.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:21.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:21.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:21.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:21.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:21.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:21.170+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:21.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:21.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:21.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:21.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:21.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:21.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:21.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:21.084+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:22.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:35:21.746+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 25035 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:35:22.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:21.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:22.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:21.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:22.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:21.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:22.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:21.797+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:22.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:22.190+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:22.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:22.190+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:22.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:22.190+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:22.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:22.190+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:22.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:22.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:22.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:22.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:22.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:22.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:22.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:22.065+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:23.061 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:22.790+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:23.061 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:22.790+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:23.061 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:22.790+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:23.062 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:22.791+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:23.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:23.155+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:23.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:23.155+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:23.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:23.155+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:23.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:23.155+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:23.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:23.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:23.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:23.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:23.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:23.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:23.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:23.059+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:24.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:23.827+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:24.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:23.827+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:24.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:23.827+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:24.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:23.828+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:24.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:24.184+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:24.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:24.184+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:24.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:24.184+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:24.525 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:24.184+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:24.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:24.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:24.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:24.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:24.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:24.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:24.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:24.101+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:25.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:25.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:25.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:25.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:25.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:25.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:25.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:25.096+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:25.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:24.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:25.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:24.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:25.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:24.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:25.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:24.847+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:25.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:25.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:25.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:25.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:25.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:25.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:25.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:25.165+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:26.068 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:25.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:26.068 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:25.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:26.069 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:25.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:26.069 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:25.802+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:26.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:26.175+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:26.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:26.175+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:26.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:26.175+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:26.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:26.175+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:26.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:26.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:26.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:26.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:26.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:26.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:26.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:26.085+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:27.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:35:26.748+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 25126 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:35:27.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:26.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:27.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:26.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:27.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:26.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:27.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:26.842+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:27.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:27.147+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:27.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:27.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:27.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:27.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:27.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:27.148+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:27.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:27.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:27.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:27.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:27.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:27.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:27.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:27.100+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:27.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:27.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:27.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:27.890+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:28.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:28.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:28.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:28.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:28.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:28.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:28.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:28.118+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:28.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:28.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:28.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:28.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:28.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:28.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:28.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:28.066+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:29.246 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:28.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:29.246 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:28.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:29.246 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:28.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:29.246 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:28.888+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:29.247 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:29.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:29.247 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:29.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:29.247 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:29.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:29.247 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:29.120+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:29.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:29.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:29.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:29.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:29.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:29.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:29.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:29.076+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:30.206 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:29.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:30.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:29.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:30.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:29.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:30.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:29.934+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:30.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:30.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:30.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:30.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:30.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:30.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:30.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:30.121+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:30.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:30.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:30.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:30.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:30.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:30.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:30.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:30.099+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:30.965+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:30.965+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:30.965+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:30.966+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:31.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:31.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:31.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:31.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:31.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:31.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:31.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:31.078+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:31.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:31.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:31.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:31.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:31.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:31.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:31.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:31.163+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:32.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:35:31.751+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 25215 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:35:32.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:31.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:32.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:31.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:32.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:31.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:32.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:31.945+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:32.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:32.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:32.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:32.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:32.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:32.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:32.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:32.196+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:32.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:32.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:32.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:32.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:32.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:32.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:32.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:32.082+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:32.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:32.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:32.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:32.934+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:33.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:33.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:33.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:33.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:33.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:33.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:33.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:33.061+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:33.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:33.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:33.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:33.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:33.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:33.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:33.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:33.149+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:34.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:34.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:34.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:34.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:34.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:34.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:34.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:34.161+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:34.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:33.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:33.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:33.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:33.917+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:34.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:34.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:34.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:34.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:34.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:34.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:34.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:34.033+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:35.208 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:34.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:35.208 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:34.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:35.208 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:34.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:35.208 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:34.931+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:35.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:35.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:35.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:35.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:35.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:35.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:35.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:35.074+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:35.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:35.205+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:35.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:35.205+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:35.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:35.205+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:35.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:35.206+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:36.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:36.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:36.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:36.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:36.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:36.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:36.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:36.084+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:36.202 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:35.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:36.203 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:35.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:36.203 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:35.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:36.203 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:35.904+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:36.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:36.199+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:36.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:36.199+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:36.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:36.199+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:36.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:36.200+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:37.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:35:36.755+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 25306 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:35:37.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:36.947+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:37.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:36.947+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:37.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:36.947+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:37.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:36.948+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:37.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:37.047+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:37.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:37.047+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:37.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:37.047+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:37.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:37.048+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:37.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:37.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:37.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:37.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:37.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:37.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:37.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:37.158+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:37.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:37.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:37.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:37.961+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:38.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:38.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:38.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:38.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:38.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:38.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:38.002+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:38.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:38.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:38.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:38.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:38.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:38.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:38.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:38.191+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:39.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:39.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:39.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:39.146+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:39.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:38.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:39.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:38.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:39.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:38.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:39.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:38.984+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:39.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:38.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:39.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:38.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:39.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:38.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:39.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:38.992+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:40.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:40.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:40.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:40.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:40.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:40.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:40.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:40.030+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:40.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:39.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:40.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:39.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:40.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:39.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:40.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:39.999+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:40.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:40.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:40.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:40.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:40.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:40.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:40.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:40.145+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:41.428 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:41.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:41.428 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:41.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:41.429 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:41.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:41.429 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:41.116+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:41.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:41.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:41.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:41.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:41.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:41.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:41.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:41.049+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:41.430 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:41.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:41.430 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:41.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:41.430 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:41.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:41.430 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:41.058+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:42.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:35:41.757+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 25396 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:35:42.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:42.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:42.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:42.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:42.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:42.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:42.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:42.007+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:42.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:42.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:42.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:42.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:42.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:42.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:42.089+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:42.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:42.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:42.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:42.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:42.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:42.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:42.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:42.046+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:43.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:43.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:43.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:43.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:43.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:43.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:43.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:43.069+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:43.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:42.997+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:43.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:42.997+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:43.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:42.997+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:43.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:42.998+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:43.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:43.093+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:43.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:43.093+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:43.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:43.093+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:43.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:43.094+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:44.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:43.954+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:44.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:43.954+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:44.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:43.954+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:44.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:43.955+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:44.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:44.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:44.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:44.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:44.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:44.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:44.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:44.078+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:44.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:44.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:44.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:44.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:44.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:44.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:44.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:44.076+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:44.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:44.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:44.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:44.978+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:45.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:45.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:45.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:45.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:45.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:45.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:45.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:45.104+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:45.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:45.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:45.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:45.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:45.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:45.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:45.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:45.036+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:46.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:46.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:46.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:46.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:46.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:46.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:46.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:46.046+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:46.075+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:46.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:46.075+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:46.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:46.075+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:46.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:46.075+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:46.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:45.967+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:46.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:45.967+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:46.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:45.967+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:46.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:45.967+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:47.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:35:46.760+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 25487 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:35:47.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:46.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:47.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:46.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:47.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:46.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:47.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:46.938+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:47.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:47.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:47.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:47.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:47.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:47.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:47.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:47.027+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:47.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:47.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:47.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:47.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:47.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:47.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:47.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:47.079+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:48.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:47.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:48.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:47.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:48.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:47.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:48.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:47.908+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:48.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:47.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:48.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:47.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:48.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:47.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:48.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:47.978+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:48.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:48.086+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:48.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:48.086+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:48.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:48.086+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:48.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:48.086+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:48.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:48.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:48.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:48.961+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:49.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:48.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:49.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:48.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:49.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:48.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:49.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:48.941+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:49.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:49.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:49.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:49.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:49.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:49.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:49.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:49.097+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:50.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:50.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:50.209 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:50.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:50.209 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:50.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:50.209 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:50.055+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:50.209 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:49.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:50.209 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:49.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:50.209 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:49.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:50.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:49.935+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:50.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:49.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:50.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:49.962+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:50.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:49.962+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:50.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:49.962+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:50.989+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:50.989+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:50.989+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:50.989+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:51.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:51.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:51.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:51.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:51.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:51.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:51.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:51.060+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:51.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:50.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:51.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:50.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:51.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:50.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:51.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:50.908+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:52.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:35:51.764+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 25576 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:35:52.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:51.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:52.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:51.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:52.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:51.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:52.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:51.986+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:52.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:51.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:52.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:51.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:52.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:51.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:52.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:51.945+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:52.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:52.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:52.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:52.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:52.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:52.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:52.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:52.095+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:53.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:52.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:53.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:52.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:53.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:52.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:53.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:52.969+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:53.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:52.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:53.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:52.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:53.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:52.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:53.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:52.965+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:53.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:53.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:53.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:53.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:53.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:53.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:53.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:53.143+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:54.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:54.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:54.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:54.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:54.143+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:54.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:53.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:54.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:53.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:54.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:53.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:54.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:53.957+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:54.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:53.918+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:54.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:53.918+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:54.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:53.918+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:54.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:53.919+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:55.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:54.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:55.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:54.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:55.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:54.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:55.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:54.927+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:54.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:54.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:55.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:54.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:55.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:54.936+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:55.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:55.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:55.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:55.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:55.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:55.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:55.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:55.103+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:56.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:55.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:56.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:55.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:56.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:55.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:56.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:55.946+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:56.148 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:55.888+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:56.148 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:55.888+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:56.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:55.888+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:56.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:55.888+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:56.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:56.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:56.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:56.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:56.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:56.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:56.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:56.145+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:57.021 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:35:56.766+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 25667 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:35:57.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:56.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:57.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:56.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:57.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:56.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:57.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:56.963+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:57.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:56.903+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:57.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:56.903+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:57.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:56.903+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:57.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:56.904+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:57.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:57.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:57.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:57.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:57.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:57.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:57.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:57.151+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:58.198 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:57.865+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:58.198 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:57.865+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:58.198 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:57.865+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:58.199 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:57.866+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:58.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:57.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:58.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:57.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:58.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:57.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:58.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:57.929+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:58.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:58.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:58.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:58.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:58.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:58.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:58.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:58.196+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:59.206 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:58.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:59.207 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:58.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:59.207 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:58.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:35:59.207 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:58.849+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:35:59.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:58.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:59.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:58.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:59.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:58.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:35:59.208 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:58.942+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:35:59.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:59.205+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:59.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:59.205+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:59.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:59.205+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:35:59.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:35:59.205+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:00.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:59.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:00.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:59.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:00.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:59.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:00.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:35:59.887+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:00.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:59.919+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:00.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:59.919+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:00.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:59.919+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:00.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:35:59.920+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:00.212+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:00.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:00.212+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:00.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:00.212+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:00.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:00.213+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:01.233 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:00.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:01.233 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:00.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:01.233 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:00.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:01.234 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:00.940+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:01.234 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:00.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:01.234 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:00.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:01.234 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:00.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:01.234 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:00.843+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:01.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:01.232+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:01.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:01.232+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:01.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:01.232+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:01.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:01.232+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:02.243 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:36:01.768+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 25757 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:36:02.243 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:01.828+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:02.243 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:01.828+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:02.243 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:01.828+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:02.244 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:01.828+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:02.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:01.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:02.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:01.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:02.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:01.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:02.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:01.938+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:02.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:02.240+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:02.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:02.240+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:02.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:02.240+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:02.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:02.240+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:03.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:02.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:03.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:02.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:03.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:02.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:03.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:02.893+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:02.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:02.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:03.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:02.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:03.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:02.833+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:03.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:03.276+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:03.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:03.276+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:03.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:03.276+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:03.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:03.277+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:04.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:03.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:04.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:03.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:04.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:03.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:04.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:03.903+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:04.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:03.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:04.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:03.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:04.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:03.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:04.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:03.848+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:04.569 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:04.286+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:04.569 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:04.286+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:04.569 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:04.286+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:04.569 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:04.286+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:04.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:04.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:04.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:04.909+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:04.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:04.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:04.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:05.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:04.825+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:05.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:05.335+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:05.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:05.335+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:05.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:05.335+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:05.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:05.335+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:05.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:06.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:05.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:06.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:05.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:06.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:05.876+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:05.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:05.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:05.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:06.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:05.863+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:06.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:06.301+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:06.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:06.301+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:06.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:06.301+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:06.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:06.302+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:07.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:36:06.772+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 25848 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:36:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:06.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:06.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:06.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:06.883+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:07.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:06.827+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:07.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:06.827+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:07.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:06.827+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:07.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:06.828+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:07.288+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:07.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:07.288+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:07.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:07.288+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:07.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:07.289+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:07.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:07.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:07.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:07.926+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:08.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:07.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:08.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:07.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:08.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:07.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:08.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:07.861+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:08.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:08.275+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:08.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:08.275+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:08.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:08.275+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:08.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:08.275+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:08.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:08.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:08.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:08.896+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:09.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:08.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:09.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:08.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:09.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:08.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:09.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:08.960+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:09.614 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:09.318+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:09.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:09.318+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:09.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:09.318+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:09.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:09.319+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:10.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:09.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:10.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:09.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:10.211 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:09.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:10.211 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:09.936+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:10.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:09.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:10.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:09.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:10.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:09.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:10.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:09.925+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:10.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:10.283+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:10.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:10.283+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:10.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:10.283+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:10.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:10.284+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:10.939+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:11.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:10.939+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:11.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:10.939+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:11.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:10.939+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:11.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:10.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:11.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:10.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:11.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:10.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:11.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:10.903+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:11.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:11.329+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:11.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:11.329+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:11.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:11.329+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:11.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:11.330+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:12.148 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:36:11.775+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 25937 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:36:12.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:11.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:12.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:11.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:12.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:11.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:12.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:11.875+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:12.150 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:11.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:12.150 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:11.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:12.150 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:11.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:12.150 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:11.910+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:12.480 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:12.373+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:12.480 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:12.373+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:12.481 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:12.373+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:12.481 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:12.373+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:12.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:12.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:12.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:12.883+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:13.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:12.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:13.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:12.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:13.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:12.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:13.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:12.862+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:13.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:13.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:13.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:13.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:13.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:13.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:13.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:13.405+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:13.931+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:14.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:13.931+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:14.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:13.931+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:14.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:13.931+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:13.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:13.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:13.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:14.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:13.872+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:14.662 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:14.368+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:14.662 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:14.368+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:14.662 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:14.368+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:14.662 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:14.368+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:14.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:14.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:14.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:14.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:14.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:14.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:14.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:14.879+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:15.220 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:14.966+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:15.220 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:14.966+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:15.220 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:14.966+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:15.220 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:14.967+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:15.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:15.340+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:15.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:15.340+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:15.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:15.340+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:15.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:15.340+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:15.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:15.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:15.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:15.895+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:16.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:16.004+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:16.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:16.004+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:16.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:16.004+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:16.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:16.005+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:16.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:16.303+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:16.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:16.303+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:16.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:16.303+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:16.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:16.303+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:17.046 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:36:16.778+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 26028 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:36:17.046 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:16.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:17.046 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:16.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:17.046 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:16.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:17.047 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:16.912+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:17.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:17.044+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:17.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:17.044+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:17.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:17.044+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:17.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:17.045+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:17.333+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:17.333+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:17.333+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:17.334+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:18.094+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:18.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:18.094+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:18.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:18.094+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:18.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:18.095+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:17.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:17.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:17.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:17.926+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:18.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:18.328+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:18.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:18.328+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:18.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:18.328+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:18.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:18.329+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:19.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:19.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:19.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:19.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:19.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:19.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:19.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:19.137+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:19.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:18.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:19.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:18.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:19.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:18.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:19.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:18.970+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:19.712 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:19.368+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:19.712 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:19.368+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:19.712 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:19.368+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:19.712 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:19.369+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:20.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:19.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:20.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:19.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:20.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:19.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:20.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:19.935+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:20.407 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:20.142+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:20.407 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:20.142+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:20.407 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:20.142+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:20.407 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:20.143+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:20.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:20.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:20.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:20.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:20.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:20.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:20.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:20.405+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:20.899+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:20.899+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:20.899+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:20.899+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:21.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:21.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:21.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:21.134+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:21.396+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:21.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:21.396+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:21.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:21.396+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:21.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:21.396+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:22.175 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:36:21.782+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 26118 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:36:22.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:21.899+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:22.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:21.899+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:22.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:21.899+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:22.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:21.900+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:22.430 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:22.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:22.430 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:22.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:22.430 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:22.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:22.430 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:22.174+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:22.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:22.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:22.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:22.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:22.428+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:22.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:22.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:22.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:22.939+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:23.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:23.152+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:23.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:23.152+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:23.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:23.152+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:23.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:23.153+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:23.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:23.421+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:23.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:23.421+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:23.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:23.421+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:23.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:23.422+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:24.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:24.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:24.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:24.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:24.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:24.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:24.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:24.107+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:24.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:23.974+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:24.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:23.974+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:24.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:23.974+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:24.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:23.975+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:24.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:24.392+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:24.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:24.392+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:24.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:24.392+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:24.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:24.392+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:25.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:24.929+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:24.929+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:24.929+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:24.930+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:25.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:25.118+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:25.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:25.118+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:25.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:25.118+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:25.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:25.118+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:25.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:25.374+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:25.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:25.374+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:25.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:25.374+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:25.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:25.375+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:26.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:26.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:26.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:26.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:26.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:26.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:26.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:26.075+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:26.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:25.890+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:26.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:25.890+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:26.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:25.890+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:26.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:25.891+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:26.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:26.419+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:26.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:26.419+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:26.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:26.419+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:26.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:26.419+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:27.114 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:36:26.785+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 26209 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:36:27.115 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:26.855+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:27.115 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:26.855+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:27.115 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:26.855+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:27.115 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:26.856+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:27.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:27.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:27.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:27.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:27.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:27.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:27.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:27.113+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:27.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:27.447+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:27.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:27.447+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:27.447+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:27.448+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:28.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:28.103+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:28.103+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:28.103+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:28.104+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:28.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:27.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:28.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:27.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:28.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:27.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:28.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:27.871+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:28.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:28.444+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:28.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:28.444+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:28.444+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:28.445+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:28.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:28.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:28.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:28.909+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:29.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:29.065+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:29.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:29.065+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:29.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:29.065+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:29.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:29.066+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:29.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:29.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:29.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:29.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:29.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:29.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:29.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:29.462+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:30.081 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:29.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:30.082 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:29.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:30.082 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:29.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:30.082 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:29.870+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:30.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:30.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:30.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:30.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:30.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:30.080+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:30.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:30.463+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:30.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:30.463+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:30.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:30.463+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:30.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:30.464+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:31.071+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:31.071+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:31.071+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:31.073+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:31.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:30.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:31.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:30.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:31.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:30.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:31.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:30.833+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:31.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:31.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:31.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:31.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:31.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:31.472+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:32.078 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:36:31.788+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 26298 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:36:32.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:31.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:32.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:31.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:32.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:31.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:32.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:31.790+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:32.459 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:32.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:32.459 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:32.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:32.459 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:32.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:32.459 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:32.076+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:32.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:32.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:32.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:32.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:32.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:32.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:32.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:32.457+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:33.088 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:32.775+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:33.088 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:32.775+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:33.088 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:32.775+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:33.089 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:32.775+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:33.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:33.086+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:33.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:33.086+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:33.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:33.086+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:33.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:33.087+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:33.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:33.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:33.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:33.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:33.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:33.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:33.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:33.427+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:34.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:33.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:34.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:33.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:34.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:33.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:34.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:33.769+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:34.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:34.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:34.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:34.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:34.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:34.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:34.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:34.078+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:34.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:34.448+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:34.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:34.448+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:34.448+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:34.448+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:35.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:34.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:35.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:34.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:35.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:34.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:35.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:34.796+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:35.476 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:35.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:35.476 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:35.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:35.476 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:35.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:35.476 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:35.126+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:35.766 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:35.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:35.766 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:35.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:35.766 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:35.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:35.767 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:35.474+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:36.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:35.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:36.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:35.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:35.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:35.765+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:36.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:36.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:36.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:36.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:36.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:36.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:36.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:36.078+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:36.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:36.517+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:36.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:36.517+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:36.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:36.517+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:36.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:36.517+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:36.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:36.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:36.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:36.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:36.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:36.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:36.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:36.721+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:37.108 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:36:36.791+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 26389 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:36:37.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:37.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:37.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:37.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:37.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:37.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:37.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:37.107+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:37.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:38.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:37.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:38.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:37.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:38.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:37.727+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:38.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:37.561+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:38.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:37.561+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:38.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:37.561+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:38.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:37.562+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:38.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:38.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:38.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:38.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:38.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:38.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:38.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:38.126+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:38.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:39.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:38.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:39.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:38.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:39.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:38.688+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:39.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:38.545+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:39.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:38.545+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:39.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:38.545+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:39.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:38.546+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:39.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:39.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:39.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:39.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:39.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:39.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:39.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:39.155+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:39.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:39.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:39.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:39.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:39.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:39.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:39.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:39.644+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:39.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:39.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:39.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:39.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:39.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:39.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:39.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:39.507+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:40.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:40.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:40.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:40.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:40.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:40.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:40.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:40.159+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:40.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:40.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:40.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:40.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:40.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:40.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:40.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:40.626+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:40.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:40.516+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:40.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:40.516+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:40.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:40.516+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:40.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:40.516+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:41.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:41.124+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:41.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:41.124+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:41.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:41.124+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:41.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:41.124+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:41.796 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:41.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:41.797 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:41.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:41.797 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:41.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:41.797 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:41.636+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:41.797 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:41.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:41.797 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:41.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:41.798 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:41.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:41.798 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:41.527+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:42.127 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:36:41.794+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 26479 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:36:42.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:42.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:42.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:42.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:42.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:42.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:42.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:42.126+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:42.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:42.651+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:42.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:42.651+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:42.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:42.651+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:42.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:42.652+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:42.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:42.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:42.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:42.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:42.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:42.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:42.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:42.529+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:43.494 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:43.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:43.494 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:43.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:43.495 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:43.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:43.495 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:43.157+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:43.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:43.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:43.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:43.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:43.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:43.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:43.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:43.700+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:43.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:43.492+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:43.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:43.492+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:43.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:43.492+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:43.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:43.492+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:44.455 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:44.175+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:44.455 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:44.175+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:44.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:44.175+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:44.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:44.175+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:44.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:44.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:44.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:44.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:44.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:44.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:44.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:44.702+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:44.453+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:44.453+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:44.453+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:44.453+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:45.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:45.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:45.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:45.155+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:45.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:45.437+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:45.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:45.437+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:45.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:45.437+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:45.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:45.438+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:46.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:45.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:46.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:45.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:46.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:45.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:46.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:45.695+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:46.487 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:46.171+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:46.487 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:46.171+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:46.487 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:46.171+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:46.487 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:46.171+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:46.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:46.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:46.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:46.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:46.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:46.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:46.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:46.662+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:46.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:46.485+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:46.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:46.485+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:46.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:46.485+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:46.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:46.485+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:47.155 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:36:46.797+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 26570 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:36:47.479 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:47.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:47.479 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:47.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:47.479 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:47.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:47.479 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:47.153+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:47.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:47.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:47.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:47.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:47.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:47.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:47.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:47.476+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:47.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:47.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:47.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:47.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:47.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:47.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:47.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:47.626+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:48.492 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:48.122+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:48.492 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:48.122+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:48.493 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:48.122+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:48.493 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:48.123+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:48.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:48.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:48.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:48.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:48.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:48.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:48.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:48.490+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:48.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:48.596+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:48.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:48.596+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:48.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:48.596+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:48.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:48.597+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:49.512 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:49.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:49.516 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:49.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:49.516 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:49.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:49.517 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:49.122+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:49.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:49.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:49.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:49.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:49.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:49.510+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:49.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:49.628+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:49.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:49.628+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:49.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:49.628+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:49.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:49.629+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:50.144+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:50.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:50.144+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:50.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:50.144+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:50.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:50.145+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:50.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:50.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:50.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:50.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:50.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:50.506+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:50.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:50.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:50.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:50.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:50.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:50.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:50.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:50.627+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:51.498 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:51.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:51.498 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:51.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:51.498 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:51.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:51.498 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:51.185+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:51.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:51.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:51.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:51.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:51.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:51.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:51.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:51.677+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:51.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:51.496+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:51.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:51.496+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:51.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:51.496+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:51.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:51.496+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:52.084 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:36:51.800+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 26659 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:36:52.475 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:52.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:52.475 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:52.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:52.475 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:52.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:52.475 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:52.211+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:52.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:52.707+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:52.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:52.707+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:52.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:52.707+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:52.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:52.708+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:52.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:52.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:52.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:52.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:52.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:52.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:52.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:52.473+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:53.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:53.249+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:53.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:53.249+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:53.503 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:53.249+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:53.503 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:53.250+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:53.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:53.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:53.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:53.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:53.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:53.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:53.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:53.687+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:53.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:53.500+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:53.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:53.500+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:53.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:53.500+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:53.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:53.500+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:54.551 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:54.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:54.552 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:54.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:54.552 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:54.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:54.552 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:54.298+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:55.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:54.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:55.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:54.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:55.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:54.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:55.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:54.658+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:55.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:54.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:55.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:54.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:55.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:54.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:55.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:54.550+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:55.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:55.298+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:55.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:55.298+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:55.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:55.298+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:55.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:55.299+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:55.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:55.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:55.638 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:55.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:55.638 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:55.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:55.638 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:55.546+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:56.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:55.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:56.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:55.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:56.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:55.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:56.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:55.636+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:56.577 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:56.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:56.577 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:56.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:56.577 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:56.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:56.577 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:56.297+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:56.828 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:36:56.804+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 26749 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:36:56.828 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:56.656+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:56.828 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:56.656+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:56.828 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:56.656+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:56.828 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:56.656+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:56.829 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:56.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:56.829 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:56.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:56.829 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:56.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:56.829 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:56.575+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:57.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:57.251+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:57.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:57.251+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:57.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:57.251+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:57.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:57.252+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:58.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:57.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:58.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:57.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:58.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:57.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:58.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:57.528+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:58.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:57.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:58.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:57.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:58.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:57.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:58.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:57.702+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:58.549 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:58.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:58.549 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:58.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:58.549 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:58.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:58.549 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:58.282+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:59.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:58.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:59.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:58.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:59.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:58.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:36:59.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:58.752+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:36:59.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:58.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:59.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:58.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:59.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:58.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:59.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:58.547+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:59.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:59.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:59.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:59.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:36:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:36:59.283+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:36:59.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:59.504+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:59.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:59.504+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:59.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:59.504+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:36:59.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:36:59.504+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:00.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:59.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:00.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:59.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:00.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:59.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:00.085 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:36:59.778+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:00.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:00.247+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:00.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:00.247+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:00.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:00.247+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:00.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:00.248+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:00.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:00.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:00.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:00.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:00.503+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:00.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:00.503+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:00.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:00.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:00.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:00.786+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:01.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:01.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:01.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:01.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:01.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:01.463+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:01.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:01.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:01.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:01.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:01.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:01.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:01.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:01.282+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:02.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:37:01.807+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 26840 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:37:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:01.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:01.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:01.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:01.800+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:02.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:02.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:02.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:02.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:02.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:02.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:02.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:02.420+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:02.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:02.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:02.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:02.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:02.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:02.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:02.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:02.290+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:02.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:02.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:02.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:02.794+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:03.638 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:03.463+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:03.638 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:03.463+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:03.638 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:03.463+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:03.638 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:03.463+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:03.638 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:03.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:03.638 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:03.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:03.638 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:03.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:03.639 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:03.339+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:03.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:03.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:04.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:03.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:04.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:03.797+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:04.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:04.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:04.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:04.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:04.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:04.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:04.290+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:04.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:04.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:04.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:04.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:04.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:04.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:04.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:04.459+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:05.129 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:04.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:05.130 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:04.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:05.130 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:04.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:05.130 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:04.832+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:05.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:05.334+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:05.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:05.334+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:05.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:05.334+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:05.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:05.334+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:05.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:05.507+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:05.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:05.507+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:05.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:05.507+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:05.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:05.507+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:05.879+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:05.879+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:05.879+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:05.880+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:06.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:06.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:06.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:06.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:06.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:06.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:06.354+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:06.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:06.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:06.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:06.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:06.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:06.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:06.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:06.486+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:07.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:37:06.810+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 26930 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:37:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:06.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:06.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:06.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:06.888+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:07.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:07.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:07.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:07.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:07.339+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:07.339+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:07.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:07.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:07.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:07.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:07.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:07.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:07.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:07.457+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:07.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:07.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:07.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:07.903+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:08.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:08.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:08.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:08.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:08.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:08.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:08.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:08.324+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:08.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:08.438+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:08.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:08.438+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:08.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:08.438+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:08.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:08.438+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:08.898+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:08.898+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:08.898+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:09.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:08.899+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:09.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:09.477+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:09.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:09.477+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:09.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:09.477+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:09.477+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:09.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:09.286+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:09.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:09.286+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:09.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:09.286+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:09.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:09.287+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:10.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:09.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:10.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:09.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:10.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:09.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:10.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:09.932+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:10.516 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:10.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:10.516 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:10.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:10.516 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:10.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:10.516 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:10.330+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:10.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:10.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:10.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:10.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:10.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:10.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:10.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:10.520+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:10.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:10.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:10.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:11.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:10.935+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:11.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:11.348+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:11.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:11.348+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:11.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:11.348+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:11.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:11.349+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:11.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:11.516+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:11.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:11.516+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:11.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:11.516+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:11.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:11.516+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:12.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:37:11.813+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 27020 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:37:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:11.944+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:11.944+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:11.944+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:11.945+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:12.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:12.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:12.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:12.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:12.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:12.368+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:12.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:12.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:12.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:12.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:12.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:12.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:12.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:12.487+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:12.940+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:12.940+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:13.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:12.940+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:13.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:12.940+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:13.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:13.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:13.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:13.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:13.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:13.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:13.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:13.483+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:13.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:13.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:13.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:13.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:13.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:13.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:13.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:13.325+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:13.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:13.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:13.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:13.946+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:14.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:14.500+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:14.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:14.500+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:14.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:14.500+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:14.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:14.500+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:14.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:14.277+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:14.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:14.277+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:14.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:14.277+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:14.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:14.278+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:15.226 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:14.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:15.226 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:14.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:15.226 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:14.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:15.226 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:14.961+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:15.493 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:15.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:15.493 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:15.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:15.493 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:15.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:15.493 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:15.263+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:15.491+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:15.491+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:15.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:15.491+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:15.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:15.491+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:16.226 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:15.913+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:16.226 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:15.913+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:16.227 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:15.913+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:16.227 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:15.913+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:16.477 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:16.224+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:16.477 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:16.224+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:16.477 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:16.224+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:16.477 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:16.225+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:16.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:16.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:16.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:16.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:16.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:16.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:16.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:16.476+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:17.266 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:37:16.816+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 27110 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:37:17.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:16.879+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:17.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:16.879+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:17.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:16.879+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:17.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:16.880+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:17.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:17.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:17.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:17.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:17.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:17.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:17.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:17.265+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:17.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:17.448+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:17.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:17.448+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:17.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:17.448+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:17.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:17.449+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:17.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:17.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:17.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:17.901+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:18.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:18.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:18.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:18.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:18.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:18.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:18.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:18.313+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:18.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:18.483+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:18.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:18.483+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:18.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:18.483+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:18.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:18.483+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:19.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:18.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:19.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:18.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:19.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:18.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:19.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:18.902+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:19.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:19.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:19.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:19.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:19.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:19.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:19.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:19.265+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:19.465+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:19.465+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:19.465+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:19.465+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:20.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:19.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:20.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:19.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:20.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:19.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:20.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:19.908+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:20.484 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:20.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:20.484 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:20.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:20.484 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:20.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:20.484 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:20.265+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:20.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:20.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:20.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:20.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:20.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:20.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:20.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:20.482+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:21.233 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:20.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:21.233 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:20.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:21.233 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:20.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:21.234 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:20.878+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:21.488 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:21.231+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:21.489 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:21.231+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:21.489 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:21.231+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:21.489 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:21.232+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:21.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:21.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:21.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:21.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:21.487+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:22.224 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:37:21.819+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 27201 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:37:22.224 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:21.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:22.225 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:21.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:22.225 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:21.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:22.225 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:21.870+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:22.477 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:22.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:22.478 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:22.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:22.478 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:22.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:22.478 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:22.223+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:22.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:22.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:22.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:22.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:22.476+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:23.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:22.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:22.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:23.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:22.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:23.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:22.825+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:23.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:23.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:23.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:23.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:23.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:23.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:23.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:23.508+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:23.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:23.267+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:23.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:23.267+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:23.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:23.267+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:23.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:23.268+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:23.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:23.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:23.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:24.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:23.859+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:24.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:24.284+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:24.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:24.284+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:24.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:24.284+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:24.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:24.285+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:24.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:24.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:24.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:24.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:24.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:24.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:24.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:24.511+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:24.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:24.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:24.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:24.902+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:25.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:25.274+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:25.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:25.274+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:25.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:25.274+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:25.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:25.275+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:25.863 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:25.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:25.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:25.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:25.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:25.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:25.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:25.528+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:25.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:25.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:26.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:25.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:26.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:25.862+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:26.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:26.275+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:26.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:26.275+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:26.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:26.275+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:26.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:26.276+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:26.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:26.507+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:26.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:26.507+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:26.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:26.507+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:26.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:26.507+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:27.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:37:26.822+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 27291 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:37:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:26.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:26.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:26.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:26.885+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:27.543 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:27.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:27.544 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:27.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:27.544 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:27.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:27.544 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:27.272+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:27.879 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:27.541+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:27.879 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:27.541+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:27.880 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:27.541+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:27.880 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:27.542+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:27.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:28.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:27.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:28.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:27.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:28.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:27.878+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:28.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:28.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:28.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:28.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:28.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:28.548+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:28.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:28.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:28.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:28.313+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:28.892+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:28.892+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:28.892+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:28.893+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:29.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:29.333+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:29.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:29.333+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:29.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:29.333+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:29.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:29.334+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:29.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:29.596+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:29.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:29.596+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:29.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:29.596+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:29.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:29.596+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:29.857+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:29.857+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:29.857+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:29.857+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:30.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:30.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:30.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:30.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:30.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:30.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:30.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:30.356+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:31.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:30.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:31.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:30.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:31.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:30.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:31.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:30.811+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:31.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:30.567+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:31.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:30.567+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:31.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:30.567+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:31.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:30.568+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:31.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:31.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:31.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:31.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:31.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:31.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:31.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:31.355+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:31.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:31.518+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:31.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:31.518+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:31.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:31.518+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:31.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:31.518+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:32.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:37:31.826+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 27381 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:37:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:31.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:31.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:31.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:31.787+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:32.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:32.500+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:32.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:32.500+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:32.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:32.500+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:32.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:32.500+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:32.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:32.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:32.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:32.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:32.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:32.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:32.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:32.338+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:32.802+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:32.802+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:33.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:32.802+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:33.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:32.803+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:33.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:33.360+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:33.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:33.360+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:33.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:33.360+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:33.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:33.361+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:33.640 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:33.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:33.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:33.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:33.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:33.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:33.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:33.471+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:34.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:33.826+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:34.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:33.826+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:34.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:33.826+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:34.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:33.827+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:34.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:34.397+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:34.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:34.398+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:34.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:34.398+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:34.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:34.398+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:34.485+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:34.485+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:34.485+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:34.485+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:34.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:34.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:34.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:34.860+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:35.383+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:35.383+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:35.383+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:35.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:35.384+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:35.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:35.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:35.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:35.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:35.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:35.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:35.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:35.495+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:35.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:35.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:35.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:35.836+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:36.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:36.390+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:36.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:36.390+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:36.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:36.390+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:36.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:36.391+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:36.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:36.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:36.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:36.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:36.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:36.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:36.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:36.507+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:37.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:37:36.830+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 27471 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:37:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:36.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:36.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:36.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:36.848+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:37.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:37.382+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:37.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:37.382+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:37.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:37.382+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:37.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:37.383+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:37.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:37.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:37.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:37.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:37.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:37.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:37.773 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:37.532+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:37.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:37.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:37.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:37.884+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:38.387+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:38.387+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:38.387+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:38.388+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:38.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:38.552+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:38.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:38.552+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:38.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:38.552+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:38.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:38.553+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:38.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:38.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:38.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:38.885+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:39.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:39.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:39.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:39.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:39.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:39.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:39.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:39.341+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:39.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:39.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:39.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:39.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:39.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:39.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:39.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:39.512+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:40.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:39.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:40.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:39.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:40.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:39.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:40.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:39.903+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:40.495 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:40.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:40.496 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:40.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:40.496 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:40.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:40.496 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:40.350+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:40.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:40.493+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:40.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:40.493+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:40.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:40.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:40.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:40.494+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:41.236 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:40.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:41.236 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:40.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:41.236 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:40.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:41.236 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:40.910+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:41.493 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:41.397+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:41.493 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:41.397+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:41.493 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:41.397+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:41.494 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:41.398+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:41.491+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:41.491+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:41.491+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:41.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:41.492+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:42.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:37:41.833+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 27562 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:37:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:41.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:41.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:41.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:41.934+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:42.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:42.380+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:42.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:42.380+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:42.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:42.380+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:42.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:42.381+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:42.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:42.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:42.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:42.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:42.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:42.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:42.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:42.463+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:42.888+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:42.888+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:42.888+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:42.889+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:43.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:43.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:43.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:43.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:43.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:43.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:43.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:43.392+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:43.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:43.470+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:43.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:43.470+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:43.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:43.470+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:43.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:43.471+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:43.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:43.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:43.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:44.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:43.873+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:44.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:44.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:44.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:44.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:44.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:44.392+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:44.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:44.393+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:44.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:44.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:44.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:44.454+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:44.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:44.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:44.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:45.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:44.897+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:45.398+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:45.398+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:45.398+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:45.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:45.398+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:45.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:45.409+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:45.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:45.409+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:45.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:45.409+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:45.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:45.410+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:45.892+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:45.892+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:45.892+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:45.893+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:46.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:46.362+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:46.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:46.362+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:46.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:46.362+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:46.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:46.362+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:46.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:46.389+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:46.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:46.389+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:46.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:46.389+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:46.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:46.390+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:47.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:37:46.836+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 27652 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:37:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:46.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:46.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:47.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:46.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:47.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:46.881+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:47.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:47.322+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:47.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:47.322+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:47.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:47.322+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:47.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:47.323+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:47.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:47.397+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:47.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:47.397+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:47.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:47.397+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:47.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:47.397+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:47.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:47.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:47.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:47.855+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:48.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:48.294+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:48.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:48.294+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:48.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:48.294+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:48.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:48.294+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:48.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:48.419+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:48.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:48.419+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:48.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:48.419+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:48.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:48.420+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:49.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:48.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:49.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:48.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:49.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:48.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:48.835+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:49.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:49.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:49.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:49.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:49.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:49.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:49.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:49.290+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:49.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:49.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:49.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:49.436+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:50.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:49.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:50.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:49.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:50.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:49.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:50.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:49.824+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:50.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:50.263+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:50.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:50.263+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:50.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:50.263+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:50.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:50.264+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:50.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:50.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:50.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:50.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:50.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:50.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:50.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:50.427+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:51.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:50.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:51.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:50.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:51.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:50.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:51.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:50.804+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:51.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:51.308+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:51.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:51.308+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:51.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:51.308+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:51.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:51.309+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:51.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:51.444+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:51.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:51.444+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:51.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:51.444+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:51.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:51.445+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:52.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:37:51.839+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 27742 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:37:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:51.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:51.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:51.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:51.805+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:52.300+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:52.300+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:52.300+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:52.301+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:52.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:52.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:52.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:52.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:52.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:52.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:52.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:52.442+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:53.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:52.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:53.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:52.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:53.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:52.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:53.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:52.769+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:53.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:53.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:53.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:53.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:53.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:53.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:53.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:53.274+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:53.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:53.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:53.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:53.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:53.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:53.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:53.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:53.437+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:53.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:53.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:53.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:53.741+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:54.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:54.258+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:54.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:54.258+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:54.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:54.258+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:54.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:54.259+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:54.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:54.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:54.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:54.427+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:55.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:54.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:55.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:54.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:55.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:54.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:55.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:54.707+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:55.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:55.230+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:55.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:55.230+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:55.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:55.230+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:55.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:55.230+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:55.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:55.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:55.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:55.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:55.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:55.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:55.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:55.432+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:56.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:55.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:56.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:55.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:56.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:55.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:56.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:55.693+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:56.444 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:56.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:56.445 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:56.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:56.445 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:56.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:56.445 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:56.207+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:56.698 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:56.443+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:56.698 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:56.443+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:56.698 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:56.443+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:56.699 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:56.443+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:56.967 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:37:56.842+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 27832 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:37:56.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:56.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:56.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:56.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:56.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:56.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:56.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:56.697+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:57.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:57.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:57.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:57.162+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:57.688 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:57.410+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:57.688 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:57.410+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:57.688 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:57.410+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:57.688 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:57.410+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:57.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:57.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:57.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:57.687+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:58.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:58.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:58.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:58.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:58.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:58.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:58.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:58.380+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:58.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:58.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:58.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:58.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:58.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:58.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:58.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:58.157+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:59.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:58.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:59.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:58.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:59.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:58.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:37:59.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:58.655+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:37:59.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:59.400+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:59.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:59.400+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:59.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:59.400+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:37:59.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:37:59.400+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:37:59.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:59.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:59.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:59.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:59.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:59.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:37:59.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:37:59.171+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:59.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:59.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:59.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:37:59.671+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:00.442 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:00.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:00.443 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:00.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:00.443 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:00.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:00.443 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:00.173+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:00.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:00.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:00.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:00.441+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:00.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:00.655+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:00.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:00.655+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:00.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:00.655+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:00.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:00.655+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:01.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:01.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:01.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:01.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:01.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:01.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:01.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:01.412+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:01.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:01.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:01.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:01.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:01.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:01.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:01.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:01.193+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:01.970 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:38:01.846+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 27923 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:38:01.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:01.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:01.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:01.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:01.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:01.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:01.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:01.707+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:02.144+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:02.144+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:02.144+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:02.144+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:02.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:02.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:02.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:02.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:02.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:02.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:02.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:02.377+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:03.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:02.737+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:03.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:02.737+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:03.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:02.737+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:03.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:02.738+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:03.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:03.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:03.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:03.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:03.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:03.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:03.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:03.159+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:03.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:03.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:03.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:03.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:03.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:03.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:03.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:03.372+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:03.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:03.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:03.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:03.753+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:04.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:04.405+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:04.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:04.405+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:04.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:04.405+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:04.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:04.405+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:04.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:04.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:04.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:04.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:04.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:04.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:04.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:04.181+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:05.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:04.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:05.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:04.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:05.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:04.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:05.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:04.767+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:05.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:05.164+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:05.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:05.164+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:05.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:05.164+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:05.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:05.165+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:05.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:05.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:05.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:05.424+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:05.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:05.424+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:05.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:05.424+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:05.774+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:05.774+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:05.774+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:05.775+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:06.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:06.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:06.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:06.212+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:06.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:06.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:06.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:06.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:06.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:06.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:06.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:06.415+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:07.182 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:38:06.849+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 28013 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:38:07.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:06.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:07.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:06.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:07.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:06.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:07.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:06.821+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:07.390+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:07.390+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:07.390+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:07.391+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:07.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:07.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:07.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:07.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:07.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:07.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:07.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:07.181+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:08.133 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:07.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:08.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:07.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:08.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:07.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:08.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:07.795+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:08.132+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:08.132+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:08.132+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:08.132+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:08.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:08.344+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:08.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:08.344+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:08.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:08.344+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:08.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:08.344+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:09.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:08.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:09.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:08.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:09.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:08.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:09.129 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:08.795+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:09.126+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:09.126+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:09.126+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:09.127+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:09.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:09.294+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:09.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:09.295+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:09.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:09.295+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:09.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:09.295+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:10.093 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:09.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:10.094 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:09.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:10.094 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:09.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:10.094 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:09.778+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:10.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:10.092+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:10.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:10.092+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:10.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:10.092+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:10.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:10.093+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:10.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:10.275+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:10.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:10.275+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:10.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:10.275+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:10.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:10.276+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:11.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:10.745+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:11.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:10.745+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:11.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:10.745+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:11.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:10.746+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:11.275+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:11.275+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:11.275+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:11.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:11.275+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:11.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:11.063+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:11.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:11.063+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:11.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:11.063+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:11.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:11.064+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:12.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:38:11.852+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 28103 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:38:12.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:11.735+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:11.735+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:11.735+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:11.736+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:12.292 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:12.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:12.292 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:12.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:12.292 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:12.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:12.292 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:12.034+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:12.734 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:12.290+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:12.734 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:12.290+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:12.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:12.290+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:12.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:12.291+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:13.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:12.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:13.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:12.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:13.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:12.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:13.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:12.733+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:13.310 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:13.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:13.310 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:13.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:13.310 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:13.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:13.310 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:13.041+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:13.581 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:13.309+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:13.581 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:13.309+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:13.581 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:13.309+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:13.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:13.309+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:14.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:13.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:14.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:13.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:14.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:13.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:14.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:13.693+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:14.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:14.026+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:14.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:14.026+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:14.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:14.026+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:14.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:14.027+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:14.722 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:14.333+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:14.722 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:14.333+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:14.722 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:14.333+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:14.722 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:14.334+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:14.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:14.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:15.000 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:14.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:15.000 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:14.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:15.000 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:14.721+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:14.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:14.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:14.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:14.999+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:15.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:15.370+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:15.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:15.370+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:15.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:15.370+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:15.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:15.370+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:15.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:15.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:15.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:15.992+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:16.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:15.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:16.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:15.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:16.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:15.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:16.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:15.752+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:16.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:16.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:16.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:16.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:16.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:16.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:16.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:16.412+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:16.980 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:38:16.855+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 28193 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:38:16.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:16.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:16.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:16.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:16.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:16.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:16.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:16.725+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:16.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:16.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:16.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:16.993+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:17.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:17.442+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:17.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:17.442+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:17.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:17.442+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:17.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:17.442+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:17.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:17.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:17.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:17.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:17.709+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:17.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:17.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:17.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:17.964+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:18.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:18.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:18.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:18.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:18.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:18.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:18.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:18.429+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:18.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:18.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:18.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:18.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:18.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:18.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:18.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:18.674+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:19.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:18.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:19.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:18.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:19.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:18.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:19.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:18.934+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:19.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:19.397+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:19.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:19.397+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:19.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:19.397+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:19.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:19.397+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:19.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:19.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:19.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:19.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:19.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:19.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:19.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:19.641+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:20.219 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:19.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:20.219 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:19.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:20.219 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:19.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:20.219 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:19.894+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:20.419+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:20.419+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:20.419+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:20.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:20.419+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:21.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:20.853+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:21.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:20.853+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:21.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:20.853+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:21.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:20.854+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:21.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:20.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:21.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:20.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:21.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:20.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:21.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:20.624+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:21.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:21.393+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:21.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:21.393+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:21.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:21.393+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:21.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:21.393+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:21.984 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:38:21.858+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 28284 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:38:21.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:21.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:21.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:21.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:21.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:21.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:21.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:21.855+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:21.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:21.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:21.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:21.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:21.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:21.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:21.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:21.635+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:22.683 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:22.389+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:22.684 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:22.389+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:22.684 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:22.389+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:22.684 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:22.389+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:23.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:22.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:23.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:22.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:23.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:22.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:23.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:22.811+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:23.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:22.682+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:23.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:22.682+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:23.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:22.682+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:23.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:22.682+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:23.640 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:23.371+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:23.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:23.371+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:23.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:23.371+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:23.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:23.371+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:24.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:23.638+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:24.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:23.638+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:24.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:23.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:24.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:23.639+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:24.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:23.838+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:24.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:23.838+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:24.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:23.838+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:24.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:23.839+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:24.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:24.403+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:24.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:24.403+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:24.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:24.403+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:24.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:24.404+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:25.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:24.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:24.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:24.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:24.625+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:25.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:24.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:25.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:24.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:25.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:24.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:25.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:24.807+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:25.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:25.360+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:25.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:25.360+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:25.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:25.360+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:25.651 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:25.360+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:25.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:25.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:25.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:25.649+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:26.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:25.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:26.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:25.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:26.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:25.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:26.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:25.842+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:26.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:26.406+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:26.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:26.406+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:26.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:26.406+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:26.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:26.406+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:26.988 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:38:26.862+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 28374 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:38:26.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:26.844+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:26.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:26.844+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:26.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:26.844+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:26.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:26.845+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:26.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:26.690+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:26.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:26.690+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:26.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:26.690+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:26.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:26.691+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:27.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:27.424+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:27.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:27.425+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:27.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:27.425+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:27.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:27.425+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:27.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:27.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:28.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:27.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:28.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:27.829+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:28.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:27.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:28.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:27.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:28.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:27.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:28.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:27.739+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:28.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:28.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:28.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:28.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:28.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:28.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:28.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:28.455+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:28.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:28.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:28.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:28.862+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:28.773+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:28.773+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:28.773+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:28.773+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:29.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:29.500+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:29.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:29.500+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:29.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:29.500+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:29.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:29.500+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:30.219 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:29.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:30.220 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:29.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:30.220 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:29.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:30.220 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:29.819+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:30.220 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:29.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:30.221 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:29.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:30.221 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:29.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:30.221 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:29.893+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:30.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:30.470+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:30.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:30.470+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:30.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:30.470+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:30.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:30.471+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:30.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:30.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:30.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:30.876+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:31.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:30.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:31.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:30.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:31.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:30.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:31.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:30.834+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:31.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:31.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:31.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:31.462+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:32.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:38:31.865+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 28464 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:38:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:31.857+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:31.857+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:31.857+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:32.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:31.858+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:31.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:31.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:31.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:31.850+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:32.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:32.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:32.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:32.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:32.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:32.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:32.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:32.423+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:32.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:32.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:32.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:32.826+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:32.824+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:32.824+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:32.824+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:32.825+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:33.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:33.469+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:33.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:33.469+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:33.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:33.469+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:33.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:33.470+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:33.814+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:33.814+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:33.814+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:33.814+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:33.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:33.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:33.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:33.861+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:34.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:34.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:34.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:34.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:34.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:34.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:34.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:34.507+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:34.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:34.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:34.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:34.909+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:34.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:34.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:34.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:34.852+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:35.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:35.515+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:35.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:35.515+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:35.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:35.515+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:35.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:35.515+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:35.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:35.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:35.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:35.932+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:36.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:35.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:36.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:35.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:36.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:35.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:36.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:35.838+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:36.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:36.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:36.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:36.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:36.848 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:36.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:36.848 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:36.547+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:37.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:38:36.869+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 28554 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:38:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:36.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:36.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:36.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:36.981+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:37.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:36.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:37.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:36.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:37.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:36.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:37.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:36.846+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:37.436 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-02-19T19:38:37.440 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T19:38:37.441 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-02-19T19:38:37.442 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-19T19:38:37.454 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-02-19T19:38:37.464 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task sleep... 2024-02-19T19:38:37.465 INFO:teuthology.task.sleep:Sleeping for 3600 seconds 2024-02-19T19:38:38.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:37.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:38.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:37.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:38.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:37.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:38.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:37.801+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:38.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:37.588+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:38.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:37.588+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:38.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:37.588+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:38.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:37.589+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:38.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:38.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:38.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:38.002+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:39.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:38.763+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:39.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:38.763+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:39.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:38.763+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:39.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:38.764+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:39.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:38.622+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:39.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:38.622+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:39.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:38.622+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:39.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:38.622+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:39.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:39.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:39.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:39.014+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:39.990 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:39.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:39.990 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:39.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:39.990 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:39.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:39.990 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:39.760+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:39.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:39.653+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:39.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:39.653+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:39.991 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:39.653+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:39.991 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:39.653+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:39.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:39.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:39.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:39.989+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:40.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:40.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:40.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:40.783+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:40.702+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:40.702+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:40.702+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:40.702+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:41.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:41.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:41.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:41.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:41.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:41.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:41.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:41.022+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:41.982 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:38:41.872+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 28645 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:38:41.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:41.743+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:41.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:41.743+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:41.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:41.743+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:41.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:41.744+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:41.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:41.715+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:41.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:41.715+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:41.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:41.715+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:41.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:41.715+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:41.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:41.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:41.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:41.981+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:43.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:42.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:43.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:42.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:43.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:42.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:43.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:42.704+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:43.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:42.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:43.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:42.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:43.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:42.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:43.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:42.718+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:43.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:43.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:43.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:43.008+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:44.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:43.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:44.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:43.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:44.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:43.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:44.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:43.671+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:44.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:43.704+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:44.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:43.704+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:44.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:43.704+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:44.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:43.704+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:44.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:44.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:44.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:44.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:44.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:44.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:44.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:44.028+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:45.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:44.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:45.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:44.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:45.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:44.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:45.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:44.706+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:45.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:44.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:45.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:44.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:45.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:44.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:45.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:44.736+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:45.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:45.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:45.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:45.008+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:46.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:45.714+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:46.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:45.714+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:46.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:45.714+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:45.714+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:46.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:45.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:46.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:45.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:46.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:45.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:46.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:45.773+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:46.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:46.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:46.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:46.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:46.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:46.038+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:47.000 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:38:46.876+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 28735 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:38:47.001 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:46.700+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:47.001 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:46.700+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:47.001 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:46.700+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:47.001 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:46.701+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:47.001 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:46.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:47.001 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:46.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:47.001 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:46.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:47.002 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:46.820+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:47.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:47.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:47.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:47.052+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:47.733+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:47.733+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:47.734+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:47.734+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:47.842+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:47.842+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:47.842+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:47.843+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:48.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:48.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:48.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:48.074+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:48.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:48.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:48.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:48.687+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:49.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:48.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:49.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:48.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:49.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:48.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:49.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:48.858+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:49.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:49.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:49.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:49.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:49.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:49.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:49.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:49.095+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:49.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:49.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:49.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:49.719+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:50.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:49.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:50.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:49.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:50.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:49.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:50.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:49.907+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:50.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:50.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:50.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:50.131+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:50.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:50.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:50.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:50.750+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:51.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:50.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:51.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:50.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:51.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:50.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:51.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:50.898+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:51.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:51.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:51.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:51.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:51.170+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:52.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:38:51.879+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 28825 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:38:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:51.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:51.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:51.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:52.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:51.766+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:52.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:51.911+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:52.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:51.911+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:52.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:51.911+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:52.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:51.912+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:52.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:52.136+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:52.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:52.136+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:52.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:52.136+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:52.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:52.137+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:53.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:52.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:53.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:52.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:53.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:52.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:53.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:52.880+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:53.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:52.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:53.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:52.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:53.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:52.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:53.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:52.754+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:53.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:53.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:53.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:53.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:53.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:53.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:53.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:53.166+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:53.836 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:53.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:53.837 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:53.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:53.837 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:53.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:53.837 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:53.707+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:54.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:53.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:54.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:53.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:54.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:53.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:54.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:53.835+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:54.213+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:54.213+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:54.213+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:54.214+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:55.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:54.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:55.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:54.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:55.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:54.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:55.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:54.832+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:54.713+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:54.713+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:54.713+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:54.714+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:55.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:55.185+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:55.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:55.185+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:55.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:55.185+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:55.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:55.186+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:56.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:55.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:56.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:55.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:56.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:55.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:56.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:55.869+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:56.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:55.744+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:56.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:55.745+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:56.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:55.745+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:56.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:55.745+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:56.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:56.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:56.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:56.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:56.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:56.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:56.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:56.234+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:57.007 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:38:56.882+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 28915 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:38:57.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:56.750+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:57.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:56.750+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:57.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:56.750+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:57.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:56.751+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:57.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:56.902+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:57.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:56.902+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:57.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:56.902+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:57.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:56.902+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:57.766 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:57.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:57.766 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:57.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:57.766 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:57.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:57.766 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:57.283+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:58.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:57.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:58.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:57.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:58.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:57.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:57.765+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:58.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:57.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:58.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:57.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:58.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:57.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:58.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:57.913+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:58.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:58.267+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:58.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:58.267+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:58.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:58.267+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:58.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:58.268+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:59.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:58.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:59.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:58.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:59.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:58.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:38:59.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:58.923+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:59.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:58.739+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:59.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:58.739+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:59.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:58.739+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:59.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:58.740+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:38:59.546 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:59.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:59.547 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:59.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:59.547 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:59.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:38:59.547 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:38:59.275+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:38:59.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:59.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:59.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:59.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:59.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:59.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:38:59.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:38:59.763+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:00.221 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:59.917+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:00.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:59.917+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:00.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:59.917+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:00.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:38:59.918+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:00.284+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:00.284+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:00.284+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:00.284+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:00.912 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:00.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:00.912 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:00.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:00.912 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:00.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:00.912 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:00.719+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:01.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:00.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:01.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:00.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:01.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:00.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:01.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:00.911+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:01.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:01.240+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:01.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:01.240+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:01.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:01.240+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:01.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:01.240+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:01.943 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:39:01.885+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 29006 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:39:01.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:01.681+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:01.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:01.681+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:01.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:01.681+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:01.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:01.681+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:02.226 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:01.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:02.226 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:01.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:02.226 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:01.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:02.226 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:01.942+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:02.225+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:02.225+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:02.225+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:02.225+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:03.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:02.678+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:03.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:02.678+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:03.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:02.678+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:03.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:02.679+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:03.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:02.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:03.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:02.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:03.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:02.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:03.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:02.894+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:03.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:03.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:03.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:03.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:03.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:03.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:03.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:03.188+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:03.865 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:03.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:03.866 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:03.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:03.866 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:03.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:03.866 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:03.632+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:04.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:03.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:04.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:03.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:04.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:03.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:04.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:03.863+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:04.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:04.236+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:04.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:04.236+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:04.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:04.236+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:04.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:04.236+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:05.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:04.641+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:05.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:04.641+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:05.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:04.641+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:05.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:04.642+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:05.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:04.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:05.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:04.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:05.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:04.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:05.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:04.873+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:05.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:05.209+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:05.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:05.209+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:05.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:05.209+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:05.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:05.209+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:06.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:05.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:06.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:05.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:06.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:05.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:06.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:05.675+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:06.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:05.865+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:06.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:05.865+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:06.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:05.865+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:06.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:05.865+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:06.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:06.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:06.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:06.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:06.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:06.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:06.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:06.162+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:07.012 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:39:06.889+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 29095 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:39:07.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:06.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:07.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:06.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:07.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:06.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:07.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:06.864+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:07.013 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:06.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:07.013 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:06.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:07.014 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:06.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:07.014 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:06.690+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:07.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:07.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:07.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:07.124+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:08.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:07.710+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:08.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:07.710+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:08.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:07.710+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:08.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:07.711+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:08.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:07.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:08.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:07.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:08.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:07.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:08.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:07.871+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:08.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:08.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:08.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:08.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:08.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:08.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:08.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:08.084+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:09.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:08.682+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:09.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:08.682+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:09.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:08.682+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:09.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:08.683+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:09.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:08.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:09.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:08.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:09.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:08.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:09.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:08.921+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:09.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:09.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:09.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:09.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:09.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:09.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:09.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:09.092+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:09.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:09.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:09.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:09.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:09.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:09.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:09.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:09.667+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:10.222 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:10.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:10.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:10.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:10.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:10.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:10.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:10.058+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:10.223 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:09.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:10.223 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:09.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:10.223 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:09.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:10.223 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:09.920+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:10.653+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:10.653+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:10.653+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:11.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:10.654+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:11.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:10.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:11.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:10.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:11.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:10.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:11.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:10.898+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:11.309 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:11.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:11.309 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:11.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:11.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:11.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:11.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:11.022+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:11.913 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:39:11.892+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 29185 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:39:11.913 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:11.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:11.913 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:11.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:11.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:11.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:11.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:11.671+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:11.914 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:11.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:11.914 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:11.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:11.914 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:11.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:11.914 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:11.861+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:11.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:11.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:11.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:11.989+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:13.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:12.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:13.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:12.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:13.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:12.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:13.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:12.698+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:13.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:12.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:13.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:12.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:13.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:12.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:13.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:12.832+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:13.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:13.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:13.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:13.004+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:13.865 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:13.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:13.866 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:13.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:13.866 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:13.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:13.866 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:13.743+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:14.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:13.864+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:14.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:13.864+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:14.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:13.864+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:14.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:13.864+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:14.040+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:14.040+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:14.040+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:14.040+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:14.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:14.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:14.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:14.703+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:15.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:14.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:15.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:14.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:15.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:14.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:15.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:14.832+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:15.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:15.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:15.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:15.026+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:15.667+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:15.667+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:15.667+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:15.668+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:16.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:15.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:16.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:15.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:16.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:15.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:16.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:15.802+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:16.354 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:16.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:16.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:16.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:16.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:16.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:16.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:16.075+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:16.909 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:39:16.895+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 29275 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:39:16.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:16.649+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:16.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:16.649+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:16.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:16.649+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:16.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:16.650+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:16.910 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:16.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:16.910 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:16.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:16.910 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:16.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:16.910 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:16.798+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:17.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:17.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:17.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:17.079+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:18.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:17.830+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:18.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:17.830+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:18.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:17.830+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:18.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:17.830+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:17.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:17.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:17.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:17.632+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:18.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:18.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:18.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:18.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:18.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:18.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:18.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:18.060+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:19.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:18.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:19.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:18.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:19.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:18.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:19.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:18.782+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:19.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:18.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:19.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:18.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:19.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:18.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:19.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:18.669+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:19.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:19.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:19.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:19.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:19.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:19.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:19.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:19.064+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:20.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:19.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:20.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:19.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:20.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:19.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:20.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:19.763+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:20.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:19.680+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:20.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:19.680+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:20.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:19.680+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:20.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:19.681+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:20.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:20.067+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:20.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:20.067+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:20.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:20.067+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:20.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:20.068+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:21.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:20.730+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:21.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:20.730+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:21.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:20.730+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:21.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:20.730+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:21.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:20.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:21.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:20.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:21.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:20.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:21.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:20.690+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:21.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:21.093+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:21.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:21.093+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:21.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:21.093+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:21.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:21.094+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:22.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:39:21.899+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 29366 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:39:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:21.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:21.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:21.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:22.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:21.714+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:22.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:21.734+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:22.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:21.734+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:22.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:21.734+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:22.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:21.735+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:22.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:22.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:22.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:22.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:22.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:22.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:22.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:22.052+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:23.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:22.743+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:23.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:22.743+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:23.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:22.743+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:23.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:22.744+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:23.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:22.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:23.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:22.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:23.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:22.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:23.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:22.751+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:23.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:23.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:23.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:23.008+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:24.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:23.718+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:24.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:23.718+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:24.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:23.718+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:24.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:23.718+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:24.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:23.761+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:24.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:23.761+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:24.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:23.761+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:24.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:23.762+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:24.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:24.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:24.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:24.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:24.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:24.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:24.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:24.030+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:25.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:24.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:25.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:24.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:25.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:24.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:25.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:24.783+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:25.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:24.743+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:25.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:24.743+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:25.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:24.743+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:25.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:24.744+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:25.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:25.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:25.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:25.011+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:25.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:25.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:25.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:25.820+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:26.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:25.768+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:26.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:25.768+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:26.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:25.768+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:26.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:25.768+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:26.448 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:26.040+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:26.449 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:26.040+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:26.449 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:26.040+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:26.449 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:26.041+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:27.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:39:26.903+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 29456 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:39:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:26.832+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:26.832+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:26.832+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:26.833+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:27.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:26.728+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:27.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:26.728+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:27.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:26.728+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:27.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:26.728+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:27.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:27.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:27.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:27.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:27.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:27.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:27.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:27.086+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:27.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:27.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:27.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:27.807+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:28.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:27.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:28.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:27.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:28.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:27.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:28.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:27.718+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:28.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:28.136+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:28.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:28.136+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:28.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:28.136+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:28.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:28.137+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:28.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:28.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:28.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:28.797+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:29.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:28.674+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:29.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:28.674+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:29.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:28.674+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:29.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:28.674+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:29.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:29.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:29.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:29.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:29.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:29.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:29.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:29.111+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:29.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:29.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:29.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:29.766+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:30.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:29.641+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:30.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:29.641+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:30.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:29.641+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:30.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:29.642+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:30.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:30.080+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:30.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:30.080+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:30.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:30.080+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:30.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:30.081+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:31.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:30.670+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:31.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:30.670+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:31.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:30.670+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:31.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:30.670+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:30.768+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:30.768+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:30.768+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:30.768+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:31.494 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:31.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:31.494 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:31.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:31.494 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:31.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:31.494 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:31.114+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:31.657+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:31.657+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:31.657+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:31.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:31.657+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:32.142 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:39:31.906+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 29546 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:39:32.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:31.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:32.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:31.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:32.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:31.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:32.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:31.784+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:32.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:32.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:32.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:32.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:32.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:32.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:32.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:32.142+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:33.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:32.615+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:33.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:32.615+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:33.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:32.615+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:33.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:32.616+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:32.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:33.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:32.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:33.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:32.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:33.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:32.800+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:33.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:33.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:33.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:33.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:33.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:33.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:33.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:33.107+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:33.838 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:33.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:33.838 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:33.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:33.839 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:33.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:33.839 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:33.587+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:34.115 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:33.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:34.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:33.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:34.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:33.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:34.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:33.838+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:34.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:34.114+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:34.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:34.114+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:34.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:34.114+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:34.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:34.115+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:34.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:34.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:34.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:34.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:34.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:34.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:34.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:34.820+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:34.900 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:34.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:34.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:34.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:34.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:34.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:34.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:34.624+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:35.085+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:35.085+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:35.085+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:35.086+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:35.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:35.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:35.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:35.804+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:36.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:35.650+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:36.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:35.650+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:36.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:35.650+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:36.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:35.650+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:36.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:36.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:36.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:36.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:36.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:36.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:36.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:36.114+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:36.809 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:36.644+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:36.809 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:36.644+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:36.809 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:36.644+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:36.809 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:36.645+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:37.112 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:39:36.910+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 29635 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:39:37.113 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:36.807+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:37.113 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:36.807+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:37.113 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:36.807+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:37.113 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:36.808+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:37.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:37.111+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:37.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:37.111+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:37.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:37.111+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:37.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:37.112+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:38.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:37.622+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:38.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:37.622+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:38.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:37.622+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:38.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:37.622+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:38.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:37.838+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:38.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:37.838+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:38.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:37.838+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:38.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:37.838+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:38.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:38.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:38.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:38.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:38.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:38.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:38.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:38.106+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:38.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:38.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:38.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:38.831+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:39.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:38.659+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:39.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:38.659+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:39.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:38.659+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:39.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:38.659+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:39.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:39.104+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:39.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:39.104+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:39.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:39.104+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:39.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:39.105+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:40.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:39.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:40.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:39.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:40.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:39.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:40.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:39.802+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:40.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:39.660+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:40.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:39.660+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:40.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:39.660+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:40.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:39.660+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:40.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:40.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:40.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:40.063+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:40.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:40.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:40.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:40.838+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:40.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:40.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:40.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:40.646+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:41.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:41.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:41.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:41.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:41.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:41.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:41.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:41.107+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:41.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:41.823+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:41.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:41.823+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:41.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:41.823+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:41.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:41.824+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:41.916 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:41.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:41.916 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:41.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:41.916 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:41.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:41.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:41.623+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:42.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:39:41.913+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 29725 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:39:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:42.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:42.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:42.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:42.082+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:43.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:42.661+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:43.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:42.661+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:43.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:42.661+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:43.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:42.661+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:43.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:42.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:43.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:42.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:43.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:42.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:43.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:42.799+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:43.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:43.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:43.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:43.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:43.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:43.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:43.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:43.106+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:44.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:43.692+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:44.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:43.692+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:44.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:43.692+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:44.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:43.693+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:44.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:43.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:44.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:43.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:44.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:43.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:44.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:43.794+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:44.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:44.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:44.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:44.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:44.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:44.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:44.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:44.092+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:44.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:44.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:44.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:44.832+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:45.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:44.725+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:45.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:44.725+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:45.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:44.725+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:45.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:44.725+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:45.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:45.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:45.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:45.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:45.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:45.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:45.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:45.124+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:45.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:45.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:46.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:45.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:46.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:45.751+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:45.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:45.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:45.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:45.879+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:46.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:46.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:46.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:46.099+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:46.918 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:46.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:46.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:46.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:46.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:46.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:46.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:46.871+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:46.919 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:46.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:46.920 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:46.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:46.920 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:46.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:46.920 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:46.788+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:47.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:39:46.917+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 29815 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:39:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:47.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:47.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:47.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:47.149+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:48.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:47.752+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:48.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:47.752+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:48.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:47.752+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:47.752+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:48.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:47.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:48.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:47.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:48.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:47.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:48.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:47.849+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:48.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:48.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:48.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:48.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:48.164+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:49.198 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:48.872+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:49.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:48.872+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:49.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:48.872+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:49.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:48.872+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:49.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:48.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:49.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:48.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:49.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:48.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:49.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:48.782+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:49.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:49.197+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:49.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:49.197+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:49.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:49.197+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:49.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:49.198+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:50.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:49.871+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:50.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:49.871+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:50.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:49.871+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:50.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:49.871+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:50.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:49.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:50.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:49.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:50.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:49.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:50.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:49.799+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:50.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:50.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:50.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:50.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:50.175+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:51.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:50.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:51.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:50.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:51.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:50.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:51.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:50.826+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:51.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:50.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:51.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:50.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:51.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:50.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:51.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:50.839+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:51.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:51.175+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:51.175+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:51.175+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:51.176+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:51.942 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:39:51.921+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 29905 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:39:51.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:51.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:51.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:51.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:51.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:51.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:51.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:51.831+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:51.944 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:51.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:51.944 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:51.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:51.944 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:51.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:51.944 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:51.831+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:52.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:52.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:52.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:52.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:52.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:52.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:52.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:52.206+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:53.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:52.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:53.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:52.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:53.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:52.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:53.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:52.836+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:53.254 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:52.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:53.254 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:52.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:53.254 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:52.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:53.254 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:52.871+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:53.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:53.252+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:53.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:53.252+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:53.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:53.252+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:53.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:53.253+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:54.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:53.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:54.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:53.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:54.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:53.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:54.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:53.855+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:54.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:53.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:54.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:53.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:54.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:53.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:54.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:53.846+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:54.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:54.294+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:54.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:54.294+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:54.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:54.294+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:54.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:54.295+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:55.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:54.827+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:55.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:54.827+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:55.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:54.827+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:55.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:54.827+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:55.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:54.890+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:55.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:54.890+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:55.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:54.890+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:55.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:54.890+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:55.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:55.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:55.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:55.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:55.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:55.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:55.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:55.323+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:56.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:55.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:56.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:55.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:56.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:55.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:56.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:55.849+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:55.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:56.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:55.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:56.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:55.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:56.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:55.876+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:56.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:56.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:56.732 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:56.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:56.732 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:56.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:56.732 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:56.367+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:57.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:39:56.924+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 29995 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:39:57.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:56.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:57.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:56.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:57.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:56.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:57.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:56.927+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:57.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:56.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:57.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:56.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:57.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:56.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:57.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:56.834+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:57.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:57.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:57.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:57.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:57.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:57.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:57.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:57.341+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:58.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:57.919+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:58.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:57.919+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:58.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:57.919+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:58.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:57.920+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:58.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:57.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:58.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:57.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:58.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:57.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:58.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:57.820+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:58.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:58.363+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:58.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:58.363+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:58.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:58.363+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:58.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:58.364+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:39:59.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:58.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:59.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:58.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:59.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:58.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:39:59.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:58.900+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:59.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:58.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:59.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:58.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:59.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:58.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:39:59.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:58.825+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:39:59.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:59.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:59.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:59.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:39:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:39:59.318+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:00.226 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:59.853+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:00.227 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:59.853+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:00.227 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:59.853+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:00.227 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:39:59.854+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:00.227 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:59.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:00.227 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:59.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:00.227 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:59.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:00.228 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:39:59.850+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:00.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:00.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:00.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:00.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:00.368+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:01.117 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:00.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:01.118 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:00.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:01.118 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:00.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:01.118 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:00.870+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:01.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:00.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:01.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:00.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:01.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:00.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:01.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:00.803+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:01.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:01.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:01.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:01.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:01.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:01.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:01.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:01.409+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:02.047 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:40:01.928+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 30085 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:40:02.047 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:01.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:02.047 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:01.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:02.047 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:01.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:02.047 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:01.842+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:02.048 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:01.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:02.048 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:01.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:02.048 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:01.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:02.048 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:01.821+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:02.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:02.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:02.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:02.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:02.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:02.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:02.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:02.417+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:03.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:02.805+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:03.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:02.805+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:03.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:02.805+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:03.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:02.806+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:03.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:02.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:03.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:02.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:03.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:02.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:03.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:02.848+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:03.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:03.427+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:03.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:03.427+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:03.763 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:03.427+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:03.763 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:03.428+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:04.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:03.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:04.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:03.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:04.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:03.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:04.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:03.761+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:04.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:03.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:04.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:03.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:04.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:03.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:04.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:03.845+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:04.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:04.433+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:04.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:04.433+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:04.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:04.433+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:04.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:04.434+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:04.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:04.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:04.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:04.799+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:04.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:04.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:04.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:04.783+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:05.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:05.438+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:05.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:05.438+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:05.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:05.438+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:05.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:05.439+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:06.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:05.790+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:06.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:05.790+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:06.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:05.790+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:39.417185+0000 (oldest deadline 2024-02-19T19:29:59.417185+0000) 2024-02-19T19:40:06.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:05.791+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:06.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:05.777+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:06.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:05.777+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:06.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:05.777+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:06.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:05.777+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:06.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:06.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:06.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:06.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:06.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:06.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:06.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:06.433+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:07.056 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:40:06.932+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 30176 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:40:07.056 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:06.839+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:07.057 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:06.773+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:07.057 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:06.773+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:07.057 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:06.773+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:07.057 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:06.774+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:07.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:07.465+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:07.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:07.465+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:07.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:07.465+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:07.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:07.466+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:08.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:07.809+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:08.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:07.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:08.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:07.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:08.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:07.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:08.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:07.755+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:08.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:08.435+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:08.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:08.435+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:08.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:08.435+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:08.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:08.435+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:09.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:08.845+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:09.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:08.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:09.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:08.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:09.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:08.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:09.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:08.757+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:09.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:09.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:09.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:09.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:09.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:09.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:09.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:09.394+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:10.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:09.861+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:09.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:09.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:09.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:09.755+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:10.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:10.384+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:10.384+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:10.384+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:10.385+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:11.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:10.846+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:10.730+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:10.731+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:10.731+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:10.731+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:11.763 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:11.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:11.764 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:11.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:11.764 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:11.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:11.764 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:11.342+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:12.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:40:11.936+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 30265 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:40:12.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:11.803+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:12.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:11.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:12.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:11.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:12.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:11.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:12.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:11.765+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:12.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:12.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:12.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:12.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:12.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:12.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:12.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:12.367+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:13.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:12.789+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:12.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:12.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:12.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:12.793+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:13.752 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:13.416+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:13.752 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:13.416+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:13.752 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:13.416+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:13.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:13.416+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:14.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:13.791+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:14.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:13.751+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:14.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:13.751+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:14.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:13.751+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:14.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:13.752+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:14.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:14.450+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:14.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:14.450+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:14.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:14.450+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:14.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:14.450+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:15.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:14.820+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:14.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:14.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:14.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:14.761+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:15.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:15.456+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:15.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:15.456+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:15.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:15.456+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:15.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:15.457+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:16.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:15.823+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:15.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:15.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:15.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:15.803+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:16.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:16.422+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:16.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:16.422+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:16.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:16.422+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:16.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:16.423+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:17.066 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:40:16.939+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 30356 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:40:17.066 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:16.777+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:17.066 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:16.784+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:17.067 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:16.784+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:17.067 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:16.784+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:17.067 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:16.784+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:17.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:17.397+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:17.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:17.397+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:17.741 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:17.397+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:17.741 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:17.397+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:18.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:17.739+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:17.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:17.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:17.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:17.811+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:18.741 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:18.351+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:18.742 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:18.351+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:18.742 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:18.351+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:18.742 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:18.352+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:19.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:18.740+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:19.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:18.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:19.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:18.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:19.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:18.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:48.046852+0000 (oldest deadline 2024-02-19T19:30:08.046852+0000) 2024-02-19T19:40:19.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:18.837+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:19.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:19.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:19.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:19.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:29:52.431518+0000 (oldest deadline 2024-02-19T19:30:12.431518+0000) 2024-02-19T19:40:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:19.310+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:20.228 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:19.800+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:20.229 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:19.789+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:20.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:20.267+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:21.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:20.813+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:21.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:20.835+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:21.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:21.268+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:22.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:40:21.943+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 30446 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:40:22.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:21.823+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:21.835+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:22.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:22.296+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:23.251 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:22.826+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:23.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:22.800+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:23.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:23.250+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:24.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:23.789+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:24.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:23.780+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:24.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:24.284+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:25.240 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:24.822+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:25.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:24.817+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:25.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:25.238+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:26.251 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:25.812+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:26.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:25.771+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:26.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:26.250+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:27.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:40:26.947+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 30537 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:40:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:26.770+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:27.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:26.754+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:27.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:27.278+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:28.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:27.770+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:27.798+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:28.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:28.287+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:29.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:28.812+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:28.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:28.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:28.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:28.829+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:29.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:29.282+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:30.229 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:29.818+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:30.229 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:29.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:30.229 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:29.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:30.230 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:29.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:30.230 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:29.831+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:30.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:30.270+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:31.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:30.803+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:31.235 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:30.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:31.235 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:30.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:31.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:30.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:31.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:30.860+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:31.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:31.234+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:32.074 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:40:31.950+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 30626 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:40:32.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:31.823+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:32.075 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:31.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:32.075 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:31.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:32.075 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:31.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:32.075 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:31.870+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:32.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:32.205+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:33.227 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:32.855+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:33.227 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:32.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:33.228 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:32.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:33.228 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:32.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:33.228 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:32.867+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:33.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:33.226+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:33.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:33.811+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:34.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:33.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:34.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:33.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:34.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:33.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:34.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:33.906+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:34.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:34.272+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:35.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:34.856+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:35.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:34.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:35.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:34.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:35.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:34.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:35.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:34.948+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:35.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:35.257+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:36.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:35.860+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:36.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:35.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:36.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:35.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:36.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:35.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:36.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:35.954+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:36.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:36.273+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:37.266 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:40:36.955+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 30717 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:40:37.267 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:36.870+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:37.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:36.945+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:37.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:36.945+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:37.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:36.945+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:37.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:36.946+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:37.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:37.265+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:38.263 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:37.836+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:38.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:37.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:38.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:37.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:38.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:37.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:38.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:37.994+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:38.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:38.262+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:39.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:38.830+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:38.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:38.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:38.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:38.950+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:39.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:39.308+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:40.229 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:39.872+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:40.230 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:39.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:40.230 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:39.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:40.230 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:39.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:40.230 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:39.959+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:40.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:40.357+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:40.357+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:40.357+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:40.358+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:41.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:40.889+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:41.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:40.917+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:41.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:40.917+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:41.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:40.917+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:41.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:40.917+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:41.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:41.339+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:41.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:41.339+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:41.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:41.339+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:41.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:41.339+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:42.157 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:40:41.959+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 30806 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:40:42.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:41.844+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:42.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:41.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:42.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:41.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:42.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:41.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:42.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:41.943+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:42.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:42.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:42.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:42.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:42.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:42.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:42.484 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:42.350+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:43.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:42.864+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:43.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:42.945+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:43.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:42.945+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:43.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:42.945+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:43.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:42.946+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:43.583 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:43.313+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:43.584 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:43.313+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:43.584 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:43.313+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:43.584 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:43.314+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:44.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:43.834+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:44.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:43.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:44.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:43.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:44.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:43.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:44.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:43.961+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:44.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:44.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:44.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:44.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:44.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:44.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:44.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:44.340+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:44.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:44.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:44.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:44.849+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:45.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:44.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:45.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:44.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:45.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:44.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:45.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:44.990+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:45.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:45.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:45.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:45.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:45.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:45.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:45.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:45.343+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:45.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:45.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:45.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:45.837+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:46.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:46.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:46.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:46.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:46.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:46.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:46.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:46.004+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:46.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:46.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:46.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:46.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:46.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:46.341+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:46.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:46.341+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:47.086 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:40:46.963+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 30897 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:40:47.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:46.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:47.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:46.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:47.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:46.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:47.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:46.835+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:47.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:47.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:47.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:47.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:47.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:47.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:47.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:47.055+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:47.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:47.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:47.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:47.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:47.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:47.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:47.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:47.307+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:48.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:48.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:48.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:48.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:48.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:48.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:48.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:48.084+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:48.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:47.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:48.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:47.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:48.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:47.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:48.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:47.863+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:48.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:48.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:48.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:48.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:48.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:48.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:48.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:48.333+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:49.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:48.912+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:48.912+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:48.912+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:48.913+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:49.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:49.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:49.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:49.073+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:49.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:49.307+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:49.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:49.307+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:49.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:49.307+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:49.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:49.308+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:50.230 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:50.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:50.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:50.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:50.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:50.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:50.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:50.027+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:50.231 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:49.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:50.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:49.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:50.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:49.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:50.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:49.948+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:50.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:50.307+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:50.307+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:50.307+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:50.308+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:51.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:51.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:51.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:51.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:51.043+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:50.967+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:50.967+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:51.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:50.967+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:51.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:50.968+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:51.699 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:51.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:51.699 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:51.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:51.699 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:51.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:51.699 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:51.350+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:52.254 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:40:51.967+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 30986 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:40:52.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:52.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:52.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:52.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:52.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:52.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:52.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:52.092+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:52.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:51.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:52.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:51.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:52.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:51.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:52.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:51.979+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:52.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:52.361+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:52.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:52.361+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:52.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:52.361+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:52.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:52.362+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:53.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:52.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:53.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:52.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:53.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:52.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:53.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:52.984+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:53.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:53.108+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:53.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:53.108+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:53.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:53.108+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:53.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:53.109+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:53.651 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:53.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:53.651 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:53.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:53.651 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:53.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:53.651 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:53.346+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:54.351 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:54.024+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:54.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:54.024+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:54.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:54.024+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:54.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:54.024+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:54.352 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:54.080+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:54.352 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:54.080+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:54.353 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:54.080+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:54.353 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:54.081+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:54.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:54.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:54.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:54.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:54.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:54.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:54.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:54.351+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:55.113 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:55.003+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:55.113 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:55.003+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:55.113 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:55.003+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:55.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:55.005+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:55.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:55.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:55.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:55.112+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:55.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:55.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:55.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:55.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:55.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:55.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:55.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:55.341+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:56.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:56.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:56.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:56.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:56.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:56.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:56.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:56.020+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:56.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:56.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:56.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:56.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:56.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:56.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:56.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:56.065+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:56.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:56.352+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:56.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:56.352+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:56.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:56.352+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:56.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:56.353+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:57.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:40:56.971+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 31077 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:40:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:56.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:56.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:56.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:56.993+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:57.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:57.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:57.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:57.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:57.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:57.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:57.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:57.059+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:57.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:57.368+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:57.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:57.368+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:57.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:57.368+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:57.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:57.368+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:58.409 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:58.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:58.410 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:58.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:58.410 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:58.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:58.410 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:58.030+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:58.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:58.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:58.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:58.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:58.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:58.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:58.411 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:58.085+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:58.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:58.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:58.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:58.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:58.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:58.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:58.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:58.409+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:40:59.458 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:59.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:59.458 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:59.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:59.458 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:59.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:40:59.458 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:40:59.079+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:59.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:59.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:59.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:59.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:59.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:59.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:40:59.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:40:59.085+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:40:59.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:59.456+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:59.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:59.456+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:59.456+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:40:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:40:59.457+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:00.448 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:00.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:00.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:00.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:00.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:00.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:00.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:00.079+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:00.449 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:00.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:00.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:00.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:00.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:00.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:00.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:00.067+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:00.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:00.446+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:00.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:00.446+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:00.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:00.446+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:00.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:00.447+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:01.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:01.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:01.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:01.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:01.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:01.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:01.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:01.038+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:01.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:01.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:01.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:01.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:01.031+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:01.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:01.447+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:01.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:01.447+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:01.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:01.447+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:01.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:01.448+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:02.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:41:01.975+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 31167 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:41:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:01.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:01.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:01.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:01.984+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:02.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:02.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:02.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:02.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:02.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:02.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:02.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:02.070+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:02.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:02.445+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:02.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:02.445+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:02.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:02.445+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:02.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:02.445+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:03.449 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:03.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:03.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:03.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:03.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:03.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:03.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:03.034+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:03.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:03.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:03.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:03.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:03.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:03.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:03.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:03.034+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:03.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:03.448+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:03.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:03.448+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:03.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:03.448+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:03.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:03.449+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:04.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:03.985+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:04.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:03.985+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:04.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:03.985+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:04.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:03.986+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:04.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:04.007+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:04.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:04.007+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:04.007+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:04.008+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:04.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:04.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:04.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:04.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:04.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:04.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:04.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:04.420+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:05.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:04.981+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:04.981+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:04.981+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:04.982+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:04.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:04.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:04.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:05.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:04.964+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:05.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:05.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:05.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:05.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:05.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:05.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:05.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:05.416+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:06.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:06.000+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:06.000+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:06.000+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:06.001+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:06.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:05.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:06.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:05.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:06.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:05.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:05.946+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:06.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:06.420+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:06.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:06.420+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:06.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:06.420+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:06.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:06.421+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:07.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:41:06.978+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 31258 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:41:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:07.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:07.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:07.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:07.041+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:06.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:06.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:06.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:06.949+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:07.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:07.388+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:07.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:07.388+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:07.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:07.388+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:07.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:07.388+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:08.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:08.017+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:08.017+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:08.017+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:08.018+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:08.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:07.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:08.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:07.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:08.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:07.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:08.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:07.900+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:08.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:08.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:08.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:08.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:08.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:08.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:08.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:08.350+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:09.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:08.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:09.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:08.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:09.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:08.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:08.975+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:09.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:08.936+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:09.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:08.936+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:09.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:08.936+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:09.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:08.936+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:09.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:09.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:09.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:09.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:09.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:09.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:09.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:09.321+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:10.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:10.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:10.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:10.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:10.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:10.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:10.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:10.017+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:10.233 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:09.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:10.233 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:09.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:10.233 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:09.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:10.233 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:09.964+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:10.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:10.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:10.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:10.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:10.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:10.348+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:11.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:10.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:11.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:10.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:11.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:10.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:11.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:10.973+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:11.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:10.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:11.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:10.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:11.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:10.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:11.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:10.999+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:11.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:11.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:11.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:11.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:11.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:11.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:11.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:11.393+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:12.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:41:11.983+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 31347 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:41:12.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:11.952+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:12.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:11.952+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:12.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:11.952+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:12.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:11.953+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:11.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:11.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:11.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:11.999+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:12.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:12.381+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:12.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:12.381+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:12.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:12.381+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:12.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:12.381+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:13.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:13.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:13.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:13.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:13.001+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:13.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:13.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:13.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:13.009+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:13.652 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:13.388+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:13.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:13.388+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:13.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:13.388+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:13.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:13.389+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:14.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:14.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:14.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:14.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:14.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:14.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:14.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:14.032+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:14.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:14.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:14.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:14.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:14.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:14.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:14.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:14.052+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:14.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:14.361+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:14.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:14.361+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:14.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:14.361+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:14.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:14.362+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:15.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:15.012+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:15.012+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:15.012+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:15.013+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:15.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:15.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:15.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:15.013+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:15.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:15.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:15.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:15.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:15.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:15.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:15.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:15.394+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:16.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:16.026+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:16.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:16.026+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:16.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:16.026+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:16.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:16.028+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:16.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:16.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:16.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:16.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:16.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:16.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:16.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:16.033+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:16.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:16.364+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:16.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:16.364+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:16.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:16.364+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:16.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:16.365+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:17.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:41:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:41:16.987+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 31438 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:41:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:17.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:17.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:17.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:17.079+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:17.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:17.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:17.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:17.080+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:17.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:17.368+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:17.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:17.368+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:17.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:17.368+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:17.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:17.369+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:18.411 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:18.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:18.412 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:18.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:18.412 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:18.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:18.412 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:18.050+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:18.412 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:18.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:18.412 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:18.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:18.412 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:18.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:18.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:18.062+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:18.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:18.410+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:18.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:18.410+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:18.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:18.410+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:18.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:18.411+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:19.398 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:19.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:19.398 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:19.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:19.398 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:19.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:19.398 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:19.062+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:19.399 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:19.090+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:19.399 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:19.090+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:19.399 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:19.090+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:19.399 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:19.091+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:19.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:19.397+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:19.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:19.397+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:19.397+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:19.398+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:20.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:20.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:20.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:20.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:20.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:20.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:20.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:20.086+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:20.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:20.099+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:20.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:20.099+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:20.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:20.099+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:20.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:20.099+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:20.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:20.363+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:20.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:20.363+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:20.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:20.363+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:20.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:20.363+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:21.398 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:21.096+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:21.398 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:21.096+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:21.398 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:21.096+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:21.398 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:21.096+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:21.398 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:21.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:21.398 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:21.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:21.399 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:21.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:21.399 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:21.134+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:21.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:21.398+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:21.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:21.398+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:21.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:21.398+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:21.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:21.398+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:22.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:41:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:41:21.990+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 31528 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:41:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:22.094+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:22.094+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:22.094+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:22.095+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:22.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:22.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:22.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:22.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:22.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:22.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:22.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:22.125+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:22.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:22.439+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:22.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:22.439+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:22.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:22.439+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:22.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:22.439+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:23.433 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:23.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:23.433 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:23.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:23.433 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:23.106+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:23.434 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:23.107+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:23.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:23.147+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:23.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:23.147+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:23.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:23.147+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:23.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:23.147+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:23.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:23.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:23.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:23.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:23.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:23.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:23.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:23.433+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:24.390 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:24.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:24.390 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:24.082+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:24.390 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:24.082+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:24.390 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:24.082+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:24.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:24.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:24.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:24.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:24.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:24.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:24.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:24.185+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:24.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:24.388+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:24.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:24.388+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:24.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:24.388+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:24.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:24.389+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:25.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:25.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:25.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:25.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:25.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:25.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:25.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:25.038+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:25.376 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:25.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:25.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:25.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:25.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:25.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:25.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:25.144+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:25.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:25.374+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:25.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:25.374+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:25.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:25.374+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:25.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:25.375+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:26.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:26.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:26.337 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:26.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:26.337 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:26.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:26.337 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:26.041+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:26.337 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:26.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:26.337 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:26.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:26.338 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:26.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:26.338 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:26.194+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:26.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:26.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:26.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:26.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:26.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:26.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:26.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:26.336+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:27.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:41:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:41:26.995+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 31619 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:41:27.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:27.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:27.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:27.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:27.191+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:27.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:26.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:27.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:26.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:27.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:26.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:27.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:26.995+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:27.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:27.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:27.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:27.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:27.583 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:27.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:27.583 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:27.314+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:28.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:27.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:28.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:27.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:28.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:27.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:28.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:27.976+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:28.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:28.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:28.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:28.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:28.192+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:28.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:28.291+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:28.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:28.291+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:28.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:28.291+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:28.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:28.292+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:29.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:29.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:29.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:29.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:29.006+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:29.165+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:29.165+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:29.165+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:29.165+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:29.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:29.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:29.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:29.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:29.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:29.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:29.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:29.319+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:30.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:30.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:30.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:30.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:30.337 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:30.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:30.337 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:30.038+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:30.337 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:30.167+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:30.337 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:30.167+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:30.337 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:30.167+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:30.337 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:30.167+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:30.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:30.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:30.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:30.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:30.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:30.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:30.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:30.336+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:31.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:31.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:31.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:31.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:31.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:31.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:31.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:31.150+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:31.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:31.062+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:31.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:31.062+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:31.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:31.062+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:31.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:31.063+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:31.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:31.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:31.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:31.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:31.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:31.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:31.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:31.306+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:32.265 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:41:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:41:31.999+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 31707 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:41:32.266 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:32.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:32.266 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:32.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:32.266 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:32.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:32.266 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:32.029+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:32.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:32.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:32.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:32.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:32.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:32.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:32.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:32.143+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:32.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:32.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:32.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:32.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:32.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:32.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:32.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:32.265+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:33.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:33.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:33.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:33.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:33.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:33.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:33.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:33.003+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:33.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:33.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:33.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:33.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:33.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:33.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:33.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:33.134+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:33.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:33.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:33.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:33.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:33.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:33.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:33.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:33.297+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:34.316 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:34.045+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:34.317 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:34.045+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:34.317 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:34.045+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:34.317 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:34.045+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:34.317 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:34.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:34.317 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:34.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:34.318 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:34.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:34.318 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:34.114+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:34.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:34.315+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:34.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:34.315+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:34.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:34.315+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:34.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:34.316+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:35.329 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:35.058+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:35.329 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:35.058+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:35.329 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:35.058+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:35.329 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:35.058+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:35.329 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:35.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:35.329 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:35.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:35.330 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:35.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:35.330 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:35.108+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:35.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:35.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:35.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:35.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:35.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:35.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:35.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:35.329+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:36.346 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:36.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:36.346 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:36.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:36.347 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:36.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:36.347 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:36.139+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:36.347 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:36.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:36.347 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:36.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:36.347 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:36.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:36.347 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:36.038+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:36.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:36.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:36.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:36.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:36.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:36.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:36.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:36.346+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:37.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:41:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:41:37.003+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 31798 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:41:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:37.013+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:37.013+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:37.013+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:37.014+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:37.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:37.098+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:37.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:37.098+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:37.098+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:37.099+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:37.676 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:37.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:37.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:37.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:37.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:37.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:37.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:37.356+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:38.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:37.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:38.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:37.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:37.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:37.996+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:38.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:38.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:38.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:38.104+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:38.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:38.308+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:38.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:38.308+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:38.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:38.308+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:38.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:38.309+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:39.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:39.008+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:39.008+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:39.008+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:39.009+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:39.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:39.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:39.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:39.065+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:39.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:39.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:39.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:39.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:39.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:39.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:39.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:39.324+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:40.315 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:40.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:40.315 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:40.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:40.315 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:40.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:40.316 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:40.052+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:40.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:40.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:40.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:40.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:40.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:40.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:40.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:40.108+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:40.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:40.313+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:40.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:40.313+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:40.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:40.313+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:40.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:40.314+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:41.074+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:41.074+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:41.074+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:41.075+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:41.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:41.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:41.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:41.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:41.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:41.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:41.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:41.075+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:41.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:41.321+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:41.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:41.321+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:41.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:41.321+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:41.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:41.322+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:42.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:41:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:41:42.007+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 31888 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:41:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:42.101+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:42.101+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:42.101+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:42.102+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:42.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:42.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:42.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:42.119+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:42.721 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:42.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:42.722 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:42.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:42.722 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:42.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:42.722 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:42.343+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:43.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:43.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:43.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:43.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:43.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:43.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:43.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:43.313+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:43.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:43.144+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:43.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:43.144+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:43.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:43.144+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:43.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:43.145+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:43.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:43.128+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:43.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:43.128+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:43.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:43.128+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:43.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:43.129+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:44.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:44.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:44.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:44.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:44.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:44.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:44.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:44.156+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:44.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:44.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:44.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:44.118+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:44.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:44.333+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:44.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:44.333+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:44.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:44.333+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:44.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:44.334+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:45.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:45.132+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:45.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:45.132+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:45.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:45.132+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:45.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:45.132+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:45.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:45.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:45.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:45.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:45.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:45.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:45.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:45.110+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:45.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:45.358+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:45.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:45.358+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:45.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:45.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:45.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:45.359+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:46.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:46.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:46.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:46.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:46.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:46.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:46.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:46.133+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:46.355 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:46.089+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:46.356 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:46.089+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:46.356 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:46.089+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:46.356 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:46.090+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:46.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:46.354+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:46.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:46.354+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:46.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:46.354+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:46.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:46.355+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:47.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:41:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:41:47.011+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 31978 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:41:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:47.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:47.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:47.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:47.171+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:47.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:47.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:47.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:47.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:47.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:47.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:47.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:47.054+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:47.767 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:47.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:47.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:47.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:47.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:47.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:47.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:47.368+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:48.344 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:48.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:48.344 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:48.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:48.344 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:48.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:48.344 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:48.078+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:48.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:48.169+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:48.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:48.169+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:48.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:48.169+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:48.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:48.169+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:48.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:48.343+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:48.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:48.343+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:48.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:48.343+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:48.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:48.343+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:49.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:49.085+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:49.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:49.085+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:49.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:49.085+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:49.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:49.085+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:49.341 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:49.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:49.341 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:49.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:49.342 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:49.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:49.342 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:49.215+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:49.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:49.339+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:49.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:49.339+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:49.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:49.339+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:49.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:49.340+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:50.318 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:50.251+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:50.319 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:50.251+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:50.319 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:50.251+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:50.319 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:50.251+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:50.319 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:50.058+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:50.319 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:50.058+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:50.319 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:50.058+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:50.319 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:50.059+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:50.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:50.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:50.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:50.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:50.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:50.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:50.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:50.318+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:51.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:51.090+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:51.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:51.090+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:51.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:51.090+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:51.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:51.091+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:51.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:51.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:51.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:51.296+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:51.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:51.237+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:51.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:51.237+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:51.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:51.237+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:51.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:51.237+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:52.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:41:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:41:52.015+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 32067 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:41:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:52.123+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:52.123+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:52.123+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:52.123+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:52.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:52.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:52.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:52.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:52.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:52.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:52.217+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:52.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:52.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:52.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:52.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:52.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:52.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:52.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:52.310+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:53.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:53.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:53.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:53.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:53.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:53.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:53.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:53.162+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:53.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:53.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:53.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:53.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:53.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:53.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:53.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:53.208+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:53.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:53.310+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:53.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:53.310+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:53.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:53.310+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:53.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:53.311+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:54.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:54.276+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:54.276+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:54.276+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:54.276+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:54.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:54.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:54.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:54.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:54.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:54.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:54.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:54.179+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:54.249+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:54.249+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:54.249+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:54.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:54.250+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:55.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:55.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:55.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:55.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:55.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:55.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:55.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:55.146+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:55.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:55.315+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:55.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:55.315+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:55.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:55.315+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:55.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:55.315+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:55.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:55.247+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:55.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:55.247+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:55.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:55.247+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:55.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:55.247+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:56.420 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:56.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:56.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:56.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:56.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:56.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:56.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:56.115+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:56.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:56.278+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:56.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:56.278+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:56.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:56.278+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:56.422 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:56.278+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:56.422 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:56.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:56.422 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:56.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:56.422 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:56.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:56.422 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:56.318+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:57.295 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:41:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:41:57.019+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 32158 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:41:57.296 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:57.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:57.296 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:57.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:57.296 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:57.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:57.296 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:57.067+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:57.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:57.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:57.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:57.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:57.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:57.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:57.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:57.327+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:57.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:57.292+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:57.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:57.292+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:57.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:57.292+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:57.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:57.292+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:58.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:58.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:58.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:58.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:58.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:58.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:58.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:58.034+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:58.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:58.303+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:58.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:58.303+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:58.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:58.303+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:58.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:58.303+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:58.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:58.338+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:58.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:58.338+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:58.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:58.338+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:58.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:58.338+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:59.328 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:59.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:59.328 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:59.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:59.329 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:59.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:41:59.329 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:41:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:41:59.078+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:41:59.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:59.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:59.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:59.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:41:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:41:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:41:59.328+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:41:59.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:59.348+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:59.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:59.348+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:59.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:59.348+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:41:59.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:41:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:41:59.349+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:00.399 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:00.055+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:00.400 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:00.055+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:00.400 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:00.055+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:00.400 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:00.056+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:00.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:00.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:00.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:00.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:00.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:00.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:00.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:00.293+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:00.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:00.398+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:00.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:00.398+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:00.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:00.398+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:00.399+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:01.367 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:01.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:01.367 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:01.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:01.367 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:01.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:01.368 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:01.092+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:01.368 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:01.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:01.368 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:01.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:01.368 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:01.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:01.368 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:01.264+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:01.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:01.366+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:01.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:01.366+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:01.366+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:01.366+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:02.280 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:42:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:42:02.024+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 32248 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:42:02.280 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:02.069+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:02.280 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:02.069+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:02.280 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:02.069+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:02.280 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:02.069+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:02.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:02.279+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:02.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:02.279+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:02.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:02.279+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:02.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:02.280+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:02.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:02.319+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:02.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:02.319+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:02.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:02.319+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:02.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:02.319+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:03.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:03.057+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:03.262 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:03.057+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:03.262 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:03.057+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:03.262 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:03.058+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:03.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:03.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:03.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:03.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:03.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:03.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:03.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:03.261+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:03.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:03.334+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:03.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:03.334+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:03.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:03.335+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:03.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:03.335+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:04.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:04.097+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:04.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:04.097+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:04.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:04.097+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:04.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:04.098+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:04.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:04.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:04.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:04.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:04.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:04.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:04.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:04.283+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:04.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:04.334+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:04.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:04.334+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:04.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:04.334+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:04.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:04.334+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:05.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:05.272+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:05.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:05.272+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:05.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:05.272+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:05.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:05.273+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:05.428 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:05.348+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:05.428 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:05.348+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:05.428 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:05.348+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:05.428 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:05.348+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:05.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:05.118+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:05.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:05.118+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:05.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:05.118+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:05.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:05.119+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:06.332 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:06.071+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:06.333 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:06.071+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:06.333 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:06.071+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:06.333 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:06.072+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:06.333 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:06.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:06.333 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:06.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:06.333 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:06.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:06.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:06.320+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:06.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:06.332+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:06.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:06.332+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:06.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:06.332+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:06.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:06.332+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:07.298 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:42:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:42:07.028+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 32336 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:42:07.298 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:07.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:07.298 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:07.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:07.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:07.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:07.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:07.031+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:07.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:07.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:07.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:07.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:07.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:07.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:07.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:07.298+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:07.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:07.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:07.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:07.376+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:08.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:08.015+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:08.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:08.015+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:08.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:08.015+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:08.016+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:08.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:08.329+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:08.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:08.329+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:08.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:08.329+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:08.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:08.329+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:08.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:08.386+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:08.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:08.386+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:08.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:08.386+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:08.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:08.387+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:09.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:08.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:09.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:08.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:09.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:08.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:09.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:08.978+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:09.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:09.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:09.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:09.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:09.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:09.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:09.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:09.284+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:09.379+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:09.379+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:09.379+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:09.379+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:10.416 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:10.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:10.416 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:10.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:10.416 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:10.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:10.416 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:10.019+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:10.417 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:10.265+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:10.417 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:10.265+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:10.417 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:10.265+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:10.417 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:10.266+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:10.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:10.414+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:10.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:10.414+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:10.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:10.414+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:10.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:10.415+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:11.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:10.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:11.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:10.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:11.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:10.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:11.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:10.988+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:11.571 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:11.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:11.572 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:11.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:11.572 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:11.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:11.572 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:11.285+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:11.572 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:11.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:11.572 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:11.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:11.573 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:11.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:11.573 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:11.420+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:12.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:42:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:42:12.032+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 32422 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:42:12.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:11.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:12.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:11.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:12.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:11.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:12.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:11.999+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:12.239+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:12.239+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:12.239+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:12.240+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:12.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:12.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:12.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:12.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:12.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:12.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:12.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:12.441+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:13.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:12.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:13.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:12.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:13.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:12.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:12.983+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:13.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:13.281+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:13.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:13.281+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:13.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:13.281+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:13.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:13.282+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:13.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:13.413+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:13.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:13.413+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:13.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:13.413+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:13.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:13.414+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:14.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:13.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:14.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:13.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:14.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:13.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:13.959+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:14.643 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:14.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:14.643 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:14.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:14.643 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:14.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:14.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:14.298+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:14.644 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:14.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:14.644 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:14.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:14.644 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:14.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:14.644 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:14.440+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:15.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:14.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:15.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:14.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:15.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:14.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:14.935+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:15.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:15.456+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:15.456+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:15.456+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:15.456+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:15.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:15.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:15.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:15.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:15.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:15.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:15.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:15.275+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:16.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:15.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:16.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:15.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:15.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:15.985+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:16.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:16.295+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:16.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:16.295+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:16.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:16.295+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:16.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:16.295+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:16.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:16.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:16.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:16.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:16.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:16.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:16.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:16.440+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:17.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:42:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:42:17.036+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 32510 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:42:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:16.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:16.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:16.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:16.957+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:17.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:17.322+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:17.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:17.322+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:17.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:17.322+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:17.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:17.323+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:17.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:17.405+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:17.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:17.406+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:17.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:17.406+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:17.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:17.406+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:18.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:17.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:18.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:17.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:17.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:17.911+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:18.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:18.304+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:18.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:18.304+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:18.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:18.304+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:18.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:18.305+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:18.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:18.375+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:18.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:18.375+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:18.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:18.375+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:18.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:18.376+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:19.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:18.863+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:19.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:18.863+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:19.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:18.863+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:19.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:18.864+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:19.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:19.291+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:19.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:19.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:19.292+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:19.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:19.342+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:19.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:19.342+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:19.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:19.342+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:19.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:19.343+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:20.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:19.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:20.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:19.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:20.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:19.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:20.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:19.834+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:20.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:20.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:20.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:20.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:20.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:20.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:20.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:20.305+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:20.367+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:20.367+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:20.367+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:20.368+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:21.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:20.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:21.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:20.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:21.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:20.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:21.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:20.871+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:21.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:21.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:21.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:21.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:21.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:21.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:21.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:21.349+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:21.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:21.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:21.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:21.416+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:22.165 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:42:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:42:22.040+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 32591 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:42:22.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:21.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:22.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:21.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:22.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:21.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:22.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:21.907+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:22.415 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:22.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:22.416 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:22.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:22.416 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:22.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:22.416 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:22.341+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:22.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:22.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:22.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:22.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:22.415+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:22.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:22.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:22.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:22.875+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:23.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:23.344+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:23.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:23.344+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:23.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:23.344+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:23.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:23.345+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:23.658 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:23.378+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:23.658 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:23.378+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:23.658 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:23.378+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:23.658 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:23.378+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:24.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:23.899+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:24.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:23.899+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:24.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:23.899+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:24.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:23.899+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:24.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:24.322+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:24.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:24.322+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:24.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:24.322+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:24.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:24.323+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:24.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:24.397+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:24.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:24.397+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:24.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:24.397+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:24.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:24.398+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:25.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:24.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:25.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:24.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:25.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:24.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:25.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:24.853+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:25.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:25.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:25.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:25.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:25.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:25.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:25.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:25.318+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:25.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:25.352+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:25.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:25.352+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:25.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:25.352+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:25.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:25.352+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:26.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:25.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:26.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:25.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:26.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:25.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:26.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:25.830+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:26.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:26.329+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:26.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:26.329+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:26.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:26.329+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:26.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:26.330+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:26.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:26.392+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:26.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:26.392+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:26.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:26.392+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:26.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:26.393+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:27.057 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:42:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:42:27.044+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 32673 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:42:27.057 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:26.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:27.057 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:26.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:27.058 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:26.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:27.058 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:26.800+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:27.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:27.367+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:27.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:27.367+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:27.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:27.367+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:27.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:27.367+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:27.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:27.354+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:27.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:27.354+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:27.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:27.354+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:27.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:27.355+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:28.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:27.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:28.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:27.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:28.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:27.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:28.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:27.843+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:28.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:28.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:28.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:28.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:28.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:28.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:28.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:28.348+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:28.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:28.409+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:28.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:28.409+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:28.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:28.409+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:28.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:28.409+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:29.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:28.857+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:29.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:28.857+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:29.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:28.857+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:28.857+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:29.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:29.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:29.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:29.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:29.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:29.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:29.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:29.351+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:29.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:29.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:29.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:29.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:29.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:29.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:29.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:29.411+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:30.236 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:29.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:30.236 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:29.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:30.236 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:29.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:30.236 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:29.856+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:30.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:30.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:30.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:30.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:30.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:30.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:30.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:30.366+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:30.399+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:30.399+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:30.399+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:30.400+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:31.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:30.843+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:31.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:30.843+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:31.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:30.843+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:30.844+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:31.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:31.321+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:31.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:31.321+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:31.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:31.321+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:31.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:31.322+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:31.406+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:31.406+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:31.406+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:31.407+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:32.172 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:42:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:42:32.048+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 32756 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:42:32.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:31.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:32.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:31.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:32.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:31.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:32.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:31.856+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:32.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:32.278+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:32.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:32.278+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:32.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:32.278+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:32.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:32.279+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:32.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:32.438+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:32.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:32.438+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:32.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:32.438+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:32.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:32.439+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:33.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:32.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:33.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:32.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:33.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:32.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:33.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:32.817+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:33.447 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:33.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:33.448 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:33.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:33.448 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:33.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:33.448 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:33.294+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:33.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:33.446+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:33.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:33.446+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:33.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:33.446+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:33.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:33.447+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:34.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:33.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:34.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:33.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:34.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:33.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:33.850+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:34.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:34.334+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:34.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:34.334+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:34.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:34.334+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:34.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:34.335+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:34.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:34.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:34.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:34.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:34.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:34.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:34.442+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:35.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:34.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:35.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:34.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:35.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:34.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:35.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:34.842+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:35.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:35.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:35.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:35.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:35.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:35.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:35.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:35.324+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:35.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:35.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:35.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:35.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:35.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:35.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:35.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:35.487+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:36.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:35.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:36.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:35.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:36.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:35.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:36.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:35.857+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:36.474 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:36.369+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:36.475 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:36.369+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:36.475 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:36.369+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:36.475 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:36.369+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:36.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:36.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:36.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:36.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:36.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:36.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:36.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:36.474+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:37.176 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:42:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:42:37.052+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 32841 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:42:37.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:36.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:37.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:36.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:37.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:36.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:37.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:36.848+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:37.469 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:37.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:37.470 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:37.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:37.470 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:37.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:37.470 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:37.326+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:37.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:37.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:37.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:37.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:37.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:37.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:37.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:37.469+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:38.230 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:37.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:38.231 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:37.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:38.231 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:37.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:38.231 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:37.834+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:38.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:38.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:38.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:38.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:38.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:38.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:38.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:38.348+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:38.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:38.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:38.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:38.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:38.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:38.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:38.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:38.442+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:39.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:38.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:39.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:38.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:39.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:38.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:38.850+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:39.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:39.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:39.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:39.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:39.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:39.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:39.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:39.346+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:39.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:39.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:39.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:39.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:39.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:39.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:39.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:39.450+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:40.237 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:39.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:40.237 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:39.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:40.237 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:39.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:40.238 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:39.850+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:40.498 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:40.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:40.498 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:40.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:40.498 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:40.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:40.498 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:40.337+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:40.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:40.497+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:40.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:40.497+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:40.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:40.497+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:40.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:40.497+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:41.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:40.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:41.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:40.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:41.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:40.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:40.870+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:41.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:41.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:41.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:41.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:41.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:41.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:41.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:41.335+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:41.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:41.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:41.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:41.499+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:42.181 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:42:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:42:42.057+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 32928 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:42:42.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:41.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:42.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:41.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:42.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:41.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:42.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:41.882+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:42.516 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:42.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:42.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:42.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:42.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:42.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:42.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:42.297+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:42.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:42.515+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:42.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:42.515+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:42.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:42.515+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:42.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:42.516+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:43.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:42.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:43.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:42.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:43.250 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:42.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:43.250 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:42.908+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:43.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:43.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:43.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:43.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:43.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:43.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:43.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:43.249+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:43.943 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:43.545+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:43.943 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:43.545+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:43.943 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:43.545+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:43.943 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:43.546+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:44.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:43.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:44.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:43.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:44.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:43.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:43.943+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:44.556 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:44.287+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:44.556 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:44.287+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:44.556 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:44.287+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:44.556 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:44.288+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:44.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:44.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:44.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:44.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:44.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:44.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:44.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:44.555+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:45.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:44.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:45.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:44.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:45.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:44.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:45.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:44.984+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:45.595 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:45.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:45.595 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:45.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:45.595 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:45.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:45.595 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:45.317+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:46.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:45.594+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:46.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:45.594+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:46.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:45.594+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:46.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:45.594+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:46.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:46.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:46.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:46.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:46.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:46.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:46.011+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:46.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:46.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:46.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:46.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:46.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:46.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:46.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:46.275+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:47.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:46.609+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:47.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:46.609+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:47.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:46.609+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:47.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:46.609+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:47.299 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:42:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:42:47.061+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 33019 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:42:47.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:47.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:47.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:47.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:47.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:47.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:47.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:47.041+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:47.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:47.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:47.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:47.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:47.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:47.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:47.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:47.299+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:48.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:47.627+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:48.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:47.627+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:48.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:47.627+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:48.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:47.627+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:48.305 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:48.304+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:48.306 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:48.304+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:48.306 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:48.304+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:48.306 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:48.305+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:48.306 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:48.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:48.306 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:48.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:48.306 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:48.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:48.306 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:48.055+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:49.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:48.651+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:49.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:48.651+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:49.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:48.651+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:49.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:48.651+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:49.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:49.266+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:49.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:49.266+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:49.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:49.266+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:49.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:49.266+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:49.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:49.060+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:49.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:49.060+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:49.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:49.060+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:49.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:49.061+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:50.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:49.629+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:50.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:49.629+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:50.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:49.629+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:50.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:49.630+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:50.307 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:50.043+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:50.307 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:50.043+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:50.307 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:50.043+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:50.308 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:50.043+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:50.600 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:50.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:50.600 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:50.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:50.600 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:50.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:50.600 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:50.307+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:51.000 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:50.599+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:51.001 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:50.599+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:51.001 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:50.599+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:51.001 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:50.599+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:51.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:51.000+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:51.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:51.000+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:51.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:51.000+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:51.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:51.000+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:51.648 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:51.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:51.648 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:51.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:51.649 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:51.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:51.649 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:51.328+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:51.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:51.647+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:51.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:51.647+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:51.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:51.647+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:51.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:51.648+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:52.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:42:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:42:52.065+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 33108 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:42:52.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:51.981+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:51.981+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:51.981+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:51.981+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:52.624 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:52.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:52.624 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:52.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:52.625 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:52.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:52.625 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:52.293+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:52.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:52.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:52.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:52.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:52.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:52.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:52.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:52.624+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:53.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:52.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:53.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:52.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:53.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:52.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:53.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:52.980+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:53.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:53.249+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:53.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:53.249+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:53.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:53.249+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:53.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:53.249+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:53.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:53.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:53.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:53.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:53.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:53.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:53.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:53.645+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:54.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:53.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:54.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:53.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:54.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:53.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:54.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:53.972+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:54.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:54.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:54.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:54.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:54.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:54.264+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:54.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:54.264+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:54.929 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:54.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:54.929 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:54.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:54.929 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:54.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:54.930 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:54.680+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:55.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:54.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:55.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:54.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:55.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:54.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:55.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:54.929+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:55.632 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:55.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:55.632 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:55.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:55.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:55.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:55.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:55.272+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:55.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:55.631+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:55.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:55.631+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:55.902 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:55.631+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:55.902 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:55.631+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:56.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:55.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:56.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:55.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:56.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:55.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:56.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:55.901+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:56.588 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:56.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:56.589 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:56.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:56.589 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:56.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:56.589 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:56.297+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:56.921 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:56.588+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:56.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:56.588+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:56.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:56.588+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:56.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:56.588+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:57.193 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:42:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:42:57.069+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 33199 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:42:57.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:56.921+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:57.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:56.921+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:57.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:56.921+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:57.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:56.921+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:57.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:57.307+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:57.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:57.307+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:57.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:57.307+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:57.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:57.307+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:57.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:57.615+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:57.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:57.615+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:57.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:57.615+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:57.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:57.615+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:58.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:57.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:58.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:57.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:58.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:57.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:58.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:57.965+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:58.590 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:58.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:58.591 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:58.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:58.591 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:58.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:58.591 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:58.332+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:58.925 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:58.590+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:58.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:58.590+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:58.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:58.590+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:58.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:58.590+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:59.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:58.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:59.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:58.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:59.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:58.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:42:59.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:58.925+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:42:59.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:59.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:59.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:59.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:59.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:59.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:42:59.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:42:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:42:59.315+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:42:59.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:59.640+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:59.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:59.640+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:59.954 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:59.640+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:42:59.954 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:42:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:42:59.640+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:00.238 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:59.952+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:00.238 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:59.952+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:00.238 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:59.952+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:00.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:42:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:42:59.953+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:00.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:00.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:00.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:00.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:00.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:00.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:00.348+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:00.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:00.669+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:00.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:00.669+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:00.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:00.669+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:00.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:00.670+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:01.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:00.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:01.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:00.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:01.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:00.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:00.970+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:01.649 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:01.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:01.649 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:01.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:01.650 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:01.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:01.650 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:01.319+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:02.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:01.648+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:02.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:01.648+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:02.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:01.648+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:02.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:01.649+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:02.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:43:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:43:02.073+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 33289 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:43:02.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:02.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:02.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:02.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:02.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:02.003+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:02.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:02.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:02.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:02.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:02.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:02.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:02.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:02.349+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:02.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:02.692+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:02.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:02.692+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:02.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:02.692+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:02.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:02.692+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:03.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:02.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:03.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:02.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:03.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:02.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:03.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:02.972+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:03.580 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:03.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:03.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:03.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:03.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:03.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:03.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:03.326+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:03.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:03.716+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:03.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:03.716+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:03.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:03.716+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:03.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:03.716+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:04.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:03.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:04.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:03.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:04.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:03.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:04.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:03.963+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:04.698 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:04.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:04.698 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:04.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:04.698 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:04.335+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:04.698 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:04.336+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:04.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:04.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:04.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:04.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:04.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:04.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:04.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:04.697+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:05.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:04.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:05.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:04.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:05.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:04.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:05.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:04.985+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:05.709 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:05.353+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:05.709 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:05.353+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:05.709 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:05.353+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:05.709 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:05.353+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:05.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:05.708+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:05.989 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:05.708+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:05.989 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:05.708+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:05.989 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:05.709+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:06.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:05.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:06.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:05.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:06.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:05.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:06.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:05.988+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:06.729 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:06.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:06.730 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:06.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:06.730 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:06.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:06.730 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:06.350+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:07.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:06.728+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:07.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:06.728+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:07.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:06.728+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:07.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:06.729+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:07.371 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:43:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:43:07.077+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 33380 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:43:07.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:07.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:07.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:07.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:07.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:07.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:07.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:07.019+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:07.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:07.371+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:07.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:07.371+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:07.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:07.371+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:07.705 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:07.372+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:08.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:07.703+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:08.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:07.703+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:08.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:07.703+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:08.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:07.704+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:08.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:08.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:08.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:08.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:08.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:08.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:08.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:08.006+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:08.739 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:08.399+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:08.739 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:08.399+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:08.739 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:08.399+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:08.739 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:08.400+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:08.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:08.738+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:08.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:08.738+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:08.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:08.738+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:08.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:08.739+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:09.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:08.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:09.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:08.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:09.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:08.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:09.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:08.990+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:09.721 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:09.407+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:09.721 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:09.407+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:09.721 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:09.407+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:09.721 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:09.407+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:09.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:09.720+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:09.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:09.720+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:09.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:09.720+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:09.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:09.720+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:10.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:09.986+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:10.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:09.986+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:10.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:09.986+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:10.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:09.986+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:10.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:10.442+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:10.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:10.442+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:10.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:10.442+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:10.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:10.443+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:11.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:10.979+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:11.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:10.979+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:11.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:10.979+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:11.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:10.980+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:11.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:10.749+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:11.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:10.749+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:11.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:10.749+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:11.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:10.749+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:11.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:11.449+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:11.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:11.449+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:11.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:11.449+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:11.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:11.449+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:12.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:11.991+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:12.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:11.991+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:12.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:11.991+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:12.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:11.992+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:12.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:11.744+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:12.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:11.744+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:12.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:11.744+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:12.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:11.744+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:12.409 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:43:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:43:12.082+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 33469 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:43:12.748 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:12.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:12.748 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:12.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:12.748 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:12.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:12.748 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:12.409+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:13.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:12.747+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:13.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:12.747+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:13.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:12.747+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:13.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:12.748+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:13.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:13.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:13.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:13.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:13.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:13.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:13.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:13.017+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:13.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:13.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:13.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:13.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:13.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:13.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:13.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:13.380+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:13.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:13.713+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:13.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:13.713+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:13.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:13.713+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:13.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:13.714+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:14.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:13.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:14.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:13.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:14.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:13.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:14.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:13.971+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:14.724 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:14.402+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:14.724 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:14.402+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:14.725 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:14.402+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:14.725 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:14.403+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:15.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:14.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:15.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:14.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:15.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:14.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:15.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:14.723+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:14.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:14.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:14.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:14.963+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:15.748 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:15.385+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:15.748 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:15.385+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:15.748 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:15.385+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:15.749 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:15.386+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:16.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:15.747+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:16.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:15.747+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:16.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:15.747+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:16.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:15.748+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:15.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:15.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:15.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:15.925+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:16.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:16.395+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:16.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:16.395+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:16.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:16.395+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:16.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:16.396+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:17.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:16.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:17.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:16.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:17.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:16.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:17.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:16.926+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:17.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:16.779+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:17.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:16.779+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:17.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:16.779+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:17.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:16.780+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:17.381 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:43:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:43:17.086+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 33559 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:43:17.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:17.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:17.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:17.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:17.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:17.380+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:17.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:17.381+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:18.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:17.921+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:18.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:17.921+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:17.921+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:17.922+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:18.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:17.767+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:18.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:17.767+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:18.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:17.767+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:18.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:17.767+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:18.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:18.370+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:18.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:18.370+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:18.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:18.370+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:18.732 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:18.370+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:19.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:18.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:19.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:18.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:19.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:18.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:19.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:18.973+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:19.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:18.730+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:19.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:18.730+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:19.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:18.730+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:19.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:18.731+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:19.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:19.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:19.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:19.326+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:20.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:19.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:20.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:19.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:20.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:19.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:20.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:19.937+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:20.239 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:19.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:20.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:19.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:20.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:19.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:20.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:19.772+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:20.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:20.302+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:20.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:20.302+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:20.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:20.302+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:20.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:20.302+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:21.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:20.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:21.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:20.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:21.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:20.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:21.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:20.899+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:20.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:20.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:20.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:20.773+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:21.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:21.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:21.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:21.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:21.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:21.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:21.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:21.327+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:22.091 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:21.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:22.091 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:21.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:22.091 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:21.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:22.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:21.894+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:22.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:21.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:22.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:21.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:22.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:21.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:22.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:21.815+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:22.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:43:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:43:22.090+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 33650 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:43:22.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:22.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:22.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:22.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:22.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:22.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:22.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:22.297+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:23.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:22.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:23.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:22.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:23.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:22.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:23.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:22.894+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:23.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:22.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:23.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:22.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:23.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:22.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:23.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:22.815+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:23.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:23.268+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:23.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:23.268+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:23.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:23.268+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:23.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:23.268+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:24.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:23.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:24.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:23.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:24.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:23.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:24.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:23.846+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:24.254 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:23.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:24.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:23.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:24.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:23.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:24.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:23.858+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:24.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:24.253+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:24.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:24.253+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:24.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:24.253+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:24.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:24.254+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:25.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:24.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:25.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:24.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:25.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:24.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:25.236 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:24.814+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:25.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:24.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:25.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:24.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:25.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:24.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:25.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:24.860+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:25.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:25.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:25.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:25.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:25.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:25.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:25.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:25.235+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:26.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:25.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:26.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:25.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:26.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:25.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:26.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:25.810+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:26.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:25.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:26.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:25.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:26.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:25.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:26.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:25.882+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:26.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:26.198+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:26.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:26.198+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:26.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:26.198+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:26.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:26.199+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:27.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:26.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:27.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:26.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:27.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:26.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:27.097 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:26.801+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:27.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:26.864+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:27.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:26.864+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:27.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:26.864+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:27.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:26.865+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:27.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:43:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:43:27.094+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 33740 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:43:27.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:27.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:27.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:27.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:27.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:27.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:27.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:27.200+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:28.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:27.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:28.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:27.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:28.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:27.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:28.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:27.893+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:27.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:27.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:27.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:27.760+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:28.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:28.165+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:28.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:28.165+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:28.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:28.165+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:28.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:28.166+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:29.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:28.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:29.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:28.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:29.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:28.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:28.943+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:28.761+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:28.761+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:28.761+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:28.762+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:29.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:29.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:29.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:29.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:29.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:29.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:29.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:29.189+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:30.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:29.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:30.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:29.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:30.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:29.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:30.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:29.772+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:30.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:29.953+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:30.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:29.953+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:30.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:29.953+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:30.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:29.953+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:30.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:30.140+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:30.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:30.140+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:30.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:30.140+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:30.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:30.141+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:31.113 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:30.818+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:31.113 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:30.818+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:31.113 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:30.818+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:31.113 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:30.819+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:31.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:30.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:31.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:30.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:31.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:30.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:31.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:30.949+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:31.442 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:31.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:31.443 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:31.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:31.443 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:31.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:31.443 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:31.113+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:32.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:31.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:32.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:31.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:32.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:31.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:32.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:31.816+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:32.080 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:31.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:32.080 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:31.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:32.080 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:31.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:32.080 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:31.953+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:32.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:43:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:43:32.099+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 33830 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:43:32.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:32.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:32.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:32.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:32.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:32.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:32.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:32.079+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:33.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:32.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:33.130 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:32.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:33.130 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:32.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:33.130 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:32.987+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:33.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:32.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:33.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:32.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:33.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:32.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:33.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:32.819+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:33.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:33.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:33.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:33.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:33.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:33.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:33.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:33.130+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:34.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:33.844+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:34.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:33.844+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:34.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:33.844+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:34.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:33.845+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:34.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:34.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:34.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:34.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:34.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:34.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:34.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:34.031+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:34.514 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:34.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:34.515 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:34.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:34.515 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:34.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:34.515 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:34.175+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:35.204 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:34.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:35.204 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:34.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:35.204 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:34.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:35.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:34.821+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:35.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:35.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:35.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:35.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:35.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:35.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:35.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:35.064+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:35.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:35.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:35.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:35.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:35.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:35.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:35.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:35.204+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:36.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:36.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:36.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:36.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:36.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:36.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:36.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:36.025+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:36.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:35.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:36.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:35.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:36.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:35.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:36.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:35.779+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:36.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:36.182+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:36.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:36.182+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:36.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:36.182+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:36.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:36.183+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:37.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:36.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:37.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:36.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:36.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:36.766+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:37.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:43:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:43:37.103+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 33920 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:43:37.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:37.155+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:37.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:37.155+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:37.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:37.155+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:37.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:37.156+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:37.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:37.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:37.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:37.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:37.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:37.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:37.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:37.047+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:38.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:37.744+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:38.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:37.744+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:38.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:37.744+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:38.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:37.744+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:38.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:38.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:38.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:38.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:38.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:38.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:38.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:38.166+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:38.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:38.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:38.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:38.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:38.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:38.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:38.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:38.024+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:39.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:38.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:39.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:38.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:39.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:38.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:39.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:38.785+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:39.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:39.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:39.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:39.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:39.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:39.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:39.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:39.189+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:39.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:39.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:39.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:39.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:39.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:39.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:39.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:39.073+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:40.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:39.744+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:40.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:39.744+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:40.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:39.744+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:40.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:39.745+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:40.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:40.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:40.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:40.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:40.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:40.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:40.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:40.081+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:40.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:40.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:40.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:40.169+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:41.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:40.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:41.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:40.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:41.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:40.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:41.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:40.706+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:41.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:41.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:41.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:41.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:41.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:41.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:41.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:41.109+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:41.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:41.213+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:41.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:41.213+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:41.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:41.214+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:41.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:41.214+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:42.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:41.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:42.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:41.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:42.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:41.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:42.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:41.704+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:42.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:43:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:43:42.107+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 34011 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:43:42.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:42.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:42.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:42.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:42.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:42.130+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:42.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:42.208+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:42.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:42.208+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:42.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:42.208+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:42.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:42.208+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:43.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:42.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:43.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:42.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:43.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:42.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:43.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:42.671+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:43.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:43.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:43.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:43.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:43.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:43.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:43.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:43.124+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:43.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:43.207+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:43.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:43.207+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:43.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:43.207+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:43.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:43.208+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:43.848 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:43.646+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:43.848 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:43.646+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:43.849 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:43.646+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:43.849 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:43.647+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:44.233 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:44.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:44.233 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:44.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:44.233 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:44.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:44.233 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:44.092+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:44.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:44.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:44.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:44.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:44.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:44.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:44.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:44.233+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:45.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:44.690+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:45.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:44.690+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:45.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:44.690+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:44.690+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:45.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:45.184+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:45.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:45.184+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:45.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:45.184+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:45.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:45.184+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:45.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:45.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:45.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:45.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:45.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:45.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:45.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:45.071+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:46.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:45.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:46.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:45.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:46.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:45.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:46.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:45.713+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:46.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:46.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:46.532 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:46.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:46.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:46.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:46.533 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:46.042+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:46.533 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:46.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:46.533 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:46.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:46.533 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:46.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:46.533 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:46.177+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:47.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:46.685+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:47.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:46.685+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:47.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:46.685+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:47.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:46.685+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:47.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:43:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:43:47.111+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 34101 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:43:47.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:47.161+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:47.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:47.161+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:47.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:47.161+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:47.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:47.162+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:47.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:47.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:47.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:47.014+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:48.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:47.640+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:48.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:47.640+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:48.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:47.640+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:47.641+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:48.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:48.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:48.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:48.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:48.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:48.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:48.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:48.022+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:48.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:48.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:48.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:48.113+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:48.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:48.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:48.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:48.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:48.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:48.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:48.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:48.604+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:49.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:49.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:49.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:49.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:49.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:49.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:49.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:49.023+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:49.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:49.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:49.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:49.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:49.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:49.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:49.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:49.090+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:49.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:49.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:49.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:49.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:49.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:49.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:49.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:49.637+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:50.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:50.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:50.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:50.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:50.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:50.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:50.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:50.108+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:50.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:49.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:50.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:49.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:50.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:49.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:50.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:49.991+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:50.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:50.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:50.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:50.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:50.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:50.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:50.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:50.626+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:51.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:50.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:51.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:50.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:51.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:50.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:51.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:50.955+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:51.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:51.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:51.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:51.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:51.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:51.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:51.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:51.129+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:51.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:51.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:51.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:51.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:51.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:51.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:51.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:51.618+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:52.240 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:43:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:43:52.115+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 34191 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:43:52.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:52.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:52.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:52.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:52.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:52.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:52.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:52.149+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:52.241 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:51.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:52.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:51.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:52.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:51.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:52.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:51.956+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:52.922 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:52.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:52.922 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:52.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:52.922 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:52.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:52.922 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:52.580+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:53.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:53.126+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:53.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:53.126+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:53.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:53.126+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:53.126+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:53.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:52.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:53.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:52.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:53.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:52.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:53.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:52.921+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:53.876 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:53.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:53.876 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:53.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:53.877 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:53.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:53.877 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:53.594+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:54.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:54.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:54.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:54.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:54.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:54.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:54.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:54.099+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:54.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:53.875+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:54.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:53.875+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:54.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:53.875+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:54.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:53.876+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:55.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:54.633+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:55.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:54.633+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:54.633+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:54.634+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:55.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:54.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:55.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:54.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:55.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:54.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:55.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:54.842+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:55.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:55.135+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:55.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:55.135+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:55.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:55.135+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:55.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:55.136+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:56.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:55.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:56.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:55.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:56.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:55.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:56.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:55.826+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:56.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:55.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:56.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:55.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:56.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:55.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:56.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:55.650+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:56.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:56.171+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:56.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:56.171+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:56.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:56.171+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:56.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:56.171+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:57.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:56.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:57.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:56.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:57.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:56.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:57.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:56.616+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:57.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:56.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:57.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:56.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:57.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:56.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:57.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:56.822+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:57.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:43:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:43:57.120+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 34282 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:43:57.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:57.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:57.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:57.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:57.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:57.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:57.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:57.171+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:58.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:57.639+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:58.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:57.639+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:58.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:57.639+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:58.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:57.640+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:58.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:57.840+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:58.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:57.840+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:58.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:57.840+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:58.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:57.841+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:58.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:58.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:58.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:58.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:58.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:58.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:58.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:58.139+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:43:58.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:58.798+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:58.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:58.798+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:58.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:58.798+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:43:58.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:58.799+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:58.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:58.633+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:58.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:58.633+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:58.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:58.633+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:43:58.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:58.634+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:43:59.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:59.173+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:59.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:59.173+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:59.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:59.173+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:43:59.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:43:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:43:59.174+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:00.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:59.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:00.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:59.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:00.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:59.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:00.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:43:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:43:59.598+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:00.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:59.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:00.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:59.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:00.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:59.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:00.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:43:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:43:59.791+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:00.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:00.203+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:00.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:00.203+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:00.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:00.203+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:00.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:00.204+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:01.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:00.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:01.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:00.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:01.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:00.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:01.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:00.603+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:01.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:00.805+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:01.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:00.806+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:01.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:00.806+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:01.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:00.806+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:01.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:01.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:01.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:01.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:01.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:01.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:01.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:01.190+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:02.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:01.562+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:01.562+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:01.562+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:01.563+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:02.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:01.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:02.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:01.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:02.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:01.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:02.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:01.777+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:02.517 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:44:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:44:02.124+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 34374 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:44:02.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:02.210+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:02.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:02.210+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:02.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:02.210+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:02.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:02.211+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:02.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:02.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:02.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:02.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:02.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:02.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:02.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:02.518+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:03.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:02.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:03.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:02.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:03.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:02.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:03.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:02.820+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:03.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:03.244+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:03.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:03.244+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:03.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:03.244+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:03.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:03.245+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:03.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:03.481+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:03.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:03.481+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:03.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:03.481+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:03.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:03.482+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:04.233 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:03.837+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:04.234 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:03.837+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:04.234 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:03.837+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:04.234 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:03.837+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:04.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:04.459+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:04.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:04.459+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:04.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:04.459+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:04.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:04.460+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:04.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:04.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:04.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:04.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:04.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:04.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:04.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:04.233+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:05.232 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:04.826+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:05.232 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:04.826+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:05.232 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:04.826+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:05.233 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:04.826+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:05.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:05.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:05.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:05.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:05.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:05.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:05.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:05.232+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:05.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:05.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:05.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:05.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:05.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:05.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:05.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:05.472+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:06.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:05.875+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:06.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:05.875+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:06.237 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:05.875+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:06.237 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:05.875+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:06.488 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:06.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:06.488 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:06.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:06.488 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:06.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:06.488 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:06.236+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:06.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:06.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:06.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:06.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:06.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:06.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:06.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:06.488+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:07.130 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:06.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:07.130 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:06.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:07.130 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:06.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:07.130 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:06.859+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:07.510 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:44:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:44:07.129+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 34465 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:44:07.510 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:07.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:07.510 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:07.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:07.510 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:07.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:07.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:07.207+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:07.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:07.509+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:07.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:07.509+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:07.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:07.509+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:07.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:07.510+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:08.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:07.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:08.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:07.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:08.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:07.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:08.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:07.816+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:08.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:08.489+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:08.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:08.489+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:08.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:08.489+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:08.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:08.490+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:08.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:08.250+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:08.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:08.250+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:08.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:08.250+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:08.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:08.251+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:09.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:08.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:09.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:08.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:09.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:08.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:09.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:08.836+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:09.498 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:09.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:09.499 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:09.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:09.499 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:09.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:09.499 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:09.207+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:09.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:09.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:09.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:09.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:09.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:09.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:09.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:09.499+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:10.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:09.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:10.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:09.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:10.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:09.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:10.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:09.800+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:10.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:10.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:10.484 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:10.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:10.484 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:10.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:10.484 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:10.165+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:10.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:10.483+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:10.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:10.483+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:10.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:10.483+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:10.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:10.484+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:11.198 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:10.790+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:11.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:10.790+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:11.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:10.790+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:11.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:10.790+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:11.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:11.198+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:11.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:11.198+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:11.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:11.198+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:11.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:11.199+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:11.806 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:11.532+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:11.806 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:11.532+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:11.807 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:11.532+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:11.807 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:11.532+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:12.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:11.805+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:12.137 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:11.805+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:12.137 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:11.805+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:12.137 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:11.806+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:12.504 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:44:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:44:12.134+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 34552 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:44:12.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:12.182+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:12.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:12.182+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:12.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:12.182+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:12.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:12.183+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:12.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:12.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:12.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:12.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:12.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:12.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:12.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:12.504+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:13.223 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:12.774+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:13.224 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:12.774+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:13.224 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:12.774+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:13.224 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:12.774+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:13.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:13.223+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:13.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:13.223+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:13.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:13.223+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:13.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:13.224+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:13.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:13.534+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:13.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:13.534+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:13.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:13.534+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:13.790 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:13.535+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:14.127 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:13.789+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:14.127 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:13.789+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:14.127 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:13.789+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:14.127 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:13.790+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:14.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:14.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:14.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:14.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:14.451 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:14.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:14.451 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:14.177+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:14.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:14.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:14.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:14.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:14.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:14.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:14.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:14.508+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:15.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:14.786+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:15.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:14.786+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:15.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:14.786+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:15.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:14.787+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:15.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:15.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:15.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:15.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:15.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:15.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:15.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:15.150+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:16.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:15.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:16.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:15.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:16.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:15.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:15.551+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:16.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:15.747+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:16.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:15.747+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:16.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:15.747+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:16.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:15.748+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:16.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:16.192+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:16.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:16.192+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:16.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:16.192+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:16.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:16.193+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:17.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:16.592+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:17.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:16.592+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:17.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:16.592+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:17.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:16.593+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:16.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:16.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:16.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:16.791+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:17.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:44:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:44:17.138+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 34643 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:44:17.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:17.179+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:17.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:17.179+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:17.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:17.179+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:17.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:17.180+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:18.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:17.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:18.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:17.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:18.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:17.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:18.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:17.804+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:17.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:17.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:17.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:17.605+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:18.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:18.207+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:18.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:18.207+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:18.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:18.207+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:18.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:18.207+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:19.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:18.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:19.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:18.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:19.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:18.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:19.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:18.831+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:19.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:18.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:19.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:18.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:19.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:18.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:19.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:18.590+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:19.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:19.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:19.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:19.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:19.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:19.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:19.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:19.252+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:19.804 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:19.548+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:19.804 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:19.548+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:19.805 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:19.548+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:19.805 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:19.549+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:20.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:19.804+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:20.243 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:19.804+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:20.243 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:19.804+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:20.243 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:19.804+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:20.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:20.286+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:20.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:20.286+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:20.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:20.286+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:20.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:20.287+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:20.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:20.561+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:20.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:20.561+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:20.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:20.561+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:20.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:20.562+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:21.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:20.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:21.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:20.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:21.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:20.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:21.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:20.832+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:21.552 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:21.286+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:21.553 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:21.286+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:21.553 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:21.286+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:21.553 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:21.287+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:21.820 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:21.548+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:21.820 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:21.548+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:21.820 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:21.548+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:21.821 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:21.548+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:22.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:21.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:22.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:21.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:22.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:21.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:22.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:21.820+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:22.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:44:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:44:22.142+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 34735 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:44:22.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:22.328+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:22.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:22.328+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:22.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:22.328+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:22.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:22.328+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:22.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:22.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:22.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:22.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:22.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:22.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:22.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:22.578+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:22.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:22.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:22.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:22.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:22.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:22.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:22.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:22.787+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:23.543 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:23.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:23.544 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:23.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:23.544 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:23.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:23.544 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:23.348+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:24.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:23.600+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:24.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:23.600+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:24.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:23.600+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:24.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:23.601+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:24.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:23.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:24.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:23.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:24.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:23.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:24.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:23.762+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:24.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:24.361+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:24.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:24.361+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:24.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:24.361+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:24.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:24.362+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:24.803 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:24.641+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:24.804 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:24.641+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:24.804 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:24.641+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:24.804 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:24.642+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:25.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:24.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:25.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:24.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:25.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:24.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:25.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:24.803+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:25.686 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:25.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:25.686 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:25.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:25.687 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:25.318+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:25.687 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:25.318+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:26.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:25.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:26.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:25.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:26.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:25.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:26.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:25.686+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:26.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:25.826+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:26.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:25.826+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:26.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:25.826+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:26.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:25.826+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:26.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:26.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:26.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:26.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:26.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:26.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:26.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:26.348+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:27.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:26.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:27.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:26.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:27.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:26.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:27.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:26.695+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:27.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:26.806+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:27.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:26.806+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:27.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:26.806+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:27.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:26.806+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:27.398 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:44:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:44:27.146+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 34826 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:44:27.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:27.398+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:27.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:27.398+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:27.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:27.398+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:27.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:27.399+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:28.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:27.657+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:28.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:27.657+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:28.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:27.657+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:28.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:27.658+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:28.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:27.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:28.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:27.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:28.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:27.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:28.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:27.759+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:28.617 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:28.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:28.618 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:28.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:28.618 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:28.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:28.618 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:28.356+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:29.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:28.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:29.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:28.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:29.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:28.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:29.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:28.618+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:28.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:28.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:28.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:28.802+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:29.612 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:29.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:29.612 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:29.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:29.612 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:29.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:29.613 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:29.332+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:30.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:29.805+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:30.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:29.805+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:30.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:29.805+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:30.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:29.806+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:29.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:29.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:29.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:29.612+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:30.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:30.311+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:30.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:30.311+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:30.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:30.311+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:30.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:30.312+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:31.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:30.580+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:31.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:30.580+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:31.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:30.580+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:31.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:30.580+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:31.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:30.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:31.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:30.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:31.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:30.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:31.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:30.812+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:31.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:31.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:31.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:31.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:31.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:31.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:31.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:31.551+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:31.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:31.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:31.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:31.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:31.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:31.323+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:31.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:31.323+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:32.151 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:31.811+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:32.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:31.811+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:32.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:31.811+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:32.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:31.812+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:32.512 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:44:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:44:32.151+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 34917 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:44:32.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:32.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:32.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:32.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:32.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:32.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:32.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:32.327+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:32.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:32.512+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:32.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:32.512+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:32.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:32.512+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:32.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:32.513+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:32.769+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:32.769+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:32.769+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:32.770+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:33.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:33.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:33.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:33.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:33.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:33.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:33.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:33.300+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:33.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:33.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:33.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:33.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:33.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:33.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:33.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:33.464+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:34.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:33.786+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:34.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:33.786+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:34.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:33.786+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:34.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:33.787+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:34.597 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:34.473+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:34.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:34.473+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:34.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:34.473+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:34.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:34.474+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:34.598 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:34.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:34.598 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:34.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:34.599 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:34.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:34.599 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:34.313+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:35.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:34.785+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:35.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:34.785+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:35.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:34.785+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:35.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:34.785+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:35.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:35.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:35.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:35.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:35.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:35.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:35.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:35.348+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:35.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:35.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:35.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:35.459+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:35.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:35.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:35.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:35.789+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:36.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:36.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:36.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:36.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:36.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:36.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:36.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:36.327+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:36.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:36.490+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:36.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:36.490+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:36.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:36.490+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:36.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:36.490+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:37.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:36.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:37.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:36.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:37.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:36.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:37.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:36.804+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:37.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:44:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:44:37.155+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 35008 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:44:37.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:37.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:37.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:37.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:37.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:37.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:37.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:37.346+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:37.813 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:37.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:37.814 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:37.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:37.814 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:37.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:37.814 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:37.537+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:38.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:37.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:38.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:37.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:38.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:37.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:38.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:37.813+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:38.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:38.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:38.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:38.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:38.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:38.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:38.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:38.314+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:38.528+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:38.528+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:38.528+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:38.528+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:39.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:38.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:39.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:38.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:39.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:38.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:38.849+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:39.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:39.273+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:39.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:39.273+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:39.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:39.273+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:39.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:39.274+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:39.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:39.494+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:39.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:39.494+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:39.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:39.494+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:39.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:39.495+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:40.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:39.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:40.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:39.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:40.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:39.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:40.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:39.889+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:40.498 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:40.230+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:40.498 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:40.230+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:40.498 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:40.230+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:40.498 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:40.231+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:40.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:40.497+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:40.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:40.497+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:40.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:40.497+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:40.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:40.498+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:41.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:40.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:41.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:40.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:41.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:40.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:41.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:40.934+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:41.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:41.259+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:41.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:41.259+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:41.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:41.259+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:41.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:41.260+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:41.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:41.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:41.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:41.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:41.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:41.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:41.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:41.499+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:42.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:44:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:44:42.160+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 35100 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:44:42.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:41.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:42.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:41.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:42.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:41.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:42.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:41.962+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:42.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:42.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:42.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:42.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:42.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:42.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:42.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:42.504+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:42.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:42.300+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:42.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:42.300+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:42.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:42.300+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:42.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:42.300+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:43.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:42.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:43.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:42.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:43.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:42.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:43.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:42.993+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:43.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:43.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:43.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:43.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:43.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:43.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:43.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:43.468+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:43.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:43.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:43.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:43.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:43.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:43.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:43.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:43.294+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:44.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:43.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:44.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:43.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:44.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:43.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:44.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:43.989+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:44.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:44.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:44.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:44.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:44.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:44.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:44.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:44.467+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:44.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:44.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:44.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:44.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:44.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:44.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:44.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:44.328+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:45.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:45.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:45.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:45.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:45.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:45.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:45.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:45.036+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:45.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:45.511+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:45.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:45.511+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:45.511+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:45.512+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:45.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:45.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:45.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:45.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:45.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:45.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:45.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:45.320+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:46.304 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:46.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:46.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:46.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:46.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:46.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:46.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:46.036+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:46.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:46.304+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:46.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:46.304+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:46.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:46.304+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:46.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:46.305+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:46.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:46.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:46.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:46.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:46.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:46.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:46.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:46.510+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:47.350 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:44:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:44:47.164+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 35191 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:44:47.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:47.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:47.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:47.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:47.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:47.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:47.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:47.075+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:47.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:47.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:47.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:47.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:47.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:47.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:47.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:47.463+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:47.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:47.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:47.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:47.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:47.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:47.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:47.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:47.350+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:48.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:48.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:48.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:48.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:48.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:48.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:48.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:48.076+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:48.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:48.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:48.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:48.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:48.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:48.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:48.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:48.419+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:48.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:48.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:48.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:48.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:48.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:48.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:48.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:48.356+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:49.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:49.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:49.326 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:49.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:49.326 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:49.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:49.326 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:49.054+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:49.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:49.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:49.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:49.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:49.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:49.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:49.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:49.422+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:49.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:49.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:49.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:49.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:49.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:49.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:49.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:49.326+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:50.370 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:50.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:50.370 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:50.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:50.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:50.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:50.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:50.101+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:50.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:50.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:50.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:50.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:50.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:50.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:50.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:50.456+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:50.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:50.370+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:50.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:50.370+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:50.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:50.370+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:50.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:50.370+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:51.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:51.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:51.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:51.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:51.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:51.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:51.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:51.107+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:51.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:51.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:51.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:51.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:51.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:51.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:51.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:51.499+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:51.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:51.358+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:51.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:51.358+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:51.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:51.358+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:51.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:51.359+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:52.473 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:44:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:44:52.169+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 35282 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:44:52.473 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:52.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:52.473 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:52.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:52.473 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:52.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:52.474 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:52.333+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:52.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:52.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:52.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:52.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:52.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:52.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:52.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:52.134+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:52.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:52.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:52.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:52.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:52.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:52.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:52.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:52.473+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:53.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:53.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:53.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:53.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:53.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:53.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:53.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:53.327+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:53.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:53.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:53.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:53.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:53.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:53.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:53.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:53.104+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:53.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:53.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:53.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:53.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:53.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:53.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:53.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:53.519+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:54.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:54.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:54.314 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:54.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:54.314 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:54.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:54.314 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:54.063+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:54.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:54.313+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:54.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:54.313+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:54.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:54.313+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:54.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:54.314+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:54.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:54.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:54.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:54.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:54.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:54.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:54.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:54.537+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:55.293 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:55.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:55.293 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:55.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:55.293 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:55.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:55.293 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:55.038+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:55.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:55.528+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:55.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:55.528+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:55.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:55.528+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:55.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:55.529+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:55.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:55.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:55.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:55.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:55.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:55.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:55.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:55.293+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:56.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:56.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:56.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:56.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:56.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:56.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:56.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:56.004+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:56.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:56.243+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:56.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:56.243+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:56.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:56.243+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:56.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:56.244+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:56.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:56.513+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:56.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:56.513+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:56.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:56.513+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:56.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:56.514+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:57.255 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:44:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:44:57.174+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 35373 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:44:57.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:56.965+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:57.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:56.965+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:57.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:56.965+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:57.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:56.965+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:57.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:57.254+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:57.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:57.254+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:57.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:57.254+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:57.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:57.255+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:57.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:57.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:57.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:57.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:57.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:57.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:57.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:57.536+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:58.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:57.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:58.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:57.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:58.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:57.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:58.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:57.942+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:58.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:58.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:58.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:58.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:58.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:58.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:58.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:58.499+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:58.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:58.284+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:58.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:58.284+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:58.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:58.284+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:58.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:58.284+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:44:59.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:58.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:59.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:58.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:59.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:58.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:44:59.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:44:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:44:58.989+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:59.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:59.465+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:59.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:59.465+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:59.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:59.465+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:44:59.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:44:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:44:59.465+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:44:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:59.328+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:59.328+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:59.328+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:44:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:44:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:44:59.329+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:00.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:00.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:00.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:00.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:00.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:00.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:00.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:00.019+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:00.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:00.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:00.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:00.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:00.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:00.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:00.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:00.463+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:00.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:00.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:00.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:00.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:00.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:00.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:00.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:00.299+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:01.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:01.000+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:01.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:01.000+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:01.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:01.000+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:01.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:01.000+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:01.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:01.499+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:01.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:01.499+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:01.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:01.499+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:01.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:01.500+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:01.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:01.259+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:01.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:01.259+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:01.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:01.259+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:01.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:01.260+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:02.266 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:45:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:45:02.178+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 35465 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:45:02.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:02.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:02.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:02.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:02.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:02.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:02.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:02.003+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:02.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:02.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:02.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:02.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:02.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:02.478+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:02.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:02.479+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:02.266+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:02.266+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:02.266+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:02.267+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:03.473 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:03.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:03.473 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:03.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:03.473 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:03.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:03.473 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:03.033+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:03.474 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:03.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:03.474 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:03.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:03.474 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:03.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:03.474 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:03.225+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:03.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:03.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:03.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:03.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:03.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:03.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:03.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:03.473+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:04.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:03.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:04.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:03.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:04.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:03.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:04.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:03.989+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:04.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:04.429+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:04.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:04.429+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:04.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:04.429+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:04.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:04.430+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:04.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:04.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:04.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:04.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:04.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:04.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:04.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:04.245+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:05.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:04.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:05.252 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:04.951+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:05.252 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:04.951+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:05.252 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:04.951+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:05.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:05.381+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:05.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:05.381+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:05.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:05.381+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:05.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:05.381+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:05.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:05.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:05.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:05.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:05.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:05.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:05.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:05.252+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:06.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:05.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:06.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:05.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:06.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:05.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:06.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:05.999+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:06.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:06.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:06.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:06.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:06.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:06.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:06.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:06.425+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:06.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:06.270+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:06.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:06.270+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:06.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:06.270+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:06.676 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:06.271+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:07.235 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:45:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:45:07.183+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 35556 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:45:07.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:06.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:07.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:06.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:07.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:06.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:07.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:06.967+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:07.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:07.415+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:07.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:07.415+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:07.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:07.415+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:07.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:07.416+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:07.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:07.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:07.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:07.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:07.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:07.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:07.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:07.236+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:08.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:07.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:08.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:07.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:08.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:07.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:08.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:07.984+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:08.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:08.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:08.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:08.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:08.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:08.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:08.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:08.275+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:08.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:08.457+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:08.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:08.457+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:08.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:08.457+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:08.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:08.458+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:09.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:08.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:09.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:08.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:09.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:08.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:09.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:08.965+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:09.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:09.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:09.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:09.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:09.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:09.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:09.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:09.248+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:09.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:09.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:09.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:09.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:09.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:09.488+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:10.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:09.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:10.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:09.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:10.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:09.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:10.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:09.979+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:10.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:10.482+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:10.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:10.482+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:10.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:10.482+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:10.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:10.482+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:10.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:10.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:10.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:10.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:10.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:10.282+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:11.330 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:11.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:11.330 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:11.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:11.331 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:11.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:11.331 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:11.027+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:11.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:11.464+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:11.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:11.464+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:11.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:11.464+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:11.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:11.464+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:11.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:11.330+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:11.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:11.330+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:11.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:11.330+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:11.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:11.330+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:12.309 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:45:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:45:12.186+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 35647 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:45:12.310 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:12.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:12.310 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:12.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:12.310 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:12.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:12.310 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:12.025+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:12.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:12.466+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:12.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:12.466+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:12.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:12.466+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:12.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:12.466+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:12.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:12.344+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:12.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:12.344+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:12.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:12.344+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:12.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:12.345+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:13.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:13.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:13.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:13.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:13.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:13.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:13.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:13.070+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:13.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:13.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:13.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:13.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:13.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:13.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:13.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:13.328+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:13.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:13.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:13.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:13.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:13.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:13.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:13.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:13.489+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:14.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:14.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:14.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:14.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:14.374 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:14.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:14.374 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:14.037+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:14.687 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:14.373+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:14.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:14.373+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:14.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:14.373+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:14.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:14.373+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:14.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:14.454+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:14.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:14.454+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:14.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:14.454+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:14.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:14.454+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:15.333 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:15.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:15.333 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:15.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:15.333 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:15.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:15.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:15.053+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:15.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:15.333+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:15.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:15.333+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:15.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:15.333+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:15.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:15.333+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:15.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:15.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:15.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:15.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:15.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:15.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:15.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:15.426+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:16.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:16.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:16.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:16.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:16.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:16.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:16.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:16.065+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:16.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:16.352+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:16.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:16.352+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:16.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:16.352+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:16.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:16.353+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:16.430+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:16.430+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:16.430+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:16.431+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:17.403 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:45:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:45:17.192+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 35730 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:45:17.403 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:17.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:17.403 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:17.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:17.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:17.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:17.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:17.088+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:17.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:17.315+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:17.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:17.315+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:17.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:17.315+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:17.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:17.316+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:17.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:17.402+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:17.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:17.402+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:17.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:17.402+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:17.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:17.403+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:18.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:18.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:18.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:18.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:18.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:18.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:18.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:18.299+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:18.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:18.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:18.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:18.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:18.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:18.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:18.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:18.083+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:18.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:18.385+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:18.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:18.385+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:18.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:18.385+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:18.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:18.386+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:19.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:19.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:19.411 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:19.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:19.411 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:19.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:19.411 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:19.055+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:19.411 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:19.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:19.411 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:19.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:19.412 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:19.271+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:19.412 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:19.272+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:19.734 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:19.410+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:19.734 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:19.410+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:19.734 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:19.410+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:19.734 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:19.411+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:20.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:20.007+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:20.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:20.007+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:20.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:20.007+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:20.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:20.007+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:20.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:20.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:20.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:20.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:20.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:20.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:20.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:20.368+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:20.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:20.261+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:20.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:20.261+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:20.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:20.261+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:20.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:20.261+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:21.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:20.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:21.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:20.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:21.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:20.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:20.994+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:21.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:21.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:21.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:21.235+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:21.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:21.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:21.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:21.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:21.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:21.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:21.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:21.328+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:22.208 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:45:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:45:22.196+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 35821 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:45:22.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:21.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:22.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:21.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:22.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:21.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:22.209 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:21.950+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:22.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:22.329+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:22.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:22.329+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:22.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:22.329+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:22.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:22.330+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:22.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:22.280+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:22.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:22.280+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:22.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:22.280+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:22.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:22.281+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:23.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:22.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:23.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:22.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:23.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:22.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:23.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:22.948+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:23.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:23.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:23.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:23.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:23.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:23.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:23.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:23.354+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:23.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:23.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:23.583 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:23.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:23.583 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:23.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:23.583 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:23.293+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:24.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:23.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:24.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:23.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:24.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:23.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:24.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:23.931+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:24.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:24.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:24.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:24.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:24.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:24.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:24.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:24.396+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:24.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:24.300+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:24.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:24.300+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:24.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:24.300+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:24.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:24.301+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:25.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:24.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:25.108 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:24.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:25.108 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:24.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:25.108 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:24.969+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:25.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:25.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:25.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:25.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:25.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:25.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:25.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:25.343+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:25.768 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:25.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:25.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:25.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:25.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:25.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:25.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:25.368+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:26.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:25.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:26.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:25.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:26.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:25.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:26.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:25.975+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:26.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:26.348+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:26.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:26.348+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:26.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:26.348+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:26.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:26.349+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:26.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:26.389+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:26.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:26.389+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:26.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:26.389+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:26.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:26.390+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:27.268 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:45:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:45:27.200+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 35911 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:45:27.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:26.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:27.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:26.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:27.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:26.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:27.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:26.992+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:27.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:27.366+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:27.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:27.366+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:27.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:27.366+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:27.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:27.366+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:27.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:27.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:27.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:27.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:27.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:27.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:27.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:27.415+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:28.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:27.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:28.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:27.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:28.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:27.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:28.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:27.948+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:28.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:28.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:28.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:28.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:28.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:28.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:28.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:28.331+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:28.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:28.434+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:28.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:28.434+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:28.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:28.434+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:28.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:28.435+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:29.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:28.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:29.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:28.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:29.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:28.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:29.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:28.943+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:29.768 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:29.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:29.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:29.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:29.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:29.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:29.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:29.290+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:29.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:29.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:29.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:29.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:29.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:29.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:29.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:29.409+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:30.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:29.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:30.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:29.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:30.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:29.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:30.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:29.937+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:30.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:30.379+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:30.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:30.379+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:30.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:30.379+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:30.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:30.381+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:30.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:30.292+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:30.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:30.292+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:30.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:30.292+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:30.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:30.292+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:31.250 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:30.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:31.250 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:30.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:31.250 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:30.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:31.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:30.958+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:31.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:31.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:31.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:31.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:31.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:31.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:31.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:31.409+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:31.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:31.287+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:31.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:31.287+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:31.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:31.287+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:31.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:31.287+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:32.214 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:45:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:45:32.205+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 36003 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:45:32.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:31.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:32.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:31.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:32.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:31.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:32.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:31.962+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:32.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:32.392+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:32.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:32.392+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:32.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:32.392+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:32.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:32.392+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:32.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:32.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:32.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:32.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:32.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:32.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:32.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:32.336+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:33.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:32.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:32.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:32.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:32.914+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:33.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:33.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:33.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:33.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:33.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:33.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:33.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:33.415+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:33.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:33.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:33.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:33.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:33.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:33.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:33.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:33.302+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:34.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:33.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:34.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:33.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:34.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:33.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:34.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:33.934+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:34.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:34.458+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:34.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:34.458+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:34.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:34.458+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:34.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:34.459+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:34.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:34.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:34.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:34.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:34.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:34.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:34.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:34.340+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:35.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:34.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:35.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:34.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:35.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:34.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:35.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:34.961+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:35.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:35.450+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:35.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:35.450+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:35.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:35.450+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:35.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:35.451+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:35.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:35.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:35.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:35.374+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:35.971+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:35.971+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:35.971+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:35.972+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:36.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:36.385+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:36.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:36.386+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:36.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:36.386+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:36.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:36.386+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:36.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:36.413+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:36.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:36.413+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:36.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:36.413+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:36.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:36.414+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:37.268 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:45:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:45:37.209+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 36093 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:45:37.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:37.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:37.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:37.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:37.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:37.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:37.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:37.002+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:37.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:37.400+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:37.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:37.400+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:37.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:37.400+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:37.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:37.401+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:37.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:37.429+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:37.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:37.429+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:37.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:37.429+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:37.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:37.429+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:38.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:38.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:38.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:38.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:38.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:38.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:38.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:38.041+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:38.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:38.477+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:38.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:38.477+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:38.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:38.477+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:38.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:38.477+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:38.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:38.391+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:38.391+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:38.391+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:38.392+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:39.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:39.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:39.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:39.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:39.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:39.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:39.428 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:39.085+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:39.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:39.427+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:39.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:39.427+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:39.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:39.427+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:39.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:39.427+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:39.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:39.439+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:39.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:39.439+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:39.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:39.439+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:39.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:39.439+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:40.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:40.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:40.249 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:40.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:40.249 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:40.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:40.249 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:40.081+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:40.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:40.473+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:40.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:40.473+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:40.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:40.473+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:40.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:40.473+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:40.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:40.471+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:40.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:40.471+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:40.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:40.471+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:40.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:40.472+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:41.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:41.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:41.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:41.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:41.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:41.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:41.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:41.078+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:41.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:41.512+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:41.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:41.512+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:41.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:41.512+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:41.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:41.512+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:41.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:41.457+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:41.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:41.457+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:41.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:41.457+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:41.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:41.457+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:42.421 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:45:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:45:42.214+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 36186 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:45:42.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:42.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:42.422 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:42.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:42.422 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:42.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:42.422 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:42.089+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:42.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:42.421+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:42.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:42.421+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:42.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:42.421+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:42.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:42.421+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:42.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:42.557+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:42.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:42.557+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:42.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:42.557+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:42.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:42.558+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:43.405 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:43.093+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:43.405 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:43.093+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:43.405 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:43.093+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:43.405 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:43.094+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:43.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:43.404+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:43.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:43.404+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:43.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:43.404+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:43.669 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:43.405+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:43.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:43.549+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:43.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:43.549+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:43.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:43.549+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:43.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:43.550+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:44.385 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:44.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:44.385 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:44.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:44.385 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:44.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:44.385 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:44.056+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:44.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:44.384+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:44.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:44.384+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:44.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:44.384+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:44.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:44.385+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:44.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:44.520+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:44.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:44.520+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:44.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:44.520+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:44.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:44.520+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:45.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:45.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:45.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:45.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:45.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:45.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:45.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:45.063+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:45.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:45.425+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:45.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:45.425+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:45.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:45.426+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:45.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:45.426+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:45.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:45.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:45.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:45.536+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:46.462 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:46.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:46.462 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:46.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:46.462 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:46.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:46.463 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:46.111+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:46.768 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:46.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:46.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:46.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:46.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:46.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:46.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:46.570+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:46.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:46.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:46.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:46.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:46.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:46.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:46.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:46.462+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:47.343 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:45:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:45:47.219+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 36272 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:45:47.344 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:47.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:47.344 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:47.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:47.344 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:47.082+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:47.344 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:47.082+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:47.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:47.583+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:47.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:47.583+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:47.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:47.583+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:47.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:47.584+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:47.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:47.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:47.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:47.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:47.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:47.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:47.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:47.487+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:48.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:48.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:48.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:48.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:48.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:48.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:48.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:48.119+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:48.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:48.499+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:48.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:48.499+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:48.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:48.499+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:48.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:48.500+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:48.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:48.563+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:48.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:48.563+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:48.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:48.563+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:48.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:48.564+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:49.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:49.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:49.504 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:49.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:49.504 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:49.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:49.504 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:49.117+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:49.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:49.503+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:49.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:49.503+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:49.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:49.503+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:49.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:49.503+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:49.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:49.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:49.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:49.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:49.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:49.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:49.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:49.579+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:50.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:50.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:50.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:50.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:50.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:50.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:50.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:50.159+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:50.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:50.504+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:50.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:50.504+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:50.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:50.504+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:50.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:50.505+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:50.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:50.552+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:50.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:50.552+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:50.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:50.552+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:50.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:50.553+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:51.483 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:51.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:51.483 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:51.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:51.483 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:51.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:51.483 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:51.172+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:51.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:51.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:51.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:51.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:51.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:51.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:51.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:51.583+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:51.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:51.482+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:51.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:51.482+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:51.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:51.482+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:51.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:51.483+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:52.518 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:45:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:45:52.223+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 36364 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:45:52.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:52.183+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:52.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:52.183+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:52.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:52.183+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:52.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:52.183+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:53.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:52.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:53.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:52.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:53.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:52.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:53.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:52.599+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:53.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:52.527+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:53.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:52.527+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:53.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:52.527+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:53.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:52.527+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:53.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:53.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:53.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:53.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:53.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:53.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:53.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:53.204+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:54.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:53.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:54.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:53.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:54.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:53.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:54.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:53.560+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:54.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:53.528+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:54.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:53.528+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:53.528+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:53.529+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:54.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:54.252+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:54.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:54.252+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:54.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:54.252+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:54.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:54.252+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:55.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:54.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:55.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:54.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:55.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:54.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:55.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:54.571+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:55.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:54.561+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:55.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:54.561+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:54.561+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:54.562+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:55.226+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:55.226+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:55.226+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:55.226+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:56.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:55.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:56.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:55.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:56.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:55.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:56.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:55.594+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:56.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:55.597+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:56.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:55.597+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:56.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:55.597+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:56.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:55.598+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:56.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:56.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:56.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:56.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:56.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:56.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:56.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:56.194+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:57.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:56.557+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:57.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:56.557+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:57.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:56.557+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:57.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:56.558+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:57.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:56.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:57.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:56.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:57.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:56.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:57.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:56.632+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:57.518 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:45:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:45:57.228+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 36454 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:45:57.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:57.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:57.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:57.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:57.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:57.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:57.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:57.149+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:58.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:57.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:58.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:57.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:58.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:57.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:58.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:57.560+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:58.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:57.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:57.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:57.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:57.649+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:58.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:58.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:58.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:58.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:58.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:58.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:58.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:58.198+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:59.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:58.651+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:59.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:58.651+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:59.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:58.651+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:45:59.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:58.652+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:45:59.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:58.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:59.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:58.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:58.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:45:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:58.592+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:45:59.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:59.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:59.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:59.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:59.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:59.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:45:59.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:45:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:45:59.167+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:00.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:59.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:00.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:59.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:00.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:59.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:00.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:45:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:45:59.632+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:00.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:59.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:00.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:59.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:00.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:59.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:00.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:45:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:45:59.552+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:00.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:00.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:00.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:00.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:00.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:00.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:00.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:00.194+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:01.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:00.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:01.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:00.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:01.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:00.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:01.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:00.594+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:01.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:00.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:01.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:00.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:01.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:00.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:01.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:00.674+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:01.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:01.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:01.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:01.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:01.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:01.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:01.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:01.169+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:01.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:01.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:01.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:01.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:01.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:01.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:01.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:01.680+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:01.971 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:01.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:01.971 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:01.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:01.971 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:01.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:01.972 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:01.635+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:02.233 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:02.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:02.234 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:02.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:02.234 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:02.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:02.234 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:02.205+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:02.518 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:46:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:46:02.233+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 36547 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:46:03.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:02.683+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:03.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:02.683+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:03.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:02.683+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:03.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:02.684+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:03.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:02.603+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:03.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:02.603+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:03.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:02.603+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:03.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:02.604+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:03.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:03.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:03.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:03.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:03.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:03.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:03.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:03.237+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:04.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:03.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:04.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:03.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:04.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:03.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:04.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:03.673+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:04.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:03.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:04.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:03.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:04.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:03.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:04.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:03.576+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:04.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:04.214+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:04.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:04.214+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:04.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:04.214+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:04.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:04.214+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:05.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:04.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:05.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:04.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:05.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:04.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:05.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:04.556+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:05.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:04.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:05.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:04.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:05.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:04.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:05.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:04.677+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:05.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:05.177+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:05.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:05.177+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:05.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:05.177+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:05.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:05.177+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:06.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:05.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:06.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:05.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:06.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:05.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:06.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:05.559+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:06.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:05.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:06.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:05.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:05.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:05.718+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:06.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:06.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:06.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:06.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:06.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:06.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:06.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:06.163+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:07.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:06.522+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:07.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:06.522+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:07.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:06.522+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:07.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:06.524+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:07.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:06.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:07.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:06.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:07.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:06.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:07.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:06.759+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:07.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:46:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:46:07.238+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 36637 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:46:07.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:07.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:07.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:07.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:07.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:07.160+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:08.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:07.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:08.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:07.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:08.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:07.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:08.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:07.570+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:07.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:07.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:07.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:07.712+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:08.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:08.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:08.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:08.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:08.163+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:09.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:08.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:09.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:08.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:09.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:08.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:09.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:08.592+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:09.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:08.729+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:09.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:08.729+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:09.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:08.729+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:09.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:08.730+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:09.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:09.203+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:09.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:09.203+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:09.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:09.203+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:09.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:09.204+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:10.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:09.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:10.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:09.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:10.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:09.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:10.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:09.580+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:10.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:09.772+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:10.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:09.772+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:10.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:09.772+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:10.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:09.773+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:10.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:10.218+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:10.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:10.218+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:10.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:10.218+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:10.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:10.219+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:11.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:10.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:11.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:10.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:11.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:10.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:11.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:10.734+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:11.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:10.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:11.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:10.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:10.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:10.583+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:11.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:11.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:11.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:11.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:11.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:11.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:11.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:11.177+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:12.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:11.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:12.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:11.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:12.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:11.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:12.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:11.616+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:12.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:11.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:12.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:11.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:12.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:11.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:12.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:11.762+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:12.479 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:46:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:46:12.243+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 36730 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:46:12.479 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:12.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:12.479 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:12.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:12.479 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:12.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:12.479 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:12.183+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:12.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:12.654+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:12.768 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:12.655+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:12.768 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:12.655+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:12.768 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:12.655+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:13.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:12.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:13.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:12.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:13.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:12.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:13.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:12.768+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:13.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:13.232+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:13.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:13.232+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:13.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:13.232+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:13.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:13.232+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:14.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:13.606+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:14.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:13.606+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:14.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:13.606+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:14.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:13.607+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:14.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:13.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:14.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:13.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:14.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:13.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:14.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:13.748+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:14.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:14.265+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:14.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:14.265+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:14.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:14.265+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:14.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:14.265+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:15.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:14.656+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:15.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:14.656+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:15.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:14.656+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:15.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:14.657+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:15.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:14.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:15.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:14.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:15.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:14.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:15.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:14.765+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:15.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:15.237+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:15.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:15.237+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:15.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:15.237+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:15.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:15.237+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:16.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:15.680+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:16.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:15.680+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:16.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:15.680+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:16.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:15.681+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:16.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:15.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:16.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:15.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:16.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:15.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:16.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:15.741+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:16.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:16.268+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:16.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:16.268+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:16.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:16.268+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:16.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:16.268+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:17.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:16.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:17.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:16.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:17.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:16.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:17.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:16.711+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:17.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:16.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:17.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:16.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:17.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:16.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:17.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:16.699+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:17.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:46:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:46:17.248+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 36819 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:46:17.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:17.287+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:17.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:17.287+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:17.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:17.287+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:17.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:17.287+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:18.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:17.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:18.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:17.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:18.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:17.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:18.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:17.759+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:18.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:17.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:18.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:17.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:18.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:17.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:18.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:17.713+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:18.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:18.249+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:18.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:18.249+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:18.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:18.249+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:18.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:18.249+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:19.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:18.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:19.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:18.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:19.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:18.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:19.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:18.786+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:19.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:18.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:19.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:18.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:19.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:18.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:19.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:18.761+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:19.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:19.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:19.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:19.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:19.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:19.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:19.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:19.285+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:20.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:19.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:20.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:19.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:20.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:19.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:20.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:19.768+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:20.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:19.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:20.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:19.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:20.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:19.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:20.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:19.740+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:20.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:20.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:20.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:20.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:20.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:20.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:20.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:20.237+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:21.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:20.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:21.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:20.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:21.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:20.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:21.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:20.772+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:21.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:20.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:20.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:20.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:20.693+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:21.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:21.219+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:21.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:21.219+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:21.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:21.219+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:21.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:21.220+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:22.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:21.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:22.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:21.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:22.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:21.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:22.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:21.804+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:22.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:21.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:22.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:21.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:22.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:21.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:22.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:21.715+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:22.518 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:46:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:46:22.252+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 36910 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:46:22.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:22.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:22.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:22.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:22.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:22.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:22.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:22.198+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:23.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:22.775+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:23.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:22.775+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:23.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:22.775+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:23.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:22.776+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:22.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:22.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:22.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:22.688+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:23.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:23.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:23.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:23.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:23.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:23.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:23.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:23.211+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:24.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:23.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:24.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:23.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:24.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:23.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:24.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:23.765+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:24.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:23.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:24.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:23.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:24.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:23.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:24.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:23.714+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:24.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:24.246+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:24.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:24.246+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:24.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:24.246+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:24.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:24.247+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:25.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:24.777+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:25.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:24.777+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:25.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:24.777+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:25.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:24.778+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:25.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:24.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:25.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:24.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:25.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:24.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:25.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:24.712+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:25.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:25.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:25.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:25.235+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:26.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:25.746+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:26.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:25.746+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:26.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:25.746+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:26.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:25.747+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:26.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:25.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:26.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:25.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:25.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:25.709+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:26.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:26.195+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:26.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:26.195+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:26.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:26.195+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:26.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:26.195+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:27.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:26.757+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:27.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:26.757+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:27.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:26.757+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:27.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:26.758+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:27.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:26.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:27.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:26.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:27.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:26.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:27.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:26.679+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:27.518 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:46:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:46:27.257+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 36999 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:46:27.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:27.228+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:27.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:27.228+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:27.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:27.228+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:27.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:27.229+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:28.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:27.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:28.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:27.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:28.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:27.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:28.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:27.779+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:28.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:27.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:28.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:27.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:28.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:27.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:28.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:27.718+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:28.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:28.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:28.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:28.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:28.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:28.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:28.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:28.210+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:29.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:28.677+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:29.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:28.677+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:29.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:28.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:29.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:28.678+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:29.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:28.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:29.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:28.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:29.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:28.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:28.813+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:29.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:29.205+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:29.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:29.205+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:29.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:29.205+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:29.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:29.206+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:30.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:29.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:30.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:29.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:30.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:29.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:30.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:29.686+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:30.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:29.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:30.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:29.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:29.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:29.794+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:30.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:30.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:30.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:30.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:30.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:30.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:30.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:30.166+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:31.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:30.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:31.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:30.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:31.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:30.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:31.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:30.781+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:31.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:30.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:31.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:30.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:31.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:30.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:31.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:30.715+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:31.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:31.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:31.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:31.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:31.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:31.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:31.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:31.209+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:32.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:31.769+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:32.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:31.769+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:32.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:31.769+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:32.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:31.771+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:32.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:31.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:32.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:31.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:32.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:31.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:32.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:31.677+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:32.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:46:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:46:32.262+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 37090 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:46:32.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:32.243+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:32.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:32.243+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:32.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:32.243+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:32.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:32.244+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:33.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:32.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:33.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:32.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:33.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:32.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:33.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:32.781+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:33.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:32.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:33.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:32.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:33.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:32.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:33.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:32.674+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:33.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:33.212+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:33.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:33.212+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:33.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:33.212+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:33.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:33.212+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:34.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:33.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:34.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:33.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:34.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:33.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:34.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:33.816+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:34.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:33.704+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:34.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:33.704+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:34.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:33.704+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:34.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:33.705+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:34.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:34.230+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:34.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:34.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:34.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:34.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:34.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:34.231+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:35.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:34.704+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:35.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:34.704+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:35.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:34.704+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:35.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:34.705+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:35.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:34.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:35.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:34.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:35.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:34.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:35.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:34.830+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:35.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:35.234+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:35.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:35.234+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:35.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:35.234+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:35.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:35.235+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:36.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:35.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:36.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:35.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:36.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:35.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:36.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:35.669+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:36.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:35.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:36.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:35.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:36.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:35.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:35.875+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:36.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:36.246+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:36.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:36.246+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:36.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:36.246+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:36.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:36.246+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:37.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:36.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:37.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:36.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:37.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:36.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:37.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:36.662+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:37.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:36.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:37.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:36.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:37.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:36.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:37.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:36.843+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:37.518 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:46:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:46:37.267+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 37181 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:46:37.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:37.278+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:37.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:37.278+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:37.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:37.278+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:37.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:37.279+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:38.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:37.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:38.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:37.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:38.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:37.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:38.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:37.624+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:38.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:37.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:38.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:37.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:38.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:37.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:38.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:37.862+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:38.635 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:38.317+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:38.635 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:38.317+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:38.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:38.317+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:38.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:38.318+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:39.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:38.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:39.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:38.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:39.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:38.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:39.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:38.636+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:39.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:38.871+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:39.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:38.871+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:39.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:38.871+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:38.871+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:39.645 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:39.311+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:39.645 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:39.311+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:39.645 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:39.311+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:39.645 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:39.311+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:39.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:39.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:39.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:39.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:39.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:39.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:39.911 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:39.645+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:40.252 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:39.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:40.252 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:39.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:40.252 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:39.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:40.252 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:39.911+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:40.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:40.306+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:40.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:40.306+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:40.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:40.306+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:40.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:40.306+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:40.876 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:40.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:40.876 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:40.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:40.876 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:40.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:40.877 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:40.616+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:41.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:40.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:41.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:40.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:41.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:40.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:41.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:40.877+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:41.639 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:41.292+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:41.639 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:41.292+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:41.640 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:41.292+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:41.640 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:41.292+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:41.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:41.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:41.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:41.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:41.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:41.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:41.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:41.640+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:42.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:41.891+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:42.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:41.891+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:42.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:41.891+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:42.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:41.892+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:42.518 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:46:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:46:42.272+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 37272 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:46:42.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:42.255+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:42.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:42.255+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:42.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:42.255+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:42.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:42.256+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:43.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:42.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:43.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:42.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:43.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:42.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:43.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:42.636+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:43.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:42.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:43.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:42.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:43.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:42.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:43.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:42.879+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:43.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:43.227+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:43.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:43.227+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:43.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:43.227+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:43.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:43.227+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:43.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:43.594+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:43.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:43.594+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:43.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:43.594+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:43.832 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:43.595+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:43.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:43.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:43.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:43.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:43.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:43.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:43.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:43.830+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:44.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:44.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:44.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:44.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:44.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:44.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:44.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:44.236+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:44.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:44.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:44.979 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:44.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:44.979 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:44.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:44.979 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:44.782+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:44.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:44.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:44.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:44.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:44.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:44.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:44.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:44.630+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:45.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:45.237+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:45.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:45.237+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:45.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:45.237+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:45.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:45.237+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:46.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:45.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:46.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:45.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:46.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:45.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:46.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:45.755+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:46.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:45.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:46.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:45.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:46.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:45.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:46.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:45.589+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:46.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:46.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:46.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:46.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:46.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:46.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:46.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:46.205+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:47.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:46.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:47.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:46.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:47.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:46.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:47.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:46.574+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:47.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:46.779+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:47.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:46.779+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:47.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:46.779+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:46.780+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:47.518 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:46:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:46:47.276+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 37363 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:46:47.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:47.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:47.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:47.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:47.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:47.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:47.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:47.174+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:48.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:47.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:48.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:47.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:48.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:47.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:48.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:47.736+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:48.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:47.545+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:48.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:47.545+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:48.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:47.545+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:48.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:47.545+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:48.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:48.206+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:48.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:48.206+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:48.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:48.206+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:48.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:48.207+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:49.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:48.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:49.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:48.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:49.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:48.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:49.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:48.699+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:49.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:48.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:48.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:48.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:48.592+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:49.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:49.255+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:49.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:49.255+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:49.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:49.255+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:49.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:49.256+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:50.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:49.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:50.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:49.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:50.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:49.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:50.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:49.697+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:50.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:49.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:50.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:49.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:50.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:49.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:50.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:49.588+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:50.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:50.243+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:50.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:50.243+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:50.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:50.243+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:50.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:50.243+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:51.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:50.688+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:51.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:50.688+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:51.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:50.688+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:51.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:50.688+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:51.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:50.555+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:51.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:50.555+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:51.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:50.555+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:51.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:50.555+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:51.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:51.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:51.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:51.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:51.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:51.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:51.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:51.213+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:52.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:51.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:52.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:51.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:52.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:51.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:52.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:51.696+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:52.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:51.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:52.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:51.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:52.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:51.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:52.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:51.571+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:52.518 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:46:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:46:52.281+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 37454 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:46:52.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:52.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:52.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:52.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:52.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:52.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:52.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:52.183+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:53.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:52.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:53.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:52.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:53.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:52.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:53.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:52.666+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:53.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:52.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:53.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:52.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:53.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:52.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:53.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:52.600+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:53.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:53.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:53.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:53.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:53.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:53.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:53.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:53.145+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:54.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:53.558+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:54.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:53.558+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:54.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:53.558+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:54.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:53.559+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:54.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:53.681+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:54.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:53.681+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:54.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:53.681+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:54.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:53.682+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:54.511 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:54.140+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:54.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:54.140+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:54.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:54.140+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:54.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:54.141+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:54.768 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:54.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:54.768 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:54.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:54.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:54.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:54.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:54.649+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:54.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:54.511+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:54.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:54.511+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:54.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:54.511+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:54.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:54.512+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:55.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:55.120+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:55.120+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:55.120+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:55.121+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:56.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:55.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:56.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:55.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:56.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:55.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:56.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:55.664+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:56.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:55.544+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:56.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:55.544+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:56.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:55.544+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:56.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:55.545+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:56.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:56.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:56.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:56.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:56.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:56.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:56.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:56.082+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:57.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:56.700+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:57.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:56.700+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:57.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:56.700+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:57.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:56.700+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:57.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:56.521+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:57.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:56.521+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:57.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:56.521+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:57.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:56.522+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:57.409 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:46:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:46:57.286+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 37545 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:46:57.409 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:57.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:57.409 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:57.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:57.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:57.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:57.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:57.063+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:57.768 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:57.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:57.768 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:57.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:57.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:57.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:57.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:57.652+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:57.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:57.527+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:57.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:57.527+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:57.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:57.527+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:57.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:57.528+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:58.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:58.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:58.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:58.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:58.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:58.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:58.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:58.046+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:59.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:58.571+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:59.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:58.571+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:59.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:58.571+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:59.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:58.572+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:46:59.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:58.647+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:59.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:58.647+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:59.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:58.647+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:58.647+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:59.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:59.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:59.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:59.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:59.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:59.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:46:59.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:59.037+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:59.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:59.605+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:59.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:59.605+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:59.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:59.605+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:46:59.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:46:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:46:59.605+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:46:59.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:59.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:46:59.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:59.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:00.000 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:59.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:00.000 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:46:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:46:59.569+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:00.253 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:46:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:59.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:00.254 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:59.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:00.254 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:59.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:00.254 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:46:59.999+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:01.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:00.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:01.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:00.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:01.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:00.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:01.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:00.603+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:01.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:00.554+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:01.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:00.554+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:01.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:00.554+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:01.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:00.554+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:01.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:01.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:01.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:01.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:01.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:01.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:01.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:01.009+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:01.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:01.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:01.997 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:01.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:01.997 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:01.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:01.997 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:01.568+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:01.997 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:01.596+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:01.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:01.596+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:01.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:01.596+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:01.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:01.597+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:02.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:01.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:02.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:01.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:02.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:01.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:02.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:01.997+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:02.581 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:47:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:47:02.291+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 37637 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:47:02.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:02.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:02.981 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:02.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:02.981 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:02.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:02.981 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:02.581+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:02.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:02.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:02.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:02.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:02.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:02.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:02.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:02.640+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:03.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:02.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:03.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:02.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:03.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:02.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:03.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:02.980+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:03.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:03.659+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:03.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:03.659+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:03.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:03.659+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:03.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:03.660+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:03.990 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:03.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:03.990 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:03.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:03.990 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:03.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:03.990 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:03.582+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:04.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:03.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:04.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:03.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:04.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:03.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:04.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:03.989+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:05.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:04.547+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:05.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:04.547+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:05.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:04.547+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:05.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:04.547+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:05.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:04.710+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:05.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:04.710+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:05.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:04.710+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:05.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:04.711+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:05.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:05.005+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:05.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:05.005+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:05.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:05.005+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:05.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:05.006+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:06.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:05.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:06.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:05.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:06.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:05.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:06.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:05.559+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:06.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:05.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:06.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:05.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:06.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:05.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:06.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:05.726+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:06.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:06.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:06.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:06.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:06.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:06.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:06.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:06.008+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:06.748 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:06.596+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:06.748 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:06.596+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:06.748 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:06.596+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:06.749 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:06.597+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:07.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:06.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:07.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:06.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:07.014 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:06.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:07.014 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:06.748+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:07.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:07.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:07.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:07.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:07.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:07.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:07.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:07.013+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:07.587 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:47:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:47:07.295+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 37728 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:47:07.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:07.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:07.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:07.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:07.969 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:07.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:07.969 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:07.718+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:07.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:07.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:07.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:07.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:07.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:07.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:07.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:07.588+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:08.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:07.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:08.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:07.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:08.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:07.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:08.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:07.969+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:08.943 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:08.682+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:08.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:08.682+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:08.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:08.682+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:08.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:08.683+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:08.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:08.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:08.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:08.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:08.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:08.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:08.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:08.605+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:09.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:08.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:09.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:08.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:09.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:08.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:09.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:08.943+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:09.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:09.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:09.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:09.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:09.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:09.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:09.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:09.709+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:09.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:09.646+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:09.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:09.646+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:09.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:09.646+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:09.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:09.647+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:10.254 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:09.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:10.254 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:09.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:10.254 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:09.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:10.254 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:09.988+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:11.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:10.646+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:11.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:10.646+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:11.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:10.646+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:11.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:10.647+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:11.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:10.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:11.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:10.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:11.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:10.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:11.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:10.719+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:11.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:11.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:11.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:11.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:11.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:11.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:11.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:11.016+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:11.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:11.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:11.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:11.622+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:11.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:11.622+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:11.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:11.623+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:11.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:11.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:11.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:11.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:11.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:11.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:11.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:11.719+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:12.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:11.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:12.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:11.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:12.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:11.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:12.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:11.978+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:12.626 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:47:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:47:12.300+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 37819 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:47:13.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:12.662+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:13.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:12.662+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:13.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:12.662+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:13.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:12.662+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:13.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:12.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:13.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:12.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:13.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:12.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:13.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:12.701+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:13.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:13.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:13.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:13.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:13.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:13.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:13.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:13.020+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:14.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:13.680+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:14.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:13.680+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:14.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:13.680+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:14.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:13.681+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:14.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:13.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:14.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:13.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:14.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:13.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:14.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:13.686+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:14.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:14.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:14.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:14.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:14.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:14.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:14.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:14.054+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:15.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:14.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:15.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:14.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:15.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:14.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:15.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:14.726+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:15.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:14.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:15.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:14.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:15.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:14.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:15.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:14.689+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:15.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:15.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:15.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:15.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:15.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:15.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:15.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:15.025+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:15.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:15.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:15.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:15.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:15.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:15.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:15.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:15.680+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:15.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:15.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:15.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:15.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:15.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:15.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:15.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:15.703+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:16.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:15.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:16.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:15.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:16.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:15.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:16.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:15.995+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:17.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:16.731+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:17.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:16.731+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:17.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:16.731+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:17.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:16.732+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:17.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:16.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:17.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:16.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:17.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:16.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:17.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:16.716+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:17.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:17.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:17.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:17.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:17.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:17.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:17.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:17.032+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:17.725 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:47:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:47:17.306+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 37910 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:47:18.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:17.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:18.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:17.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:18.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:17.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:18.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:17.726+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:17.731+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:17.731+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:17.731+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:17.732+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:18.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:18.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:18.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:18.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:18.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:18.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:18.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:18.066+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:19.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:18.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:19.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:18.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:19.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:18.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:19.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:18.726+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:19.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:18.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:19.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:18.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:19.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:18.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:19.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:18.771+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:19.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:19.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:19.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:19.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:19.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:19.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:19.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:19.063+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:20.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:19.734+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:20.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:19.734+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:20.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:19.734+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:20.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:19.736+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:20.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:19.743+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:20.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:19.743+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:20.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:19.743+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:20.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:19.744+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:20.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:20.086+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:20.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:20.086+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:20.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:20.086+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:20.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:20.087+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:21.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:20.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:21.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:20.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:21.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:20.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:21.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:20.763+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:21.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:20.792+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:21.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:20.792+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:21.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:20.792+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:20.793+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:21.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:21.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:21.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:21.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:21.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:21.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:21.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:21.056+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:22.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:22.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:22.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:22.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:22.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:22.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:22.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:22.020+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:22.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:21.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:21.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:21.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:21.836+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:21.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:21.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:21.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:21.807+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:22.768 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:47:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:47:22.310+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 38002 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:47:23.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:22.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:23.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:22.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:23.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:22.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:23.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:22.979+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:23.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:22.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:23.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:22.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:23.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:22.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:22.857+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:22.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:22.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:22.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:22.804+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:23.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:23.789+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:23.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:23.789+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:23.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:23.789+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:23.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:23.790+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:24.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:23.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:24.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:23.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:24.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:23.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:24.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:23.905+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:24.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:24.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:24.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:24.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:24.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:24.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:24.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:24.013+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:25.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:24.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:25.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:24.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:25.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:24.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:25.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:24.871+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:25.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:24.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:25.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:24.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:25.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:24.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:25.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:24.764+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:25.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:25.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:25.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:25.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:25.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:25.043+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:26.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:25.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:26.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:25.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:26.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:25.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:26.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:25.889+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:26.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:25.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:26.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:25.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:25.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:25.717+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:26.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:26.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:26.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:26.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:26.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:26.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:26.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:26.035+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:26.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:26.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:26.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:26.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:26.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:26.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:26.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:26.866+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:26.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:26.716+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:26.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:26.716+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:26.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:26.716+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:26.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:26.717+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:27.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:26.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:27.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:26.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:27.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:26.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:27.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:26.988+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:27.705 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:47:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:47:27.315+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 38093 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:47:28.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:27.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:28.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:27.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:28.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:27.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:28.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:27.889+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:28.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:27.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:28.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:27.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:28.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:27.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:28.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:27.706+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:28.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:28.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:28.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:28.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:28.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:28.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:28.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:28.027+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:29.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:28.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:29.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:28.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:29.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:28.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:29.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:28.873+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:29.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:28.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:29.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:28.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:29.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:28.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:29.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:28.695+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:29.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:29.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:29.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:29.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:29.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:29.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:29.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:29.077+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:30.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:29.882+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:30.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:29.882+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:30.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:29.882+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:30.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:29.883+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:30.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:29.680+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:30.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:29.680+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:30.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:29.680+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:30.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:29.681+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:30.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:30.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:30.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:30.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:30.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:30.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:30.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:30.090+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:30.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:30.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:30.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:30.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:30.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:30.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:30.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:30.851+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:30.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:30.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:30.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:30.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:30.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:30.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:30.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:30.733+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:31.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:31.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:31.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:31.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:31.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:31.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:31.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:31.123+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:32.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:31.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:32.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:31.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:32.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:31.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:32.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:31.877+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:32.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:31.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:32.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:31.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:32.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:31.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:32.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:31.695+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:32.441 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:47:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:47:32.321+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 38184 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:47:32.442 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:32.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:32.442 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:32.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:32.442 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:32.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:32.442 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:32.137+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:32.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:32.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:32.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:32.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:32.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:32.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:32.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:32.663+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:33.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:33.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:33.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:33.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:33.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:33.096+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:33.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:32.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:33.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:32.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:33.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:32.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:33.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:32.879+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:33.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:33.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:33.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:33.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:33.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:33.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:33.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:33.700+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:34.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:33.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:34.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:33.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:34.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:33.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:34.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:33.874+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:34.131 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:34.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:34.131 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:34.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:34.131 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:34.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:34.131 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:34.077+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:35.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:34.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:35.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:34.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:35.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:34.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:35.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:34.685+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:35.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:34.912+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:35.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:34.912+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:35.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:34.912+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:35.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:34.913+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:35.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:35.092+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:35.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:35.092+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:35.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:35.092+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:35.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:35.093+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:36.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:35.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:36.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:35.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:36.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:35.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:36.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:35.868+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:36.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:35.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:36.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:35.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:36.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:35.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:36.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:35.692+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:36.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:36.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:36.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:36.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:36.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:36.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:36.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:36.072+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:37.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:36.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:37.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:36.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:37.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:36.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:37.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:36.710+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:37.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:36.886+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:37.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:36.886+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:37.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:36.886+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:37.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:36.887+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:37.345 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:47:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:47:37.326+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 38275 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:47:37.346 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:37.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:37.346 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:37.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:37.346 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:37.095+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:37.346 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:37.095+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:37.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:37.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:37.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:37.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:37.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:37.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:37.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:37.672+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:38.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:37.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:38.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:37.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:38.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:37.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:38.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:37.922+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:38.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:38.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:38.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:38.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:38.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:38.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:38.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:38.138+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:38.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:38.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:38.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:38.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:38.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:38.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:38.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:38.625+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:39.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:38.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:39.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:38.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:39.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:38.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:39.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:38.915+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:39.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:39.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:39.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:39.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:39.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:39.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:39.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:39.097+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:39.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:39.610+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:39.911 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:39.610+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:39.911 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:39.610+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:39.911 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:39.611+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:40.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:39.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:40.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:39.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:40.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:39.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:40.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:39.911+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:40.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:40.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:40.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:40.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:40.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:40.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:40.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:40.065+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:41.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:40.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:41.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:40.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:41.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:40.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:41.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:40.899+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:41.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:40.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:41.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:40.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:41.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:40.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:41.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:40.661+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:41.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:41.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:41.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:41.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:41.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:41.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:41.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:41.066+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:42.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:41.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:42.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:41.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:42.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:41.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:42.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:41.942+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:42.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:41.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:42.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:41.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:42.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:41.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:42.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:41.700+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:42.331 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:42.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:42.331 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:42.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:42.332 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:42.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:42.332 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:42.078+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:42.727 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:47:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:47:42.331+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 38369 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:47:43.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:42.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:43.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:42.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:43.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:42.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:43.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:42.728+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:43.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:42.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:43.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:42.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:43.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:42.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:43.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:42.942+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:43.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:43.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:43.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:43.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:43.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:43.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:43.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:43.074+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:43.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:43.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:43.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:43.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:43.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:43.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:43.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:43.716+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:44.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:43.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:44.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:43.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:44.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:43.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:44.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:43.936+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:44.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:44.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:44.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:44.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:44.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:44.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:44.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:44.079+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:44.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:44.666+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:44.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:44.666+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:44.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:44.666+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:44.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:44.667+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:45.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:45.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:45.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:45.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:45.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:45.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:45.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:45.110+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:45.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:44.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:45.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:44.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:45.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:44.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:45.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:44.940+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:46.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:45.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:46.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:45.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:46.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:45.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:46.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:45.936+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:46.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:45.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:46.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:45.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:46.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:45.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:46.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:45.697+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:46.418 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:46.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:46.418 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:46.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:46.418 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:46.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:46.418 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:46.157+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:46.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:46.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:46.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:46.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:46.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:46.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:46.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:46.661+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:47.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:46.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:47.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:46.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:47.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:46.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:47.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:46.976+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:47.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:47.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:47.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:47.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:47.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:47.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:47.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:47.136+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:47.628 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:47:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:47:47.336+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 38460 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:47:47.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:47.628+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:47.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:47.628+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:47.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:47.628+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:47.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:47.629+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:48.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:47.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:48.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:47.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:48.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:47.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:48.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:47.989+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:48.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:48.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:48.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:48.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:48.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:48.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:48.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:48.170+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:49.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:48.651+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:49.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:48.651+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:49.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:48.651+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:49.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:48.652+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:49.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:49.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:49.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:49.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:49.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:49.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:49.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:49.202+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:49.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:49.032+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:49.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:49.032+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:49.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:49.032+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:49.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:49.033+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:50.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:49.665+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:50.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:49.665+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:50.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:49.665+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:50.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:49.665+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:50.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:50.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:50.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:50.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:50.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:50.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:50.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:50.037+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:50.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:50.164+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:50.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:50.164+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:50.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:50.164+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:50.165+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:51.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:50.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:51.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:50.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:51.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:50.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:51.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:50.659+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:51.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:51.043+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:51.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:51.043+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:51.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:51.043+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:51.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:51.044+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:51.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:51.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:51.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:51.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:51.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:51.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:51.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:51.136+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:52.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:51.620+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:52.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:51.620+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:52.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:51.620+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:52.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:51.620+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:52.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:52.086+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:52.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:52.086+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:52.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:52.086+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:52.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:52.087+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:52.342 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:52.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:52.342 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:52.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:52.343 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:52.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:52.343 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:52.174+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:52.620 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:47:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:47:52.341+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 38551 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:47:53.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:52.621+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:53.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:52.621+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:53.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:52.621+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:53.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:52.622+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:53.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:53.118+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:53.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:53.118+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:53.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:53.118+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:53.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:53.119+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:53.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:53.188+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:53.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:53.188+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:53.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:53.188+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:53.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:53.189+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:54.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:53.637+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:54.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:53.637+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:54.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:53.637+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:54.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:53.638+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:54.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:54.150+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:54.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:54.150+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:54.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:54.150+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:54.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:54.150+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:54.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:54.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:54.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:54.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:54.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:54.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:54.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:54.141+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:55.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:54.607+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:55.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:54.607+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:55.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:54.607+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:55.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:54.608+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:55.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:55.182+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:55.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:55.182+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:55.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:55.182+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:55.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:55.183+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:55.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:55.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:55.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:55.144+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:56.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:55.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:56.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:55.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:56.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:55.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:56.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:55.591+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:56.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:56.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:56.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:56.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:56.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:56.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:56.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:56.192+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:56.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:56.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:56.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:56.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:56.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:56.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:56.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:56.163+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:57.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:56.557+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:57.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:56.557+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:57.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:56.557+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:57.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:56.557+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:57.470 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:47:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:47:57.346+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 38642 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:47:57.470 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:57.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:57.470 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:57.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:57.470 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:57.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:57.470 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:57.159+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:57.471 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:57.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:57.471 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:57.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:57.471 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:57.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:57.471 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:57.186+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:57.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:57.522+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:57.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:57.522+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:57.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:57.522+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:57.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:57.523+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:58.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:58.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:58.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:58.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:58.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:58.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:58.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:58.114+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:58.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:58.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:58.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:58.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:58.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:58.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:58.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:58.169+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:59.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:58.571+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:59.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:58.571+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:59.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:58.571+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:47:59.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:58.571+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:47:59.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:59.097+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:59.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:59.097+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:59.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:59.097+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:47:59.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:47:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:47:59.097+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:47:59.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:59.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:59.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:59.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:59.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:59.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:47:59.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:47:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:47:59.175+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:00.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:59.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:00.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:59.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:00.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:59.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:00.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:47:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:47:59.570+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:00.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:00.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:00.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:00.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:00.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:00.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:00.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:00.054+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:00.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:00.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:00.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:00.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:00.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:00.143+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:01.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:00.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:01.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:00.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:01.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:00.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:01.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:00.593+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:01.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:01.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:01.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:01.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:01.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:01.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:01.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:01.094+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:01.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:01.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:01.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:01.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:01.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:01.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:01.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:01.172+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:02.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:01.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:02.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:01.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:02.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:01.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:02.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:01.574+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:02.351 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:02.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:02.352 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:02.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:02.352 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:02.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:02.352 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:02.197+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:02.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:02.100+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:02.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:02.100+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:02.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:02.100+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:02.353 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:02.100+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:02.615 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:48:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:48:02.351+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 38733 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:48:03.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:02.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:03.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:02.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:03.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:02.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:03.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:02.616+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:03.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:03.142+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:03.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:03.142+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:03.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:03.142+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:03.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:03.143+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:03.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:03.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:03.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:03.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:03.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:03.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:03.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:03.231+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:04.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:03.577+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:04.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:03.577+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:04.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:03.577+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:04.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:03.578+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:04.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:04.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:04.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:04.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:04.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:04.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:04.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:04.134+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:04.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:04.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:04.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:04.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:04.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:04.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:04.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:04.202+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:05.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:04.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:05.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:04.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:05.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:04.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:05.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:04.616+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:05.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:05.131+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:05.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:05.131+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:05.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:05.131+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:05.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:05.132+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:05.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:05.179+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:05.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:05.179+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:05.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:05.179+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:05.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:05.180+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:06.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:05.605+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:06.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:05.605+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:06.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:05.605+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:06.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:05.606+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:06.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:06.166+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:06.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:06.166+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:06.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:06.166+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:06.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:06.167+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:06.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:06.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:06.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:06.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:06.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:06.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:06.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:06.175+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:07.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:06.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:07.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:06.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:07.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:06.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:07.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:06.570+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:07.480 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:48:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:48:07.357+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 38824 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:48:07.481 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:07.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:07.481 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:07.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:07.481 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:07.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:07.481 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:07.150+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:07.481 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:07.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:07.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:07.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:07.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:07.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:07.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:07.208+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:07.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:07.576+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:07.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:07.576+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:07.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:07.576+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:07.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:07.577+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:08.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:08.214+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:08.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:08.214+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:08.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:08.214+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:08.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:08.215+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:08.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:08.141+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:08.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:08.141+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:08.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:08.141+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:08.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:08.142+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:09.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:08.578+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:09.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:08.578+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:09.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:08.578+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:09.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:08.579+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:09.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:09.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:09.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:09.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:09.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:09.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:09.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:09.185+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:09.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:09.134+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:09.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:09.134+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:09.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:09.134+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:09.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:09.135+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:10.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:09.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:10.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:09.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:10.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:09.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:10.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:09.617+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:10.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:10.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:10.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:10.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:10.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:10.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:10.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:10.236+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:10.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:10.107+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:10.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:10.107+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:10.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:10.107+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:10.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:10.108+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:10.994 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:10.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:10.994 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:10.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:10.994 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:10.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:10.994 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:10.636+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:11.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:11.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:11.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:11.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:11.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:11.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:11.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:11.114+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:11.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:11.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:11.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:11.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:11.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:11.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:11.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:11.236+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:12.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:11.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:12.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:11.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:12.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:11.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:12.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:11.679+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:12.362 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:12.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:12.362 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:12.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:12.362 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:12.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:12.362 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:12.078+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:12.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:12.253+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:12.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:12.253+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:12.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:12.253+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:12.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:12.253+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:12.646 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:48:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:48:12.362+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 38915 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:48:13.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:12.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:13.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:12.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:13.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:12.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:13.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:12.646+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:13.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:13.060+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:13.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:13.060+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:13.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:13.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:13.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:13.061+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:13.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:13.234+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:13.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:13.234+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:13.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:13.234+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:13.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:13.234+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:14.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:13.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:14.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:13.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:14.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:13.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:14.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:13.607+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:14.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:14.050+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:14.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:14.050+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:14.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:14.050+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:14.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:14.051+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:14.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:14.259+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:14.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:14.259+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:14.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:14.259+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:14.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:14.259+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:15.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:14.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:15.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:14.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:15.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:14.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:15.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:14.604+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:15.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:15.063+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:15.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:15.063+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:15.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:15.063+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:15.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:15.063+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:15.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:15.219+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:15.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:15.219+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:15.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:15.219+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:15.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:15.219+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:16.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:15.577+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:16.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:15.577+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:16.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:15.577+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:16.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:15.577+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:16.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:16.260+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:16.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:16.260+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:16.414 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:16.260+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:16.414 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:16.260+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:16.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:16.084+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:16.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:16.084+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:16.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:16.084+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:16.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:16.085+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:16.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:16.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:16.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:16.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:16.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:16.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:16.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:16.604+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:17.366 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:17.082+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:17.367 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:17.082+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:17.367 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:17.082+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:17.367 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:17.083+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:17.367 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:17.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:17.367 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:17.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:17.367 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:17.231+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:17.368 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:17.232+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:17.649 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:48:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:48:17.366+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 39005 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:48:18.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:17.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:18.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:17.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:18.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:17.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:18.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:17.650+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:18.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:18.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:18.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:18.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:18.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:18.198+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:18.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:18.199+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:18.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:18.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:18.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:18.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:18.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:18.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:18.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:18.113+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:19.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:18.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:19.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:18.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:19.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:18.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:19.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:18.611+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:19.518 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:19.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:19.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:19.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:19.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:19.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:19.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:19.181+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:19.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:19.126+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:19.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:19.126+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:19.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:19.126+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:19.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:19.127+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:20.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:19.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:20.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:19.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:20.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:19.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:20.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:19.639+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:20.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:20.082+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:20.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:20.082+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:20.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:20.082+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:20.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:20.083+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:20.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:20.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:20.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:20.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:20.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:20.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:20.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:20.133+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:21.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:20.652+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:21.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:20.652+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:21.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:20.652+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:21.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:20.653+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:21.459 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:21.052+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:21.459 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:21.052+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:21.459 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:21.052+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:21.459 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:21.053+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:21.460 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:21.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:21.460 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:21.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:21.460 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:21.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:21.460 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:21.090+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:21.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:21.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:21.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:21.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:21.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:21.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:21.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:21.662+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:22.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:22.042+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:22.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:22.043+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:22.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:22.043+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:22.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:22.044+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:22.372 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:22.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:22.372 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:22.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:22.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:22.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:22.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:22.059+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:22.679 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:48:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:48:22.371+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 39097 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:48:23.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:22.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:23.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:22.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:23.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:22.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:23.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:22.680+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:23.282 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:23.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:23.282 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:23.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:23.282 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:23.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:23.282 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:23.072+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:23.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:23.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:23.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:23.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:23.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:23.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:23.283 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:23.028+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:24.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:23.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:24.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:23.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:24.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:23.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:24.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:23.714+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:24.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:24.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:24.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:24.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:24.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:24.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:24.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:24.005+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:24.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:24.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:24.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:24.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:24.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:24.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:24.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:24.035+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:25.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:24.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:25.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:24.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:25.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:24.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:25.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:24.699+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:25.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:25.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:25.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:25.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:25.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:25.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:25.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:25.052+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:25.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:25.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:25.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:25.050+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:26.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:25.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:26.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:25.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:26.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:25.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:25.659+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:26.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:26.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:26.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:26.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:26.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:26.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:26.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:26.049+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:26.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:26.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:26.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:26.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:26.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:26.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:26.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:26.019+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:26.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:26.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:26.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:26.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:26.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:26.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:26.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:26.662+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:27.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:26.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:27.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:26.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:27.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:26.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:27.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:26.987+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:27.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:27.021+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:27.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:27.021+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:27.021+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:27.022+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:27.697 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:48:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:48:27.376+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 39188 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:48:28.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:27.697+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:28.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:27.697+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:28.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:27.697+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:28.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:27.698+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:28.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:28.041+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:28.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:28.041+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:28.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:28.041+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:28.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:28.042+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:28.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:28.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:28.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:28.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:28.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:28.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:28.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:28.025+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:29.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:28.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:29.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:28.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:29.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:28.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:29.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:28.739+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:29.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:29.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:29.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:29.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:29.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:29.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:29.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:29.022+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:29.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:29.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:29.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:29.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:29.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:29.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:29.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:29.027+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:30.002 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:29.723+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:30.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:29.723+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:30.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:29.723+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:30.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:29.724+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:30.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:30.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:30.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:30.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:30.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:30.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:30.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:30.003+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:30.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:30.061+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:30.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:30.061+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:30.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:30.061+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:30.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:30.062+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:30.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:30.703+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:30.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:30.703+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:30.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:30.703+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:30.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:30.704+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:31.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:30.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:31.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:30.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:31.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:30.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:31.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:30.965+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:31.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:31.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:31.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:31.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:31.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:31.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:31.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:31.104+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:31.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:31.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:31.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:31.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:31.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:31.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:31.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:31.679+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:32.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:31.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:32.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:31.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:32.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:31.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:32.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:31.964+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:32.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:32.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:32.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:32.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:32.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:32.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:32.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:32.060+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:32.639 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:48:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:48:32.381+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 39279 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:48:33.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:32.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:33.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:32.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:33.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:32.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:33.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:32.640+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:33.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:33.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:33.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:33.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:33.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:33.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:33.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:33.006+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:33.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:33.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:33.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:33.024+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:33.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:33.666+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:33.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:33.666+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:33.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:33.666+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:33.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:33.666+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:33.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:33.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:33.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:33.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:33.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:33.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:34.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:33.990+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:34.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:33.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:34.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:33.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:34.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:33.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:34.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:33.990+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:34.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:34.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:34.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:34.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:34.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:34.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:34.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:34.686+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:35.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:35.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:35.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:35.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:35.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:35.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:35.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:35.037+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:35.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:34.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:35.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:34.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:35.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:34.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:35.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:34.978+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:35.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:35.724+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:35.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:35.724+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:35.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:35.724+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:35.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:35.725+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:36.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:35.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:36.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:35.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:36.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:35.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:35.979+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:36.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:36.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:36.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:36.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:36.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:36.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:36.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:36.017+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:36.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:36.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:36.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:36.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:36.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:36.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:36.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:36.754+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:37.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:36.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:37.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:36.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:37.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:36.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:37.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:36.951+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:37.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:37.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:37.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:37.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:37.020+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:37.747 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:48:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:48:37.386+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 39370 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:48:38.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:37.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:38.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:37.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:38.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:37.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:38.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:37.748+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:38.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:37.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:38.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:37.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:38.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:37.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:38.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:37.934+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:38.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:38.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:38.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:38.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:38.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:38.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:38.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:38.038+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:39.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:38.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:39.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:38.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:39.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:38.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:39.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:38.906+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:39.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:38.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:39.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:38.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:39.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:38.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:39.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:38.719+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:39.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:39.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:39.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:39.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:39.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:39.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:39.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:39.017+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:40.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:39.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:40.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:39.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:40.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:39.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:40.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:39.758+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:40.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:39.892+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:40.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:39.892+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:40.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:39.892+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:40.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:39.892+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:40.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:40.044+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:40.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:40.044+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:40.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:40.044+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:40.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:40.045+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:41.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:40.890+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:41.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:40.890+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:41.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:40.890+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:41.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:40.890+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:41.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:40.729+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:41.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:40.729+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:41.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:40.729+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:41.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:40.730+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:41.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:41.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:41.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:41.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:41.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:41.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:41.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:41.093+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:42.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:41.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:42.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:41.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:42.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:41.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:42.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:41.743+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:42.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:41.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:42.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:41.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:42.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:41.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:42.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:41.887+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:42.392 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:42.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:42.392 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:42.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:42.392 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:42.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:42.392 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:42.057+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:42.737 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:48:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:48:42.391+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 39462 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:48:43.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:42.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:43.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:42.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:43.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:42.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:43.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:42.739+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:43.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:42.843+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:43.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:42.843+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:43.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:42.843+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:43.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:42.843+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:43.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:43.084+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:43.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:43.085+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:43.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:43.085+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:43.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:43.085+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:44.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:43.798+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:44.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:43.798+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:44.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:43.798+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:44.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:43.798+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:44.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:43.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:44.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:43.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:44.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:43.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:44.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:43.742+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:44.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:44.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:44.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:44.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:44.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:44.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:44.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:44.039+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:45.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:44.840+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:45.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:44.840+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:45.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:44.840+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:45.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:44.841+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:45.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:44.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:45.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:44.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:45.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:44.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:45.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:44.707+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:45.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:45.024+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:45.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:45.024+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:45.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:45.024+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:45.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:45.025+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:45.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:45.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:45.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:45.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:45.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:45.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:45.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:45.713+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:45.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:45.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:45.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:45.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:45.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:45.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:45.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:45.846+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:46.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:45.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:46.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:45.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:46.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:45.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:46.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:45.984+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:47.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:46.682+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:47.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:46.682+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:47.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:46.682+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:47.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:46.682+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:47.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:46.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:47.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:46.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:47.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:46.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:47.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:46.833+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:47.397 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:47.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:47.397 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:47.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:47.397 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:47.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:47.397 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:47.026+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:47.727 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:48:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:48:47.397+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 39553 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:48:48.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:47.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:48.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:47.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:48.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:47.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:48.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:47.728+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:48.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:47.862+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:48.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:47.862+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:48.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:47.862+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:48.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:47.862+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:48.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:48.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:48.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:48.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:48.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:48.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:48.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:48.037+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:49.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:48.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:49.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:48.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:49.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:48.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:49.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:48.830+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:49.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:48.756+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:49.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:48.756+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:49.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:48.756+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:49.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:48.756+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:49.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:49.041+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:49.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:49.041+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:49.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:49.041+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:49.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:49.042+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:50.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:49.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:50.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:49.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:50.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:49.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:50.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:49.803+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:50.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:49.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:50.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:49.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:50.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:49.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:50.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:49.760+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:50.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:50.068+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:50.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:50.068+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:50.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:50.068+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:50.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:50.069+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:51.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:50.792+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:51.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:50.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:51.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:50.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:51.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:50.793+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:51.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:50.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:51.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:50.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:51.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:50.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:51.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:50.822+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:51.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:51.096+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:51.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:51.096+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:51.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:51.096+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:51.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:51.097+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:52.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:51.756+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:52.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:51.756+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:52.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:51.756+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:52.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:51.757+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:52.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:51.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:52.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:51.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:52.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:51.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:52.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:51.808+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:52.402 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:52.115+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:52.402 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:52.115+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:52.402 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:52.115+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:52.403 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:52.115+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:52.713 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:48:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:48:52.402+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 39644 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:48:53.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:52.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:53.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:52.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:53.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:52.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:53.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:52.714+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:53.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:52.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:53.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:52.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:53.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:52.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:53.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:52.771+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:53.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:53.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:53.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:53.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:53.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:53.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:53.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:53.119+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:54.002 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:53.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:54.002 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:53.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:54.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:53.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:54.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:53.797+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:54.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:53.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:54.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:53.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:54.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:53.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:54.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:53.706+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:54.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:54.083+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:54.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:54.083+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:54.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:54.083+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:54.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:54.084+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:55.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:54.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:55.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:54.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:55.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:54.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:55.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:54.753+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:55.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:54.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:55.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:54.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:55.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:54.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:55.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:54.813+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:55.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:55.127+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:55.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:55.127+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:55.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:55.127+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:55.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:55.127+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:56.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:55.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:56.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:55.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:56.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:55.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:56.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:55.845+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:56.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:55.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:56.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:55.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:56.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:55.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:56.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:55.777+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:56.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:56.166+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:56.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:56.166+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:56.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:56.166+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:56.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:56.168+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:57.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:56.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:57.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:56.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:57.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:56.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:57.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:56.750+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:57.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:56.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:57.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:56.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:57.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:56.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:57.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:56.814+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:57.430 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:48:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:48:57.407+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 39735 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:48:57.430 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:57.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:57.430 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:57.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:57.430 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:57.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:57.431 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:57.174+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:58.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:57.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:58.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:57.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:58.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:57.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:58.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:57.743+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:58.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:57.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:58.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:57.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:58.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:57.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:58.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:57.855+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:58.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:58.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:58.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:58.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:58.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:58.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:58.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:58.223+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:59.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:58.789+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:59.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:58.789+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:59.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:58.789+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:48:59.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:58.789+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:48:59.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:58.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:59.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:58.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:59.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:58.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:48:59.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:58.846+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:48:59.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:59.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:59.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:59.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:59.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:59.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:48:59.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:48:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:48:59.202+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:00.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:59.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:00.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:59.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:00.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:59.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:00.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:48:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:48:59.813+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:00.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:59.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:00.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:59.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:00.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:59.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:00.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:48:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:48:59.822+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:00.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:00.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:00.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:00.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:00.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:00.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:00.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:00.175+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:01.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:00.858+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:01.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:00.858+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:01.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:00.858+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:01.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:00.859+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:01.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:00.865+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:01.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:00.865+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:01.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:00.865+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:01.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:00.865+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:01.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:01.179+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:01.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:01.179+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:01.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:01.179+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:01.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:01.179+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:02.206 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:01.864+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:02.206 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:01.864+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:02.207 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:01.864+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:02.207 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:01.865+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:02.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:01.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:02.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:01.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:02.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:01.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:02.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:01.879+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:02.518 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:49:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:49:02.412+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 39826 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:49:02.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:02.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:02.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:02.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:02.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:02.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:02.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:02.206+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:03.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:02.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:03.218 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:02.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:03.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:02.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:03.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:02.855+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:03.219 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:02.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:03.219 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:02.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:03.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:02.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:03.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:02.870+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:03.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:03.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:03.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:03.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:03.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:03.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:03.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:03.236+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:04.244 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:03.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:04.244 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:03.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:04.245 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:03.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:04.245 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:03.860+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:04.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:03.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:04.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:03.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:04.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:03.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:04.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:03.858+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:04.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:04.244+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:04.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:04.244+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:04.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:04.244+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:04.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:04.245+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:05.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:04.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:05.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:04.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:05.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:04.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:05.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:04.845+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:05.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:04.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:05.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:04.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:05.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:04.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:05.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:04.855+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:05.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:05.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:05.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:05.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:05.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:05.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:05.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:05.200+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:06.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:05.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:06.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:05.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:06.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:05.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:06.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:05.898+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:06.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:05.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:06.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:05.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:06.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:05.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:06.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:05.886+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:06.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:06.186+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:06.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:06.186+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:06.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:06.186+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:06.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:06.187+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:07.198 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:06.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:07.198 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:06.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:07.199 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:06.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:07.199 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:06.845+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:07.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:06.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:07.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:06.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:07.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:06.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:07.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:06.859+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:07.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:49:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:49:07.417+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 39917 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:49:07.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:07.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:07.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:07.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:07.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:07.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:07.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:07.199+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:08.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:07.837+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:08.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:07.837+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:08.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:07.837+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:08.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:07.838+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:08.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:07.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:08.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:07.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:08.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:07.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:08.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:07.845+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:08.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:08.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:08.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:08.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:08.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:08.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:08.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:08.210+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:09.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:08.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:09.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:08.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:09.242 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:08.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:09.242 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:08.850+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:09.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:08.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:09.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:08.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:09.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:08.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:09.242 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:08.813+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:09.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:09.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:09.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:09.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:09.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:09.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:09.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:09.242+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:10.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:09.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:10.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:09.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:10.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:09.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:10.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:09.860+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:10.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:09.804+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:10.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:09.804+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:10.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:09.804+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:10.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:09.804+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:10.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:10.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:10.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:10.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:10.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:10.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:10.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:10.265+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:11.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:10.895+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:11.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:10.895+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:11.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:10.895+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:11.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:10.895+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:11.221 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:10.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:11.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:10.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:11.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:10.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:11.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:10.803+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:11.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:11.220+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:11.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:11.220+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:11.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:11.220+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:11.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:11.221+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:12.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:11.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:12.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:11.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:12.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:11.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:12.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:11.894+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:12.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:11.755+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:12.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:11.755+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:12.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:11.755+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:12.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:11.755+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:12.518 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:49:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:49:12.422+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 40008 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:49:12.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:12.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:12.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:12.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:12.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:12.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:12.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:12.227+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:12.930 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:12.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:12.930 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:12.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:12.930 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:12.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:12.930 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:12.736+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:13.232 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:12.930+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:13.232 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:12.930+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:13.233 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:12.930+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:13.233 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:12.930+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:13.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:13.232+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:13.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:13.232+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:13.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:13.232+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:13.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:13.233+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:13.924 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:13.724+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:13.924 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:13.724+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:13.924 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:13.724+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:13.924 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:13.725+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:14.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:13.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:14.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:13.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:14.242 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:13.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:14.242 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:13.925+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:14.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:14.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:14.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:14.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:14.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:14.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:14.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:14.242+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:15.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:14.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:15.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:14.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:15.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:14.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:15.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:14.936+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:15.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:14.705+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:15.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:14.705+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:15.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:14.705+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:15.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:14.705+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:15.506 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:15.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:15.506 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:15.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:15.506 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:15.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:15.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:15.194+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:15.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:15.742+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:15.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:15.742+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:15.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:15.742+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:15.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:15.742+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:16.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:15.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:16.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:15.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:16.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:15.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:16.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:15.905+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:16.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:16.187+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:16.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:16.187+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:16.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:16.187+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:16.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:16.188+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:16.993 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:16.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:16.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:16.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:16.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:16.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:16.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:16.732+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:16.994 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:16.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:16.994 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:16.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:16.994 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:16.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:16.994 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:16.876+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:17.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:17.160+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:17.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:17.160+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:17.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:17.160+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:17.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:17.160+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:17.720 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:49:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:49:17.427+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 40100 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:49:18.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:17.719+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:18.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:17.719+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:18.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:17.719+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:18.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:17.720+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:18.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:17.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:18.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:17.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:18.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:17.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:18.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:17.910+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:18.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:18.128+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:18.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:18.128+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:18.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:18.128+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:18.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:18.129+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:19.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:18.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:19.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:18.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:19.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:18.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:19.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:18.935+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:19.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:18.766+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:19.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:18.766+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:19.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:18.766+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:19.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:18.766+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:19.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:19.137+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:19.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:19.137+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:19.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:19.137+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:19.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:19.138+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:20.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:19.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:20.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:19.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:20.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:19.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:20.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:19.932+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:20.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:19.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:20.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:19.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:20.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:19.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:20.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:19.727+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:20.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:20.160+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:20.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:20.160+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:20.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:20.160+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:20.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:20.161+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:21.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:20.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:21.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:20.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:21.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:20.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:21.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:20.911+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:21.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:20.752+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:21.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:20.752+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:21.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:20.752+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:21.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:20.752+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:21.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:21.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:21.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:21.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:21.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:21.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:21.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:21.182+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:21.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:21.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:21.950 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:21.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:21.950 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:21.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:21.950 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:21.722+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:22.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:22.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:22.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:22.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:22.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:22.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:22.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:22.169+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:21.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:21.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:21.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:21.950+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:22.762 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:49:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:49:22.432+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 40191 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:49:23.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:22.762+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:23.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:22.762+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:23.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:22.762+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:23.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:22.762+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:23.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:22.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:23.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:22.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:23.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:22.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:22.953+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:23.383 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:23.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:23.383 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:23.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:23.383 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:23.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:23.383 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:23.133+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:24.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:23.927+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:24.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:23.927+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:24.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:23.927+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:24.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:23.928+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:24.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:23.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:24.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:23.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:24.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:23.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:24.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:23.809+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:24.351 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:24.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:24.351 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:24.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:24.351 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:24.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:24.351 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:24.092+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:25.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:24.958+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:25.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:24.958+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:25.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:24.958+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:25.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:24.959+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:25.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:24.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:25.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:24.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:25.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:24.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:25.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:24.838+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:25.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:25.045+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:25.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:25.045+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:25.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:25.045+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:25.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:25.046+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:26.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:25.997+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:26.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:25.997+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:26.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:25.997+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:26.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:25.998+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:26.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:25.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:26.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:25.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:26.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:25.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:26.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:25.874+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:25.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:25.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:25.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:25.991+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:27.090 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:26.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:27.091 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:26.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:27.091 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:26.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:27.091 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:26.991+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:27.091 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:26.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:27.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:26.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:27.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:26.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:27.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:26.984+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:27.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:26.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:27.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:26.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:27.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:26.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:27.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:26.832+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:27.768 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:49:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:49:27.437+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 40282 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:49:28.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:27.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:28.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:27.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:28.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:27.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:28.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:27.965+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:28.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:27.989+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:27.989+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:27.989+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:27.989+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:27.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:27.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:27.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:27.825+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:29.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:28.952+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:29.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:28.952+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:29.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:28.952+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:29.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:28.953+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:29.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:29.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:29.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:29.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:29.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:29.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:29.003+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:28.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:28.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:28.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:28.845+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:30.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:29.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:30.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:29.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:30.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:29.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:30.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:29.868+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:30.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:29.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:30.266 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:29.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:30.266 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:29.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:30.266 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:29.915+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:30.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:30.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:30.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:30.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:30.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:30.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:30.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:30.026+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:31.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:30.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:31.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:30.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:31.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:30.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:31.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:30.981+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:31.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:30.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:31.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:30.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:31.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:30.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:31.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:30.913+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:30.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:30.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:30.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:30.931+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:32.138 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:32.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:32.138 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:32.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:32.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:32.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:32.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:32.008+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:32.139 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:31.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:32.139 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:31.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:32.139 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:31.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:32.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:31.957+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:32.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:31.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:32.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:31.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:32.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:31.870+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:32.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:31.870+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:32.768 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:49:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:49:32.443+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 40373 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:49:33.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:32.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:32.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:32.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:32.825+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:33.004+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:33.004+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:33.004+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:33.005+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:33.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:32.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:33.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:32.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:33.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:32.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:32.930+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:34.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:34.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:34.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:34.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:34.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:34.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:34.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:34.027+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:34.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:33.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:33.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:33.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:33.978+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:34.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:33.837+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:34.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:33.837+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:34.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:33.837+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:34.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:33.838+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:35.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:34.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:35.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:34.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:35.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:34.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:35.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:34.872+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:35.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:34.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:35.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:34.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:35.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:34.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:35.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:34.945+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:35.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:35.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:35.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:35.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:35.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:35.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:35.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:35.029+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:35.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:35.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:35.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:35.899+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:35.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:35.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:35.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:35.974+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:36.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:36.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:36.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:36.058+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:37.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:37.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:37.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:37.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:37.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:37.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:37.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:37.027+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:37.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:36.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:37.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:36.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:37.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:36.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:37.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:36.984+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:37.188 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:36.918+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:37.188 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:36.918+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:37.188 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:36.918+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:37.188 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:36.918+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:37.768 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:49:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:49:37.448+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 40465 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:49:38.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:37.947+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:38.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:37.947+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:38.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:37.947+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:38.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:37.948+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:38.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:37.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:38.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:37.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:37.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:37.979+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:37.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:37.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:37.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:37.959+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:39.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:38.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:39.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:38.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:39.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:38.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:39.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:39.000+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:38.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:38.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:38.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:38.938+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:38.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:38.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:39.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:38.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:39.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:38.974+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:40.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:39.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:40.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:39.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:40.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:39.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:40.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:39.965+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:40.267 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:39.931+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:40.267 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:39.931+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:40.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:39.931+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:40.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:39.932+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:40.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:40.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:40.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:40.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:40.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:40.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:40.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:40.040+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:41.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:40.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:41.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:40.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:41.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:40.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:41.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:40.957+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:41.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:41.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:41.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:41.047+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:41.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:40.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:41.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:40.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:41.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:40.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:41.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:40.944+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:42.236 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:41.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:42.237 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:41.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:42.237 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:41.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:42.237 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:41.945+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:42.237 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:42.075+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:42.237 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:42.075+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:42.237 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:42.075+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:42.238 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:42.076+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:42.238 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:41.916+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:42.238 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:41.916+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:42.238 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:41.916+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:42.238 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:41.916+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:42.518 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:49:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:49:42.454+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 40556 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:49:43.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:42.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:43.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:42.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:43.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:42.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:43.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:42.897+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:43.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:43.109+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:43.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:43.109+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:43.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:43.109+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:43.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:43.110+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:43.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:42.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:43.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:42.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:43.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:42.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:43.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:42.926+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:44.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:43.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:44.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:43.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:44.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:43.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:44.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:43.928+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:44.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:43.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:43.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:43.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:43.939+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:44.124+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:44.124+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:44.124+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:44.124+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:45.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:45.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:45.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:45.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:45.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:45.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:45.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:45.080+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:45.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:44.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:45.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:44.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:45.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:44.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:45.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:44.915+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:44.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:44.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:44.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:44.950+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:46.227 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:45.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:46.228 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:45.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:46.228 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:45.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:46.228 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:45.908+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:46.228 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:45.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:46.228 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:45.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:46.229 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:45.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:46.229 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:45.963+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:46.229 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:46.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:46.229 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:46.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:46.229 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:46.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:46.229 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:46.086+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:47.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:47.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:47.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:47.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:47.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:47.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:47.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:47.040+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:47.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:46.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:47.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:46.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:47.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:46.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:47.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:46.974+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:46.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:46.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:46.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:46.929+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:47.581 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:49:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:49:47.459+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 40647 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:49:48.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:47.884+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:48.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:47.884+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:48.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:47.884+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:48.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:47.884+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:48.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:47.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:48.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:47.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:48.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:47.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:48.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:47.962+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:48.053+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:48.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:48.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:48.054+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:49.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:48.932+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:49.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:48.932+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:49.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:48.932+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:49.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:48.933+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:48.949+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:48.949+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:48.949+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:48.949+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:49.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:49.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:49.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:49.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:49.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:49.079+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:50.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:50.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:50.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:50.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:50.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:50.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:50.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:50.064+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:50.267 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:49.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:50.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:49.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:50.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:49.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:50.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:49.945+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:50.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:49.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:50.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:49.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:50.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:49.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:50.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:49.886+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:51.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:50.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:51.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:50.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:51.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:50.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:51.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:50.913+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:51.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:51.109+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:51.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:51.109+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:51.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:51.109+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:51.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:51.109+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:50.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:50.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:50.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:50.861+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:52.144 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:51.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:52.144 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:51.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:52.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:51.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:52.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:51.848+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:52.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:51.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:52.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:51.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:52.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:51.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:52.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:51.909+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:52.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:52.144+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:52.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:52.144+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:52.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:52.144+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:52.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:52.145+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:52.768 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:49:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:49:52.464+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 40738 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:49:53.188 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:52.872+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:53.189 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:52.872+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:53.189 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:52.872+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:53.189 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:52.873+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:53.189 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:52.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:53.189 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:52.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:53.189 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:52.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:53.189 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:52.846+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:53.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:53.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:53.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:53.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:53.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:53.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:53.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:53.189+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:54.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:53.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:54.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:53.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:54.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:53.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:54.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:53.874+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:54.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:53.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:54.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:53.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:54.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:53.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:54.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:53.859+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:54.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:54.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:54.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:54.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:54.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:54.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:54.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:54.205+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:55.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:54.844+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:55.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:54.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:55.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:54.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:55.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:54.845+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:55.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:54.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:55.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:54.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:55.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:54.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:55.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:54.814+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:55.442 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:55.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:55.442 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:55.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:55.442 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:55.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:55.443 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:55.177+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:56.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:55.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:56.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:55.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:56.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:55.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:56.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:55.831+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:56.135 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:55.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:56.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:55.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:56.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:55.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:56.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:55.847+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:56.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:56.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:56.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:56.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:56.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:56.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:56.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:56.135+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:57.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:56.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:57.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:56.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:57.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:56.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:57.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:56.834+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:57.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:56.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:57.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:56.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:57.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:56.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:57.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:56.782+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:57.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:57.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:57.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:57.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:57.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:57.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:57.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:57.120+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:57.769 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:49:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:49:57.470+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 40829 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:49:58.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:57.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:58.093 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:57.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:58.093 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:57.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:58.093 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:57.799+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:58.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:57.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:58.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:57.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:58.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:57.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:58.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:57.821+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:58.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:58.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:58.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:58.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:58.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:58.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:58.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:58.093+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:49:59.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:58.792+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:59.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:58.792+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:59.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:58.792+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:49:59.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:58.792+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:59.118 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:58.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:59.118 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:58.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:59.118 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:58.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:49:59.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:58.799+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:49:59.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:59.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:59.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:59.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:59.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:59.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:49:59.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:49:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:49:59.117+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:00.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:59.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:00.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:59.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:00.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:59.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:00.148 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:49:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:49:59.815+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:00.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:59.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:00.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:59.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:00.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:59.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:00.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:49:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:49:59.820+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:00.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:00.147+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:00.147+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:00.147+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:00.147+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:01.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:00.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:01.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:00.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:01.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:00.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:01.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:00.858+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:01.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:00.798+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:01.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:00.798+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:01.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:00.798+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:01.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:00.798+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:01.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:01.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:01.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:01.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:01.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:01.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:01.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:01.155+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:02.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:01.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:02.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:01.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:02.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:01.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:02.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:01.845+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:02.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:01.836+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:02.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:01.836+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:02.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:01.836+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:02.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:01.836+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:02.418 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:02.145+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:02.418 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:02.145+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:02.418 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:02.145+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:02.418 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:02.146+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:02.769 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:50:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:50:02.475+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 40921 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:50:03.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:02.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:03.135 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:02.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:03.135 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:02.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:03.135 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:02.858+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:03.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:02.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:03.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:02.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:03.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:02.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:03.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:02.838+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:03.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:03.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:03.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:03.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:03.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:03.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:03.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:03.135+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:04.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:03.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:04.163 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:03.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:04.163 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:03.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:04.163 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:03.897+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:04.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:03.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:04.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:03.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:04.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:03.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:04.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:03.884+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:04.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:04.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:04.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:04.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:04.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:04.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:04.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:04.163+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:04.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:04.881+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:04.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:04.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:04.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:04.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:04.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:04.882+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:05.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:04.917+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:05.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:04.917+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:05.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:04.917+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:05.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:04.918+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:05.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:05.171+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:05.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:05.171+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:05.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:05.171+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:05.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:05.172+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:06.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:05.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:06.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:05.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:06.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:05.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:06.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:05.930+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:06.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:05.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:06.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:05.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:06.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:05.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:06.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:05.899+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:06.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:06.153+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:06.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:06.153+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:06.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:06.153+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:06.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:06.154+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:07.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:06.968+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:06.968+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:06.968+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:06.968+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:06.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:06.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:06.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:06.899+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:07.115+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:07.115+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:07.115+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:07.116+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:07.769 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:50:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:50:07.481+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 41011 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:50:08.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:07.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:08.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:07.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:08.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:07.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:07.981+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:08.114+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:08.114+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:08.114+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:08.115+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:08.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:07.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:08.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:07.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:08.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:07.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:08.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:07.900+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:09.106 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:09.008+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:09.106 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:09.008+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:09.106 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:09.008+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:09.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:09.008+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:09.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:08.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:09.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:08.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:09.108 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:08.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:09.108 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:08.854+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:09.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:09.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:09.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:09.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:09.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:09.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:09.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:09.106+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:10.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:09.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:10.135 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:09.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:10.135 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:09.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:10.135 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:09.807+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:10.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:09.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:10.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:09.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:10.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:09.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:10.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:09.989+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:10.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:10.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:10.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:10.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:10.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:10.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:10.135+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:11.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:10.997+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:11.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:10.997+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:11.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:10.997+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:11.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:10.998+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:11.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:10.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:11.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:10.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:11.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:10.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:11.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:10.845+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:11.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:11.151+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:11.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:11.151+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:11.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:11.151+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:11.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:11.152+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:12.108 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:11.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:12.108 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:11.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:12.109 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:11.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:12.109 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:11.958+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:12.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:11.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:12.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:11.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:12.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:11.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:12.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:11.798+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:12.487 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:12.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:12.487 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:12.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:12.487 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:12.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:12.487 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:12.108+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:12.769 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:50:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:50:12.486+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 41103 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:50:13.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:13.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:13.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:13.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:13.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:13.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:13.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:13.064+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:12.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:12.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:12.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:12.945+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:12.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:12.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:12.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:13.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:12.841+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:14.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:14.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:14.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:14.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:14.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:14.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:14.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:14.060+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:13.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:13.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:13.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:13.985+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:14.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:13.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:13.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:13.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:13.874+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:15.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:15.025+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:15.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:15.025+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:15.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:15.025+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:15.025+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:15.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:15.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:15.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:15.031+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:14.827+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:14.827+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:14.827+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:14.827+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:16.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:15.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:16.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:15.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:16.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:15.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:15.981+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:16.009+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:16.009+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:16.009+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:16.010+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:15.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:15.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:15.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:15.819+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:17.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:16.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:17.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:16.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:17.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:16.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:17.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:16.934+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:17.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:17.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:17.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:17.017+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:17.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:16.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:16.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:16.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:16.860+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:17.768 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:50:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:50:17.492+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 41194 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:50:18.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:18.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:18.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:18.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:18.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:18.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:18.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:18.067+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:18.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:17.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:18.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:17.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:18.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:17.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:18.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:17.858+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:17.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:17.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:17.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:17.887+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:19.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:18.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:19.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:18.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:19.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:18.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:19.097 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:18.839+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:19.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:18.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:19.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:18.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:19.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:18.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:19.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:18.822+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:19.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:19.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:19.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:19.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:19.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:19.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:19.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:19.094+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:20.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:19.852+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:20.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:19.852+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:20.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:19.852+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:20.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:19.852+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:20.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:19.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:20.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:19.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:20.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:19.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:20.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:19.887+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:20.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:20.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:20.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:20.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:20.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:20.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:20.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:20.141+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:21.144 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:20.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:21.145 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:20.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:21.145 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:20.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:21.145 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:20.887+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:21.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:20.834+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:21.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:20.834+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:21.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:20.834+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:21.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:20.834+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:21.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:21.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:21.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:21.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:21.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:21.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:21.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:21.145+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:22.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:22.138+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:22.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:22.138+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:22.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:21.876+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:22.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:21.876+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:22.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:21.876+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:22.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:21.877+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:22.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:21.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:22.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:21.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:22.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:21.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:22.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:21.861+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:22.498 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:22.138+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:22.498 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:22.139+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:22.769 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:50:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:50:22.497+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 41286 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:50:23.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:22.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:23.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:22.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:23.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:22.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:23.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:22.864+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:23.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:23.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:23.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:23.113+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:22.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:22.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:22.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:22.902+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:24.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:24.065+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:24.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:24.065+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:24.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:24.065+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:24.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:24.065+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:24.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:23.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:24.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:23.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:24.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:23.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:24.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:23.872+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:23.855+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:24.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:23.855+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:24.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:23.855+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:24.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:23.856+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:25.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:25.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:25.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:25.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:25.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:25.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:25.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:25.031+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:24.828+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:24.828+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:24.828+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:24.828+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:24.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:25.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:24.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:25.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:24.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:25.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:24.899+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:26.163 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:25.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:26.163 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:25.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:26.163 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:25.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:26.163 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:25.905+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:26.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:25.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:26.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:25.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:26.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:25.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:26.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:25.875+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:26.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:25.997+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:26.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:25.997+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:26.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:25.997+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:26.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:25.998+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:27.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:26.921+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:27.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:26.921+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:27.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:26.921+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:26.922+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:27.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:27.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:27.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:27.021+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:26.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:26.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:26.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:26.858+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:27.769 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:50:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:50:27.503+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 41376 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:50:28.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:27.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:28.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:27.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:28.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:27.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:28.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:27.969+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:28.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:28.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:28.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:28.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:28.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:28.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:28.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:28.039+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:27.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:28.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:27.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:28.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:27.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:28.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:27.864+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:29.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:29.015+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:29.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:29.015+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:29.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:29.015+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:29.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:29.016+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:29.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:29.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:29.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:29.051+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:29.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:28.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:29.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:28.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:29.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:28.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:29.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:28.847+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:30.102 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:30.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:30.102 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:30.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:30.102 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:30.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:30.102 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:30.036+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:30.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:29.844+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:30.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:29.844+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:30.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:29.844+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:30.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:29.844+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:30.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:30.101+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:30.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:30.101+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:30.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:30.101+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:30.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:30.102+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:31.148 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:31.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:31.148 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:31.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:31.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:31.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:31.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:31.013+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:31.149 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:30.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:31.149 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:30.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:31.149 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:30.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:31.149 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:30.800+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:31.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:31.148+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:31.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:31.148+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:31.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:31.148+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:31.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:31.148+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:32.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:32.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:32.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:32.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:32.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:32.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:32.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:32.029+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:32.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:31.830+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:32.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:31.830+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:32.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:31.830+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:32.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:31.831+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:32.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:32.152+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:32.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:32.152+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:32.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:32.152+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:32.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:32.153+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:32.769 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:50:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:50:32.508+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 41468 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:50:33.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:32.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:33.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:32.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:33.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:32.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:33.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:32.839+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:33.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:33.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:33.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:33.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:33.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:33.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:33.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:33.076+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:33.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:33.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:33.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:33.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:33.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:33.165+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:33.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:33.166+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:34.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:33.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:34.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:33.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:34.193 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:33.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:34.193 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:33.850+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:34.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:34.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:34.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:34.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:34.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:34.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:34.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:34.055+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:34.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:34.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:34.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:34.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:34.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:34.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:34.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:34.191+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:35.228 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:35.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:35.228 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:35.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:35.229 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:35.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:35.229 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:35.067+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:35.229 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:34.852+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:35.229 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:34.852+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:35.229 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:34.852+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:35.229 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:34.852+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:35.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:35.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:35.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:35.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:35.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:35.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:35.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:35.228+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:35.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:35.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:35.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:35.818+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:36.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:36.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:36.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:36.059+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:36.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:36.277+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:36.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:36.277+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:36.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:36.277+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:36.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:36.277+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:37.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:37.081+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:37.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:37.081+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:37.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:37.081+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:37.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:37.081+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:37.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:36.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:37.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:36.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:37.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:36.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:37.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:36.855+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:37.533 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:50:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:50:37.514+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 41559 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:50:37.534 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:37.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:37.534 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:37.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:37.534 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:37.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:37.534 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:37.273+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:38.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:38.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:38.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:38.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:38.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:38.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:38.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:38.045+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:37.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:37.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:37.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:37.839+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:38.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:38.269+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:38.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:38.269+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:38.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:38.269+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:38.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:38.270+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:39.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:39.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:39.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:39.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:39.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:39.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:39.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:39.076+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:38.875+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:38.875+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:38.875+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:38.875+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:39.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:39.272+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:39.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:39.272+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:39.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:39.272+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:39.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:39.273+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:40.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:40.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:40.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:40.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:40.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:40.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:40.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:40.049+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:40.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:39.830+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:40.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:39.830+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:40.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:39.830+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:40.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:39.830+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:40.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:40.300+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:40.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:40.300+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:40.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:40.300+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:40.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:40.301+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:41.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:41.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:41.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:41.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:41.259 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:41.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:41.259 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:41.001+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:41.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:40.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:41.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:40.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:41.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:40.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:41.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:40.803+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:41.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:41.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:41.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:41.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:41.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:41.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:41.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:41.258+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:42.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:41.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:42.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:41.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:42.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:41.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:07.877655+0000 (oldest deadline 2024-02-19T19:40:27.877655+0000) 2024-02-19T19:50:42.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:41.772+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:42.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:41.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:41.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:41.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:41.982+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:42.642 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:50:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:50:42.519+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 41653 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:50:42.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:42.295+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:42.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:42.295+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:42.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:42.295+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:42.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:42.296+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:42.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:42.752+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:43.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:42.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:43.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:42.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:43.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:42.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:43.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:42.961+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:43.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:43.280+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:43.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:43.280+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:43.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:43.280+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:43.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:43.281+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:43.924 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:43.789+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:44.239 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:43.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:44.239 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:43.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:44.239 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:43.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:44.239 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:43.925+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:44.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:44.238+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:44.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:44.238+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:44.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:44.238+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:44.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:44.239+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:44.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:44.811+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:45.231 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:44.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:45.232 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:44.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:45.232 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:44.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:45.232 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:44.887+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:45.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:45.231+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:45.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:45.231+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:45.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:45.231+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:45.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:45.232+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:46.253 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:45.809+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:46.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:45.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:46.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:45.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:46.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:45.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:46.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:45.843+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:46.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:46.253+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:46.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:46.253+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:46.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:46.253+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:46.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:46.254+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:47.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:46.814+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:47.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:46.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:47.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:46.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:46.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:46.839+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:47.534 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:50:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:50:47.525+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 41743 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:50:47.534 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:47.278+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:47.535 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:47.278+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:47.535 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:47.278+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:47.535 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:47.279+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:48.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:47.848+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:48.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:47.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:48.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:47.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:48.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:47.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:47.839+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:48.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:48.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:48.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:48.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:48.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:48.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:48.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:48.271+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:49.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:48.834+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:49.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:48.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:49.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:48.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:49.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:48.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:49.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:48.884+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:49.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:49.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:49.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:49.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:49.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:49.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:49.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:49.297+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:50.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:49.869+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:50.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:49.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:50.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:49.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:50.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:49.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:50.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:49.879+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:50.738 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:50.329+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:50.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:50.329+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:50.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:50.329+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:50.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:50.330+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:51.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:50.860+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:51.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:50.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:51.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:50.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:51.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:50.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:51.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:50.881+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:51.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:51.322+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:51.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:51.322+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:51.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:51.322+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:51.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:51.322+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:52.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:51.857+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:52.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:51.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:52.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:51.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:51.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:19.595090+0000 (oldest deadline 2024-02-19T19:40:39.595090+0000) 2024-02-19T19:50:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:51.836+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:52.657 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:50:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:50:52.530+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 41835 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:50:52.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:52.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:52.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:52.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:52.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:52.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:52.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:52.356+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:53.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:52.843+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:53.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:52.861+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:53.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:53.341+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:53.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:53.341+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:53.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:53.341+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:53.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:53.342+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:54.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:53.902+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:54.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:53.798+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:54.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:54.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:54.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:54.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:54.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:54.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:40:24.210143+0000 (oldest deadline 2024-02-19T19:40:44.210143+0000) 2024-02-19T19:50:54.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:54.358+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:55.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:54.927+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:55.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:54.773+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:55.744 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:55.390+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:56.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:55.744+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:56.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:55.963+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:56.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:56.353+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:57.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:56.756+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:57.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:56.961+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:57.659 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:57.351+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:57.659 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:50:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:50:57.535+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 41926 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:50:57.953 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:57.762+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:58.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:57.985+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:58.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:58.357+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:59.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:58.965+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:50:59.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:58.794+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:50:59.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:50:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:50:59.371+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:00.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:50:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:50:59.923+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:00.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:50:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:50:59.795+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:00.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:00.363+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:01.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:00.898+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:01.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:00.843+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:01.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:01.396+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:02.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:01.886+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:02.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:01.858+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:02.662 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:51:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:51:02.540+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 42018 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:51:02.663 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:02.362+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:02.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:02.875+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:02.999 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:02.901+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:03.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:03.397+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:04.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:03.855+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:04.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:03.875+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:04.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:04.445+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:05.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:04.885+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:05.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:04.871+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:05.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:05.449+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:06.100 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:05.866+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:06.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:05.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:06.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:05.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:06.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:05.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:06.101 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:05.824+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:06.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:06.413+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:07.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:06.886+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:07.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:06.849+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:07.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:06.849+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:06.849+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:06.849+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:07.668 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:51:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:51:07.545+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 42108 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:51:07.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:07.412+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:08.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:07.861+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:08.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:07.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:08.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:07.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:08.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:07.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:08.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:07.895+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:08.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:08.416+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:09.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:08.905+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:09.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:08.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:08.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:08.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:08.849+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:09.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:09.464+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:10.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:09.913+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:10.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:09.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:10.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:09.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:10.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:09.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:10.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:09.881+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:10.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:10.425+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:11.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:10.919+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:11.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:10.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:11.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:10.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:11.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:10.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:11.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:10.906+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:11.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:11.405+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:12.243 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:11.950+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:12.244 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:11.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:12.244 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:11.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:12.244 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:11.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:12.244 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:11.905+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:12.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:12.417+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:12.888 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:51:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:51:12.550+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 42200 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:51:13.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:12.924+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:13.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:12.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:13.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:12.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:12.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:12.888+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:13.690 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:13.389+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:14.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:13.904+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:14.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:13.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:14.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:13.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:14.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:13.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:14.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:13.897+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:14.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:14.411+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:15.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:14.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:14.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:14.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:14.863+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:14.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:14.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:14.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:15.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:14.860+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:15.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:15.445+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:15.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:15.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:15.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:15.903+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:15.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:15.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:15.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:15.874+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:16.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:16.476+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:17.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:16.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:17.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:16.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:17.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:16.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:17.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:16.872+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:16.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:16.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:16.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:16.862+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:17.772 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:51:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:51:17.555+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 42291 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:51:17.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:17.464+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:17.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:17.464+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:17.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:17.464+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:17.773 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:17.465+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:18.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:17.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:18.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:17.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:18.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:17.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:18.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:17.904+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:18.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:17.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:18.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:17.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:18.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:17.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:18.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:17.910+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:18.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:18.511+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:18.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:18.511+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:18.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:18.511+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:18.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:18.511+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:19.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:18.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:19.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:18.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:18.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:18.887+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:18.936+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:18.936+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:18.936+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:18.936+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:19.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:19.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:19.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:19.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:19.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:19.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:19.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:19.506+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:20.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:19.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:20.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:19.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:19.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:19.862+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:19.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:19.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:19.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:19.955+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:20.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:20.481+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:20.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:20.481+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:20.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:20.481+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:20.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:20.482+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:21.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:20.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:21.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:20.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:21.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:20.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:21.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:20.902+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:20.970+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:20.970+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:20.970+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:20.970+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:21.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:21.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:21.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:21.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:21.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:21.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:21.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:21.468+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:22.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:21.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:22.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:21.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:22.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:21.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:22.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:21.907+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:22.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:22.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:22.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:22.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:22.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:22.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:22.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:22.002+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:22.769 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:51:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:51:22.561+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 42383 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:51:22.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:22.486+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:22.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:22.486+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:22.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:22.486+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:22.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:22.487+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:23.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:22.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:23.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:22.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:23.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:22.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:23.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:22.861+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:23.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:22.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:23.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:22.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:23.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:22.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:23.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:22.966+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:23.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:23.477+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:23.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:23.477+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:23.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:23.477+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:23.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:23.478+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:24.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:23.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:24.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:23.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:24.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:23.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:24.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:23.833+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:24.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:23.953+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:24.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:23.953+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:24.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:23.953+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:24.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:23.954+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:24.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:24.493+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:24.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:24.493+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:24.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:24.493+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:24.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:24.494+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:25.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:24.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:25.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:24.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:25.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:24.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:25.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:24.879+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:24.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:24.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:24.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:24.946+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:25.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:25.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:25.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:25.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:25.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:25.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:25.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:25.493+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:26.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:25.873+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:26.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:25.873+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:26.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:25.873+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:25.874+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:26.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:25.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:26.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:25.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:26.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:25.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:26.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:25.982+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:26.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:26.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:26.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:26.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:26.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:26.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:26.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:26.515+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:27.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:26.865+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:27.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:26.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:27.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:26.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:27.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:26.866+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:27.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:27.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:27.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:27.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:27.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:27.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:27.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:27.015+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:27.832 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:51:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:51:27.566+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 42472 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:51:27.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:27.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:27.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:27.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:27.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:27.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:27.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:27.536+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:28.229 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:27.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:28.229 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:27.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:28.229 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:27.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:28.229 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:27.985+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:28.229 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:27.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:28.230 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:27.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:28.230 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:27.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:28.230 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:27.832+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:28.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:28.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:28.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:28.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:28.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:28.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:28.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:28.527+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:29.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:28.830+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:29.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:28.830+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:28.830+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:28.831+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:29.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:29.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:29.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:29.004+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:29.825 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:29.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:29.825 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:29.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:29.826 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:29.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:29.826 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:29.527+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:30.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:29.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:30.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:29.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:30.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:29.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:30.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:29.825+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:30.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:30.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:30.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:30.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:30.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:30.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:30.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:30.029+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:30.845 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:30.565+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:30.846 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:30.565+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:30.846 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:30.565+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:30.846 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:30.566+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:31.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:30.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:31.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:30.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:31.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:30.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:31.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:30.845+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:31.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:30.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:31.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:30.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:31.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:30.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:31.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:30.993+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:31.850 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:31.592+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:31.850 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:31.592+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:31.850 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:31.592+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:31.851 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:31.593+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:32.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:31.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:32.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:31.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:32.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:31.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:31.850+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:32.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:31.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:32.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:31.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:32.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:31.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:32.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:31.992+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:32.883 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:51:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:51:32.572+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 42564 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:51:32.883 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:32.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:32.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:32.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:32.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:32.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:32.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:32.608+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:32.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:32.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:32.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:33.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:32.995+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:32.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:32.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:32.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:32.883+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:33.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:33.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:33.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:33.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:33.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:33.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:33.896 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:33.595+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:34.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:33.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:34.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:33.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:34.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:33.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:34.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:33.895+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:34.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:33.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:34.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:33.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:34.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:33.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:34.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:33.975+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:34.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:34.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:34.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:34.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:34.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:34.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:34.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:34.588+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:35.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:35.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:35.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:35.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:35.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:35.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:35.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:35.018+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:34.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:34.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:34.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:34.882+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:35.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:35.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:35.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:35.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:35.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:35.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:35.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:35.605+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:36.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:36.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:36.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:36.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:36.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:36.026+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:35.914+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:35.914+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:35.914+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:35.915+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:36.913 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:36.600+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:36.913 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:36.600+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:36.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:36.600+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:36.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:36.601+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:37.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:36.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:37.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:36.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:37.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:36.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:36.913+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:37.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:37.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:37.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:37.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:37.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:37.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:37.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:37.017+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:37.902 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:51:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:51:37.578+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 42655 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:51:37.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:37.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:37.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:37.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:37.903 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:37.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:37.903 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:37.571+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:38.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:37.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:38.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:37.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:38.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:37.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:37.902+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:38.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:38.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:38.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:38.062+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:38.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:38.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:38.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:38.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:38.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:38.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:38.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:38.594+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:39.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:39.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:39.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:39.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:39.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:39.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:39.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:39.045+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:38.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:38.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:38.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:38.905+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:39.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:39.592+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:39.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:39.592+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:39.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:39.592+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:39.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:39.592+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:40.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:39.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:40.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:39.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:39.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:39.935+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:40.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:40.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:40.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:40.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:40.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:40.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:40.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:40.011+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:40.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:40.562+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:40.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:40.562+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:40.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:40.562+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:40.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:40.563+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:41.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:40.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:41.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:40.939+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:41.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:40.939+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:40.939+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:41.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:41.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:41.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:41.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:41.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:41.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:41.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:41.039+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:41.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:41.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:41.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:41.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:41.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:41.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:41.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:41.543+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:42.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:41.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:42.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:41.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:42.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:41.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:41.946+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:42.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:42.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:42.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:42.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:42.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:42.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:42.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:42.001+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:42.945 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:51:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:51:42.583+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 42747 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:51:42.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:42.521+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:42.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:42.521+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:42.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:42.521+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:42.947 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:42.522+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:43.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:43.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:43.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:43.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:43.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:43.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:43.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:43.013+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:42.944+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:42.944+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:42.944+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:42.945+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:43.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:43.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:43.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:43.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:43.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:43.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:43.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:43.569+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:43.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:43.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:43.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:43.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:43.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:43.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:43.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:43.905+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:44.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:43.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:44.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:43.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:44.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:43.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:44.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:43.982+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:44.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:44.618+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:44.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:44.618+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:44.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:44.618+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:44.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:44.619+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:45.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:45.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:45.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:45.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:45.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:45.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:45.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:45.021+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:44.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:44.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:44.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:44.894+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:45.851 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:45.595+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:45.851 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:45.595+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:45.851 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:45.595+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:45.851 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:45.596+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:46.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:46.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:46.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:46.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:46.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:46.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:46.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:46.028+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:45.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:45.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:45.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:45.851+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:46.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:46.808+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:46.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:46.808+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:46.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:46.808+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:46.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:46.809+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:46.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:46.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:46.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:46.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:46.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:46.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:46.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:46.574+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:47.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:46.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:47.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:46.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:47.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:46.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:47.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:46.987+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:48.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:51:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:51:47.589+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 42837 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:51:48.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:47.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:47.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:47.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:47.768+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:47.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:47.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:47.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:48.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:47.612+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:48.418 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:48.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:48.418 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:48.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:48.418 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:48.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:48.418 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:48.033+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:48.745 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:48.571+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:48.746 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:48.571+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:48.746 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:48.571+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:48.746 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:48.572+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:49.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:48.745+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:49.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:48.745+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:49.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:48.745+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:49.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:48.745+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:49.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:49.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:49.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:49.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:49.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:49.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:49.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:49.009+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:49.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:49.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:49.981 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:49.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:49.981 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:49.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:49.981 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:49.622+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:49.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:49.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:49.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:49.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:49.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:49.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:49.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:49.781+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:50.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:49.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:50.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:49.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:50.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:49.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:50.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:49.980+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:50.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:50.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:50.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:50.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:50.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:50.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:50.996 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:50.825+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:50.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:50.630+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:50.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:50.630+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:50.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:50.630+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:50.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:50.631+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:51.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:50.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:50.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:50.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:50.994+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:51.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:51.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:51.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:51.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:51.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:51.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:51.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:51.624+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:51.953 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:51.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:51.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:51.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:51.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:51.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:51.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:51.780+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:52.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:51.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:52.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:51.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:51.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:51.952+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:52.956 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:51:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:51:52.595+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 42929 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:51:52.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:52.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:52.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:52.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:52.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:52.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:52.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:52.580+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:52.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:52.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:52.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:52.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:52.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:52.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:52.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:52.740+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:53.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:52.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:53.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:52.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:53.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:52.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:53.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:52.955+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:53.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:53.566+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:53.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:53.566+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:53.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:53.566+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:53.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:53.567+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:54.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:53.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:54.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:53.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:54.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:53.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:53.764+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:54.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:53.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:54.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:53.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:54.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:53.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:54.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:53.987+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:54.803 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:54.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:54.803 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:54.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:54.803 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:54.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:54.804 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:54.540+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:55.251 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:54.802+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:55.251 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:54.802+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:55.251 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:54.802+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:55.251 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:54.803+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:55.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:54.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:55.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:54.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:55.252 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:54.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:55.252 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:54.976+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:55.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:55.756+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:55.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:55.756+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:55.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:55.756+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:55.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:55.756+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:55.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:55.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:55.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:55.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:55.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:55.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:55.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:55.518+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:56.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:55.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:56.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:55.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:56.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:55.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:56.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:55.935+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:56.928 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:56.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:56.929 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:56.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:56.929 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:56.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:56.929 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:56.787+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:56.929 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:56.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:56.930 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:56.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:56.930 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:56.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:56.930 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:56.556+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:57.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:56.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:57.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:56.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:57.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:56.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:57.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:56.928+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:57.835 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:51:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:51:57.600+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 43020 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:51:57.836 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:57.584+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:57.836 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:57.584+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:57.836 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:57.584+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:57.836 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:57.585+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:58.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:57.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:58.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:57.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:58.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:57.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:58.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:57.905+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:57.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:57.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:57.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:57.835+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:58.849 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:58.608+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:58.850 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:58.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:58.850 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:58.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:58.850 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:58.609+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:59.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:58.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:59.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:58.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:59.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:58.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:59.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:58.849+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:51:59.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:58.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:59.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:58.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:59.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:58.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:51:59.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:58.903+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:59.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:59.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:59.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:59.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:59.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:59.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:51:59.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:51:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:51:59.571+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:51:59.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:59.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:59.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:59.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:59.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:59.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:51:59.911 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:51:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:51:59.813+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:00.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:59.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:00.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:59.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:00.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:59.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:00.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:51:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:51:59.909+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:00.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:00.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:00.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:00.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:00.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:00.536+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:00.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:00.537+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:00.902 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:00.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:00.902 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:00.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:00.902 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:00.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:00.902 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:00.785+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:01.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:00.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:01.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:00.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:01.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:00.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:01.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:00.900+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:01.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:01.538+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:01.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:01.538+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:01.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:01.538+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:01.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:01.539+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:02.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:01.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:02.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:01.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:02.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:01.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:01.794+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:02.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:01.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:02.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:01.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:02.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:01.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:02.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:01.947+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:02.834 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:52:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:52:02.606+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 43112 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:52:02.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:02.538+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:02.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:02.538+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:02.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:02.538+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:02.835 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:02.538+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:03.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:02.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:03.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:02.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:03.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:02.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:02.834+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:03.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:02.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:03.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:02.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:03.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:02.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:03.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:02.931+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:03.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:03.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:03.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:03.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:03.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:03.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:03.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:03.506+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:04.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:03.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:04.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:03.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:04.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:03.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:03.814+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:04.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:03.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:04.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:03.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:04.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:03.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:04.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:03.916+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:04.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:04.529+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:04.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:04.529+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:04.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:04.529+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:04.530+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:05.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:04.805+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:05.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:04.805+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:04.805+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:04.805+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:04.940+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:04.940+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:04.940+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:04.941+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:05.843 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:05.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:05.843 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:05.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:05.843 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:05.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:05.843 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:05.527+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:06.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:05.969+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:06.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:05.969+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:06.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:05.969+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:06.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:05.969+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:05.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:05.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:05.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:05.843+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:06.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:06.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:06.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:06.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:06.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:06.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:06.508+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:07.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:06.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:07.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:06.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:06.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:06.853+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:06.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:06.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:06.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:06.972+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:07.897 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:52:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:52:07.612+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 43202 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:52:07.898 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:07.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:07.898 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:07.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:07.898 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:07.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:07.898 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:07.543+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:08.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:07.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:07.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:07.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:07.897+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:08.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:07.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:08.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:07.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:08.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:07.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:08.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:07.968+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:08.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:08.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:08.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:08.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:08.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:08.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:08.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:08.519+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:09.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:09.005+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:09.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:09.005+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:09.005+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:09.006+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:08.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:08.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:08.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:08.855+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:09.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:09.538+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:09.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:09.538+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:09.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:09.538+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:09.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:09.539+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:10.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:10.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:10.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:10.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:10.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:10.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:10.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:10.002+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:09.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:09.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:09.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:09.902+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:10.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:10.554+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:10.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:10.554+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:10.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:10.554+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:10.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:10.555+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:11.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:10.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:10.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:10.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:10.894+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:11.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:11.011+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:11.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:11.011+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:11.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:11.011+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:11.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:11.011+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:11.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:11.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:11.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:11.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:11.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:11.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:11.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:11.518+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:12.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:11.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:12.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:11.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:11.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:11.909+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:12.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:12.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:12.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:12.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:12.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:12.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:12.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:12.026+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:12.769 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:52:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:52:12.618+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 43294 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:52:12.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:12.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:12.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:12.498+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:12.499+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:13.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:12.917+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:13.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:12.917+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:13.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:12.917+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:12.918+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:13.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:13.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:13.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:13.038+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:13.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:13.504+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:13.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:13.504+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:13.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:13.504+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:13.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:13.504+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:14.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:13.917+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:14.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:13.917+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:13.917+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:13.917+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:14.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:14.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:14.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:14.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:14.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:14.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:14.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:14.067+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:14.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:14.483+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:14.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:14.483+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:14.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:14.483+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:14.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:14.483+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:15.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:14.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:15.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:14.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:14.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:14.932+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:15.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:15.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:15.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:15.025+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:15.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:15.530+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:15.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:15.530+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:15.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:15.530+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:15.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:15.531+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:16.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:16.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:16.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:16.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:16.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:16.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:16.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:16.064+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:15.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:15.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:15.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:15.898+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:16.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:16.513+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:16.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:16.513+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:16.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:16.513+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:16.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:16.514+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:17.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:17.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:17.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:17.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:17.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:17.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:17.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:17.087+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:16.942+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:16.942+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:16.942+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:16.943+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:17.769 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:52:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:52:17.623+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 43385 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:52:17.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:17.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:17.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:17.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:17.505+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:18.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:18.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:18.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:18.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:18.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:18.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:18.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:18.105+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:17.983+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:17.984+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:17.984+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:17.984+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:18.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:18.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:18.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:18.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:18.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:18.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:18.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:18.539+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:19.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:18.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:19.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:18.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:19.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:18.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:18.969+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:19.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:19.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:19.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:19.124+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:19.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:19.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:19.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:19.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:19.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:19.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:19.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:19.544+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:20.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:19.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:20.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:19.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:20.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:19.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:19.942+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:20.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:20.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:20.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:20.152+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:20.964 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:20.572+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:20.964 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:20.572+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:20.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:20.572+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:20.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:20.573+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:21.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:20.963+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:21.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:20.963+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:20.963+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:20.964+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:21.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:21.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:21.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:21.124+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:22.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:21.530+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:22.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:21.530+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:22.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:21.530+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:22.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:21.531+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:22.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:22.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:22.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:22.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:22.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:22.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:22.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:22.152+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:22.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:22.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:22.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:22.003+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:22.899 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:52:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:52:22.629+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 43477 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:52:22.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:22.551+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:22.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:22.551+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:22.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:22.551+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:22.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:22.551+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:23.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:22.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:23.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:22.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:23.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:22.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:23.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:22.987+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:23.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:23.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:23.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:23.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:23.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:23.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:23.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:23.167+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:23.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:23.522+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:23.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:23.522+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:23.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:23.522+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:23.777 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:23.523+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:24.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:24.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:24.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:24.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:24.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:24.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:24.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:24.037+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:24.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:24.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:24.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:24.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:24.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:24.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:24.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:24.132+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:25.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:24.562+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:25.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:24.562+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:25.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:24.562+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:25.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:24.563+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:25.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:25.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:25.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:25.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:25.167+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:25.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:25.055+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:25.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:25.055+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:25.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:25.055+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:25.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:25.056+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:25.971 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:25.580+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:25.971 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:25.580+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:25.971 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:25.580+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:25.971 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:25.580+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:26.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:26.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:26.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:26.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:26.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:26.029+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:26.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:26.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:26.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:26.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:26.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:26.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:26.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:26.120+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:27.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:26.564+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:27.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:26.564+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:27.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:26.564+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:27.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:26.565+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:27.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:27.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:27.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:27.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:27.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:27.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:27.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:27.069+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:27.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:27.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:27.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:27.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:27.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:27.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:27.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:27.119+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:28.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:52:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:52:27.634+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 43567 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:52:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:27.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:27.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:27.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:27.545+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:28.505 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:28.067+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:28.506 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:28.067+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:28.506 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:28.067+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:28.506 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:28.067+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:28.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:28.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:28.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:28.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:28.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:28.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:28.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:28.104+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:28.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:28.504+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:28.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:28.504+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:28.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:28.504+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:28.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:28.505+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:29.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:29.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:29.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:29.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:29.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:29.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:29.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:29.114+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:29.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:29.100+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:29.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:29.100+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:29.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:29.100+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:29.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:29.101+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:30.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:29.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:30.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:29.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:29.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:29.545+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:30.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:30.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:30.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:30.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:30.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:30.152+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:30.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:30.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:30.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:30.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:30.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:30.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:30.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:30.090+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:31.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:30.538+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:31.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:30.538+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:31.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:30.538+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:31.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:30.539+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:31.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:31.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:31.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:31.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:31.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:31.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:31.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:31.138+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:31.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:31.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:31.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:31.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:31.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:31.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:31.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:31.156+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:32.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:31.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:32.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:31.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:32.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:31.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:32.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:31.550+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:32.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:32.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:32.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:32.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:32.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:32.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:32.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:32.154+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:32.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:32.186+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:32.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:32.186+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:32.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:32.186+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:32.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:32.186+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:33.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:52:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:52:32.640+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 43659 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:52:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:32.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:32.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:32.526+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:32.527+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:33.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:33.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:33.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:33.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:33.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:33.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:33.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:33.228+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:33.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:33.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:33.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:33.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:33.152+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:33.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:33.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:33.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:33.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:33.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:33.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:33.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:33.554+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:34.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:34.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:34.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:34.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:34.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:34.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:34.132+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:34.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:34.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:34.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:34.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:34.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:34.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:34.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:34.200+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:35.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:34.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:35.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:34.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:35.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:34.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:35.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:34.576+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:35.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:35.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:35.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:35.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:35.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:35.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:35.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:35.109+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:35.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:35.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:35.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:35.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:35.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:35.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:35.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:35.204+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:36.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:35.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:36.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:35.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:35.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:35.538+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:36.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:36.192+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:36.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:36.192+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:36.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:36.192+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:36.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:36.192+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:36.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:36.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:36.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:36.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:36.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:36.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:36.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:36.069+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:37.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:36.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:36.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:36.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:36.524+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:37.490 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:37.200+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:37.490 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:37.200+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:37.490 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:37.200+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:37.490 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:37.201+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:37.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:37.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:37.491 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:37.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:37.491 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:37.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:37.491 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:37.060+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:37.766 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:52:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:52:37.646+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 43750 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:52:37.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:37.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:37.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:37.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:37.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:37.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:37.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:37.490+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:38.470 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:38.159+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:38.470 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:38.159+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:38.470 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:38.159+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:38.470 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:38.159+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:38.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:38.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:38.471 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:38.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:38.471 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:38.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:38.471 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:38.068+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:38.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:38.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:38.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:38.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:38.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:38.469+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:39.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:39.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:39.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:39.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:39.079+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:39.133+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:39.133+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:39.133+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:39.134+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:39.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:39.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:39.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:39.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:39.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:39.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:39.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:39.493+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:40.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:40.085+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:40.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:40.085+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:40.085+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:40.086+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:40.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:40.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:40.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:40.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:40.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:40.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:40.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:40.046+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:41.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:40.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:41.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:40.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:41.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:40.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:40.525+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:41.331 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:41.101+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:41.331 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:41.101+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:41.331 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:41.101+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:41.331 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:41.102+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:41.332 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:41.039+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:41.332 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:41.039+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:41.332 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:41.039+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:41.332 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:41.039+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:41.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:41.571+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:41.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:41.571+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:41.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:41.571+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:41.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:41.572+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:42.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:42.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:42.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:42.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:42.026+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:42.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:42.135+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:42.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:42.135+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:42.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:42.135+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:42.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:42.136+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:43.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:52:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:52:42.652+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 43842 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:52:43.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:42.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:43.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:42.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:43.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:42.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:43.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:42.622+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:43.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:43.100+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:43.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:43.100+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:43.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:43.100+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:43.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:43.100+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:43.040+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:43.040+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:43.040+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:43.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:43.040+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:43.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:43.612+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:43.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:43.612+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:43.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:43.612+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:43.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:43.613+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:44.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:44.133+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:44.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:44.133+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:44.133+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:44.133+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:44.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:44.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:44.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:44.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:44.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:44.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:44.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:44.046+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:45.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:44.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:45.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:44.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:44.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:44.661+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:45.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:45.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:45.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:45.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:45.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:45.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:45.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:45.084+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:45.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:45.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:45.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:45.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:45.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:45.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:45.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:45.053+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:46.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:45.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:46.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:45.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:46.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:45.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:46.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:45.664+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:46.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:46.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:46.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:46.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:46.051+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:46.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:46.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:46.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:46.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:46.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:46.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:46.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:46.029+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:47.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:46.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:47.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:46.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:47.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:46.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:47.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:46.617+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:47.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:47.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:47.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:47.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:47.018+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:47.027+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:47.027+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:47.027+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:47.028+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:48.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:52:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:52:47.657+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 43929 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:52:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:47.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:47.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:47.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:47.617+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:48.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:48.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:48.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:48.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:48.074+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:48.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:48.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:48.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:48.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:48.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:48.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:48.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:48.033+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:49.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:48.596+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:49.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:48.596+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:48.596+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:48.596+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:49.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:49.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:49.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:49.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:49.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:49.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:49.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:49.026+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:49.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:49.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:49.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:49.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:49.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:49.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:49.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:49.096+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:50.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:49.557+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:50.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:49.557+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:49.557+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:49.558+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:50.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:50.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:50.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:50.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:50.033+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:50.144+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:50.144+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:50.144+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:50.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:50.145+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:50.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:50.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:50.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:50.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:50.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:50.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:50.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:50.547+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:51.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:51.102+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:51.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:51.102+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:51.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:51.102+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:51.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:51.103+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:50.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:50.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:50.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:50.986+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:51.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:51.563+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:51.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:51.563+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:51.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:51.563+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:51.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:51.564+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:52.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:52.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:52.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:52.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:52.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:52.096+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:51.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:51.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:51.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:51.976+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:52.784 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:52:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:52:52.663+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 44018 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:52:52.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:52.525+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:52.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:52.525+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:52.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:52.525+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:52.785 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:52.525+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:53.080 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:52.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:53.080 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:52.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:53.080 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:52.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:53.081 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:52.981+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:53.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:53.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:53.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:53.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:53.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:53.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:53.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:53.079+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:53.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:53.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:53.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:53.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:53.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:53.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:53.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:53.544+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:54.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:54.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:54.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:54.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:54.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:54.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:54.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:54.091+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:53.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:53.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:53.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:54.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:53.969+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:55.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:54.590+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:55.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:54.590+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:55.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:54.590+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:55.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:54.590+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:55.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:55.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:55.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:55.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:55.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:55.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:55.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:55.015+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:55.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:55.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:55.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:55.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:55.053+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:56.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:55.571+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:56.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:55.571+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:56.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:55.571+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:56.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:55.572+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:56.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:56.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:56.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:56.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:56.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:56.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:56.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:56.006+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:56.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:56.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:56.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:56.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:56.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:56.029+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:56.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:56.585+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:56.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:56.585+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:56.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:56.585+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:56.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:56.585+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:57.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:57.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:57.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:57.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:57.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:57.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:57.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:57.020+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:57.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:56.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:57.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:56.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:57.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:56.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:57.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:56.969+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:57.977 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:52:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:52:57.669+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 44100 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:52:57.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:57.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:57.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:57.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:57.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:57.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:57.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:57.577+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:58.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:57.976+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:57.976+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:57.976+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:57.977+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:58.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:57.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:58.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:57.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:58.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:57.977+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:58.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:57.978+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:58.976 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:58.605+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:58.976 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:58.605+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:58.976 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:58.605+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:58.976 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:58.605+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:59.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:58.974+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:59.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:58.974+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:59.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:58.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:52:59.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:52:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:52:58.975+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:52:59.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:58.984+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:59.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:58.984+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:59.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:58.984+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:52:59.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:58.984+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:52:59.972 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:59.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:59.972 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:59.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:59.972 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:59.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:52:59.972 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:52:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:52:59.576+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:00.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:59.971+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:00.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:59.971+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:59.971+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:52:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:52:59.972+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:00.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:00.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:00.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:00.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:00.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:00.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:00.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:00.013+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:00.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:00.563+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:00.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:00.563+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:00.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:00.563+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:00.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:00.564+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:01.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:00.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:00.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:00.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:00.966+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:01.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:00.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:01.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:00.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:01.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:00.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:01.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:00.993+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:01.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:01.584+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:01.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:01.584+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:01.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:01.584+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:01.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:01.585+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:02.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:01.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:02.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:01.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:02.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:01.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:02.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:01.983+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:02.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:02.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:02.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:02.014+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:02.835 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:53:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:53:02.675+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 44183 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:53:02.835 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:02.565+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:02.835 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:02.565+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:02.835 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:02.565+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:02.835 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:02.566+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:03.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:03.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:03.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:03.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:03.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:03.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:03.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:03.001+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:03.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:03.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:03.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:03.022+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:03.997 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:03.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:03.998 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:03.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:03.998 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:03.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:03.998 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:03.543+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:04.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:04.011+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:04.011+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:04.011+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:04.012+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:04.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:03.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:04.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:03.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:04.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:03.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:04.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:03.997+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:04.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:04.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:04.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:04.504+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:05.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:05.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:05.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:05.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:05.025+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:04.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:04.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:04.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:05.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:04.982+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:05.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:05.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:05.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:05.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:05.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:05.468+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:06.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:05.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:06.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:05.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:06.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:05.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:06.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:05.988+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:06.041+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:06.041+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:06.041+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:06.041+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:06.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:06.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:06.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:06.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:06.507+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:06.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:06.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:06.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:06.941+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:07.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:07.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:07.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:07.046+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:07.801 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:53:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:53:07.680+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 44267 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:53:07.802 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:07.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:07.802 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:07.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:07.802 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:07.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:07.802 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:07.538+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:08.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:07.965+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:08.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:07.965+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:08.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:07.965+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:08.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:07.965+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:08.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:08.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:08.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:08.049+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:09.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:08.565+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:09.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:08.565+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:09.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:08.565+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:09.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:08.566+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:09.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:09.075+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:09.075+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:09.075+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:09.076+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:09.006+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:09.006+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:09.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:09.006+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:09.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:09.007+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:09.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:09.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:09.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:09.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:09.518+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:10.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:10.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:10.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:10.031+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:10.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:10.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:10.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:10.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:10.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:10.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:10.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:10.028+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:10.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:10.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:10.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:11.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:10.554+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:11.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:11.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:11.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:11.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:11.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:11.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:11.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:11.077+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:11.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:11.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:11.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:11.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:11.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:11.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:11.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:11.059+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:12.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:11.528+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:12.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:11.528+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:12.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:11.528+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:12.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:11.529+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:12.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:12.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:12.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:12.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:12.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:12.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:12.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:12.063+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:12.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:12.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:12.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:12.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:12.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:12.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:12.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:12.077+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:12.769 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:53:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:53:12.686+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 44353 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:53:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:12.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:12.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:12.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:12.511+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:13.125 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:13.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:13.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:13.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:13.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:13.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:13.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:13.014+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:13.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:13.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:13.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:13.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:13.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:13.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:13.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:13.125+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:13.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:13.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:13.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:13.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:13.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:13.488+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:13.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:13.489+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:14.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:14.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:14.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:14.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:14.088+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:14.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:14.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:14.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:14.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:14.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:14.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:14.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:14.027+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:14.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:14.522+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:14.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:14.522+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:14.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:14.522+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:14.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:14.524+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:15.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:14.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:14.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:14.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:14.998+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:15.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:15.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:15.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:15.133+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:16.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:15.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:16.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:15.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:15.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:15.540+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:16.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:16.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:16.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:16.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:16.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:16.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:16.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:16.030+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:16.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:16.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:16.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:16.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:16.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:16.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:16.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:16.157+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:16.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:16.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:16.990 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:16.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:16.990 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:16.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:16.990 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:16.583+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:17.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:16.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:17.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:16.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:16.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:16.989+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:17.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:17.188+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:17.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:17.188+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:17.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:17.188+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:17.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:17.189+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:17.961 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:53:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:53:17.692+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 44444 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:53:17.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:17.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:17.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:17.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:17.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:17.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:17.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:17.595+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:18.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:17.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:17.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:17.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:17.961+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:18.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:18.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:18.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:18.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:18.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:18.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:18.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:18.210+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:18.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:18.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:18.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:18.574+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:18.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:18.574+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:18.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:18.575+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:19.248 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:18.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:19.248 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:18.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:19.248 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:18.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:19.248 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:18.925+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:19.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:19.246+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:19.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:19.246+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:19.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:19.246+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:19.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:19.247+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:19.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:19.552+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:19.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:19.552+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:19.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:19.552+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:19.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:19.553+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:20.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:20.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:20.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:20.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:20.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:20.203+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:19.971+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:19.971+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:19.971+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:19.972+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:20.948 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:20.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:20.948 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:20.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:20.948 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:20.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:20.948 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:20.591+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:21.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:20.946+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:21.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:20.946+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:21.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:20.946+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:21.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:20.947+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:21.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:21.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:21.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:21.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:21.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:21.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:21.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:21.163+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:21.912 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:21.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:21.912 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:21.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:21.912 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:21.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:21.912 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:21.623+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:22.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:21.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:21.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:21.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:21.911+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:22.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:22.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:22.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:22.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:22.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:22.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:22.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:22.122+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:22.907 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:53:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:53:22.698+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 44535 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:53:22.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:22.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:22.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:22.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:22.908 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:22.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:22.908 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:22.617+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:23.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:22.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:22.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:22.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:22.907+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:23.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:23.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:23.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:23.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:23.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:23.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:23.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:23.101+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:23.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:23.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:23.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:23.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:23.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:23.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:23.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:23.660+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:24.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:24.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:24.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:24.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:24.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:24.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:24.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:24.103+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:23.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:23.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:23.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:23.880+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:25.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:24.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:25.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:24.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:25.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:24.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:25.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:24.844+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:25.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:24.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:25.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:24.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:25.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:24.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:25.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:24.706+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:25.106+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:25.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:25.106+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:25.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:25.106+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:25.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:25.106+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:26.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:25.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:25.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:25.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:25.805+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:25.677+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:25.677+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:26.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:25.677+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:26.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:25.678+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:26.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:26.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:26.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:26.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:26.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:26.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:26.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:26.145+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:27.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:26.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:26.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:26.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:26.630+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:27.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:26.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:27.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:26.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:27.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:26.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:27.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:26.844+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:27.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:27.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:27.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:27.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:27.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:27.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:27.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:27.097+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:28.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:53:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:53:27.703+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 44626 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:53:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:27.654+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:27.654+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:27.654+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:27.655+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:28.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:27.808+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:28.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:27.808+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:28.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:27.808+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:28.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:27.808+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:28.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:28.061+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:28.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:28.061+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:28.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:28.061+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:28.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:28.061+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:29.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:28.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:29.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:28.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:29.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:28.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:29.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:28.804+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:28.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:28.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:28.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:29.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:28.698+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:29.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:29.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:29.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:29.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:29.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:29.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:29.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:29.039+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:29.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:29.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:29.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:29.700+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:30.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:29.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:30.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:29.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:30.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:29.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:30.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:29.762+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:30.483 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:30.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:30.483 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:30.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:30.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:30.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:30.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:30.066+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:30.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:30.665+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:30.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:30.665+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:30.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:30.665+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:30.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:30.666+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:31.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:30.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:31.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:30.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:31.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:30.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:31.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:30.753+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:31.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:31.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:31.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:31.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:31.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:31.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:31.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:31.044+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:32.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:31.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:32.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:31.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:32.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:31.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:32.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:31.691+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:31.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:31.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:31.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:31.720+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:32.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:32.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:32.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:32.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:32.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:32.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:32.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:32.085+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:33.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:53:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:53:32.709+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 44717 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:53:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:32.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:32.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:32.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:32.737+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:33.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:32.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:32.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:32.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:32.770+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:33.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:33.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:33.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:33.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:33.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:33.100+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:34.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:33.748+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:34.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:33.748+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:34.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:33.748+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:34.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:33.749+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:34.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:33.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:34.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:33.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:34.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:33.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:34.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:33.788+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:34.475 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:34.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:34.476 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:34.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:34.476 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:34.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:34.476 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:34.101+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:35.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:34.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:35.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:34.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:35.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:34.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:35.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:34.799+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:34.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:34.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:34.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:35.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:34.741+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:35.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:35.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:35.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:35.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:35.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:35.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:35.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:35.063+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:36.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:35.775+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:35.775+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:35.775+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:35.776+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:36.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:35.730+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:36.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:35.730+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:36.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:35.730+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:35.731+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:36.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:36.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:36.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:36.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:36.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:36.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:36.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:36.090+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:37.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:36.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:36.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:36.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:36.779+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:36.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:36.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:36.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:36.718+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:37.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:37.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:37.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:37.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:37.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:37.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:37.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:37.132+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:38.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:53:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:53:37.715+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 44809 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:53:38.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:37.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:38.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:37.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:38.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:37.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:38.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:37.820+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:38.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:37.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:38.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:37.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:38.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:37.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:38.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:37.694+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:38.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:38.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:38.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:38.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:38.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:38.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:38.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:38.146+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:38.794+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:38.794+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:38.794+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:38.795+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:38.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:39.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:38.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:39.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:38.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:39.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:38.659+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:39.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:39.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:39.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:39.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:39.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:39.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:39.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:39.122+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:39.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:39.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:39.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:39.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:39.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:39.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:39.809 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:39.661+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:40.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:39.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:40.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:39.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:40.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:39.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:40.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:39.808+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:40.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:40.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:40.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:40.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:40.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:40.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:40.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:40.159+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:41.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:40.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:40.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:40.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:40.831+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:40.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:40.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:40.669+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:40.670+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:41.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:41.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:41.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:41.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:41.115+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:42.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:41.690+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:41.690+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:41.690+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:41.690+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:42.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:41.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:42.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:41.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:42.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:41.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:42.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:41.852+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:42.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:42.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:42.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:42.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:42.142+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:43.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:53:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:53:42.721+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 44876 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:53:43.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:42.886+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:43.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:42.886+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:43.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:42.886+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:43.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:42.887+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:43.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:42.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:43.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:42.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:43.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:42.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:43.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:42.654+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:43.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:43.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:43.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:43.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:43.137+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:43.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:43.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:43.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:43.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:43.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:43.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:43.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:43.649+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:44.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:43.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:43.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:43.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:43.904+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:44.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:44.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:44.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:44.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:44.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:44.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:44.153+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:44.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:44.625+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:44.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:44.625+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:44.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:44.625+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:44.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:44.626+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:45.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:45.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:45.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:45.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:45.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:45.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:45.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:45.137+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:44.954+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:44.954+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:44.954+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:44.955+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:45.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:45.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:45.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:45.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:45.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:45.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:45.948 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:45.658+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:46.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:45.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:45.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:45.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:45.947+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:46.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:46.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:46.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:46.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:46.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:46.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:46.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:46.122+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:47.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:46.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:46.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:46.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:46.938+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:47.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:46.697+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:47.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:46.697+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:47.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:46.697+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:47.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:46.698+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:47.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:47.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:47.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:47.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:47.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:47.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:47.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:47.134+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:48.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:53:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:53:47.726+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 44922 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:53:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:47.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:47.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:47.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:47.713+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:48.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:47.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:48.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:47.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:48.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:47.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:48.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:47.938+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:48.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:48.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:48.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:48.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:48.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:48.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:48.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:48.127+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:48.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:48.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:48.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:48.672+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:48.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:49.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:48.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:49.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:48.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:49.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:48.900+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:49.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:49.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:49.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:49.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:49.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:49.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:49.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:49.135+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:49.896 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:49.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:49.897 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:49.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:49.897 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:49.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:49.897 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:49.670+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:50.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:49.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:50.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:49.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:50.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:49.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:50.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:49.896+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:50.164+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:50.164+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:50.164+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:50.164+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:50.909 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:50.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:50.909 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:50.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:50.909 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:50.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:50.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:50.627+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:51.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:50.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:51.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:50.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:51.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:50.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:51.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:50.909+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:51.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:51.178+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:51.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:51.178+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:51.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:51.178+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:51.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:51.179+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:51.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:51.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:51.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:51.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:51.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:51.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:51.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:51.861+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:51.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:51.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:51.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:51.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:51.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:51.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:51.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:51.649+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:52.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:52.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:52.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:52.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:52.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:52.182+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:52.911 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:53:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:53:52.732+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 44968 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:53:52.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:52.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:52.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:52.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:52.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:52.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:52.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:52.645+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:53.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:52.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:53.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:52.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:53.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:52.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:53.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:52.911+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:53.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:53.178+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:53.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:53.178+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:53.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:53.178+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:53.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:53.178+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:53.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:53.614+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:53.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:53.614+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:53.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:53.614+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:53.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:53.615+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:54.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:53.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:54.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:53.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:54.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:53.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:54.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:53.958+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:54.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:54.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:54.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:54.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:54.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:54.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:54.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:54.216+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:54.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:54.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:54.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:54.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:54.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:54.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:54.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:54.592+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:55.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:54.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:55.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:54.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:55.236 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:54.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:55.236 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:54.943+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:55.234+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:55.234+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:55.234+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:55.234+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:55.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:55.610+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:55.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:55.610+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:55.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:55.610+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:55.969 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:55.610+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:56.238 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:55.967+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:56.238 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:55.967+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:56.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:55.967+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:56.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:55.968+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:56.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:56.237+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:56.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:56.237+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:56.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:56.237+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:56.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:56.238+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:56.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:56.571+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:56.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:56.571+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:56.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:56.571+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:56.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:56.572+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:57.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:56.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:57.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:56.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:57.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:56.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:57.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:56.971+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:57.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:57.259+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:57.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:57.259+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:57.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:57.259+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:57.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:57.260+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:57.859 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:53:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:53:57.738+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45015 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:53:57.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:57.557+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:57.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:57.557+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:57.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:57.557+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:57.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:57.557+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:58.222 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:57.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:58.223 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:57.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:58.223 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:57.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:58.223 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:57.943+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:58.509 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:58.221+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:58.509 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:58.221+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:58.509 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:58.221+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:58.509 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:58.222+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:58.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:58.508+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:58.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:58.508+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:58.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:58.508+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:58.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:58.509+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:53:59.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:58.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:59.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:58.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:59.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:58.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:53:59.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:58.927+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:59.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:59.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:59.511 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:59.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:59.511 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:59.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:53:59.511 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:53:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:53:59.192+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:53:59.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:59.509+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:59.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:59.509+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:59.509+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:53:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:53:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:53:59.510+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:00.197 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:59.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:00.197 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:59.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:00.197 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:59.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:00.197 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:53:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:53:59.879+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:00.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:00.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:00.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:00.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:00.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:00.196+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:00.908 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:00.523+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:00.908 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:00.523+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:00.908 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:00.523+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:00.908 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:00.524+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:01.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:00.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:01.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:00.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:01.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:00.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:01.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:00.908+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:01.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:01.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:01.491 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:01.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:01.491 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:01.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:01.491 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:01.176+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:01.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:01.489+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:01.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:01.489+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:01.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:01.489+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:01.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:01.490+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:02.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:01.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:02.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:01.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:02.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:01.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:02.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:01.903+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:02.444 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:02.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:02.444 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:02.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:02.444 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:02.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:02.444 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:02.161+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:02.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:02.443+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:02.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:02.443+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:02.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:02.443+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:02.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:02.443+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:03.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:54:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:54:02.744+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45061 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:54:03.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:02.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:03.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:02.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:03.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:02.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:03.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:02.911+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:03.476 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:03.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:03.476 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:03.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:03.477 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:03.133+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:03.477 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:03.133+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:03.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:03.475+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:03.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:03.475+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:03.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:03.475+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:03.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:03.476+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:04.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:03.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:04.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:03.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:04.121 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:03.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:04.121 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:03.868+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:04.515 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:04.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:04.515 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:04.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:04.515 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:04.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:04.515 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:04.119+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:04.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:04.514+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:04.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:04.514+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:04.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:04.514+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:04.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:04.515+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:05.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:04.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:05.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:04.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:05.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:04.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:05.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:04.827+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:05.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:05.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:05.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:05.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:05.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:05.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:05.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:05.118+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:06.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:05.794+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:06.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:05.794+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:06.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:05.794+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:06.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:05.796+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:05.561+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:05.562+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:05.562+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:06.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:05.562+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:06.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:06.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:06.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:06.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:06.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:06.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:06.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:06.110+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:07.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:06.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:07.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:06.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:07.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:06.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:07.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:06.792+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:07.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:06.549+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:07.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:06.549+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:07.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:06.549+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:07.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:06.549+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:07.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:07.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:07.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:07.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:07.087+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:07.813 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:54:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:54:07.749+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45107 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:54:07.813 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:07.541+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:07.813 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:07.541+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:07.813 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:07.541+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:07.814 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:07.541+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:08.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:07.812+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:07.812+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:07.812+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:07.813+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:08.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:08.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:08.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:08.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:08.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:08.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:08.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:08.051+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:08.841 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:08.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:08.841 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:08.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:08.841 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:08.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:08.841 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:08.539+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:09.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:08.839+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:09.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:08.839+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:09.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:08.839+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:09.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:08.840+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:09.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:09.101+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:09.101+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:09.101+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:09.101+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:09.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:09.583+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:09.859 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:09.583+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:09.859 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:09.583+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:09.859 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:09.584+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:10.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:09.857+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:10.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:09.857+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:10.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:09.857+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:10.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:09.858+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:10.419 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:10.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:10.419 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:10.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:10.419 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:10.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:10.419 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:10.151+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:10.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:10.555+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:10.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:10.555+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:10.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:10.555+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:10.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:10.556+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:11.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:10.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:11.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:10.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:11.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:10.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:11.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:10.876+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:11.516 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:11.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:11.516 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:11.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:11.516 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:11.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:11.516 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:11.173+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:11.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:11.515+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:11.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:11.515+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:11.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:11.515+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:11.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:11.515+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:12.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:11.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:12.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:11.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:12.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:11.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:12.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:11.882+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:12.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:12.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:12.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:12.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:12.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:12.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:12.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:12.174+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:12.880 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:54:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:54:12.755+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45153 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:54:12.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:12.565+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:12.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:12.565+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:12.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:12.565+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:12.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:12.566+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:13.186 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:12.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:13.186 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:12.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:13.186 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:12.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:13.186 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:12.880+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:13.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:13.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:13.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:13.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:13.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:13.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:13.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:13.185+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:13.834 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:13.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:13.834 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:13.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:13.834 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:13.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:13.835 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:13.604+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:14.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:13.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:14.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:13.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:14.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:13.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:14.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:13.834+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:14.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:14.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:14.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:14.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:14.168+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:14.877 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:14.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:14.877 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:14.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:14.877 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:14.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:14.877 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:14.608+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:15.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:15.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:15.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:15.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:15.125+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:14.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:14.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:14.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:14.876+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:16.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:15.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:15.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:15.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:15.855+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:16.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:15.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:16.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:15.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:16.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:15.658+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:16.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:15.659+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:16.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:16.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:16.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:16.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:16.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:16.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:16.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:16.132+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:16.838 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:16.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:16.838 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:16.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:16.838 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:16.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:16.838 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:16.694+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:17.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:16.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:17.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:16.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:17.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:16.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:17.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:16.837+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:17.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:17.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:17.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:17.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:17.089+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:18.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:54:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:54:17.762+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45200 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:54:18.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:17.735+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:18.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:17.735+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:18.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:17.735+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:18.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:17.736+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:17.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:17.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:17.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:17.799+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:18.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:18.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:18.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:18.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:18.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:18.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:18.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:18.114+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:19.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:18.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:19.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:18.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:19.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:18.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:19.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:18.750+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:19.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:18.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:19.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:18.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:19.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:18.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:19.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:18.726+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:19.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:19.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:19.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:19.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:19.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:19.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:19.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:19.069+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:20.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:19.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:20.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:19.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:20.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:19.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:20.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:19.727+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:20.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:19.683+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:20.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:19.683+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:20.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:19.683+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:20.025 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:19.684+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:20.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:20.117+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:20.117+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:20.117+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:20.118+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:21.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:20.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:21.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:20.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:21.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:20.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:21.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:20.744+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:20.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:21.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:20.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:21.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:20.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:21.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:20.693+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:21.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:21.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:21.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:21.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:21.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:21.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:21.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:21.156+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:22.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:21.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:21.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:21.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:21.793+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:22.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:21.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:22.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:21.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:22.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:21.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:22.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:21.739+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:22.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:22.147+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:22.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:22.147+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:22.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:22.147+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:22.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:22.148+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:23.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:54:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:54:22.767+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45246 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:54:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:22.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:22.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:22.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:22.719+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:23.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:22.805+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:23.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:22.805+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:23.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:22.805+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:23.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:22.806+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:23.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:23.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:23.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:23.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:23.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:23.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:23.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:23.122+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:24.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:23.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:24.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:23.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:24.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:23.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:24.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:23.780+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:24.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:23.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:24.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:23.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:24.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:23.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:24.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:23.671+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:24.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:24.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:24.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:24.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:24.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:24.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:24.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:24.113+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:24.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:24.768+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:24.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:24.769+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:24.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:24.769+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:24.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:24.769+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:24.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:24.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:24.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:24.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:24.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:24.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:24.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:24.707+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:25.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:25.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:25.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:25.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:25.113+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:26.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:25.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:25.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:25.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:25.728+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:25.818+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:25.818+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:25.818+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:26.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:25.818+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:26.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:26.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:26.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:26.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:26.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:26.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:26.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:26.111+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:27.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:26.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:26.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:26.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:26.814+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:27.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:26.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:27.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:26.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:27.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:26.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:27.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:26.750+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:27.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:27.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:27.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:27.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:27.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:27.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:27.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:27.124+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:28.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:54:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:54:27.773+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45292 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:54:28.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:27.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:28.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:27.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:28.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:27.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:28.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:27.732+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:27.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:28.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:27.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:28.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:27.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:28.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:27.781+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:28.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:28.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:28.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:28.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:28.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:28.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:28.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:28.103+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:29.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:28.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:29.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:28.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:29.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:28.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:29.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:28.757+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:28.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:28.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:28.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:29.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:28.768+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:29.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:29.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:29.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:29.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:29.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:29.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:29.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:29.063+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:30.051 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:29.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:30.051 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:29.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:30.051 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:29.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:30.052 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:29.786+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:30.052 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:29.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:30.052 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:29.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:30.052 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:29.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:30.052 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:29.807+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:30.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:30.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:30.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:30.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:30.050+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:31.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:30.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:30.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:30.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:30.826+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:30.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:30.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:30.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:30.829+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:31.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:31.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:31.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:31.069+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:32.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:31.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:32.104 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:31.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:32.104 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:31.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:32.104 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:31.808+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:32.104 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:31.782+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:32.104 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:31.782+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:32.104 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:31.782+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:32.105 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:31.783+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:32.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:32.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:32.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:32.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:32.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:32.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:32.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:32.103+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:33.095 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:54:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:54:32.779+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45338 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:54:33.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:32.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:33.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:32.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:33.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:32.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:33.096 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:32.780+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:33.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:32.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:33.097 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:32.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:33.097 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:32.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:33.097 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:32.831+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:33.094+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:33.094+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:33.094+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:33.094+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:34.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:33.820+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:34.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:33.820+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:34.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:33.820+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:34.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:33.821+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:34.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:33.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:34.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:33.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:34.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:33.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:34.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:33.809+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:34.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:34.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:34.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:34.124+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:35.053 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:34.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:35.053 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:34.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:35.053 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:34.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:35.053 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:34.797+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:35.054 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:34.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:35.054 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:34.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:35.054 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:34.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:35.054 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:34.804+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:35.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:35.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:35.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:35.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:35.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:35.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:35.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:35.098+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:36.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:35.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:36.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:35.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:36.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:35.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:36.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:35.760+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:36.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:35.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:36.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:35.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:36.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:35.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:36.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:35.762+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:36.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:36.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:36.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:36.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:36.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:36.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:36.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:36.064+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:37.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:36.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:36.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:36.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:36.788+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:36.747+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:36.747+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:37.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:36.747+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:37.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:36.748+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:37.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:37.031+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:37.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:37.031+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:37.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:37.031+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:37.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:37.032+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:38.049 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:54:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:54:37.785+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45385 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:54:38.049 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:37.794+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:38.049 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:37.794+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:38.049 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:37.794+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:38.050 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:37.795+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:38.050 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:37.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:38.050 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:37.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:38.050 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:37.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:38.050 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:37.813+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:38.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:38.047+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:38.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:38.047+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:38.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:38.047+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:38.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:38.048+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:39.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:38.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:39.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:38.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:39.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:38.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:39.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:38.798+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:39.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:38.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:39.043 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:38.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:39.043 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:38.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:39.043 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:38.785+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:39.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:39.040+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:39.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:39.040+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:39.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:39.040+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:39.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:39.041+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:40.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:39.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:40.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:39.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:40.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:39.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:40.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:39.764+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:40.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:39.752+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:40.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:39.752+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:40.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:39.752+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:40.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:39.753+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:40.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:40.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:40.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:40.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:40.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:40.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:40.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:40.068+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:41.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:40.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:40.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:40.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:40.750+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:40.775+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:40.775+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:41.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:40.775+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:41.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:40.776+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:41.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:41.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:41.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:41.084+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:41.085+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:42.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:41.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:41.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:41.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:41.741+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:42.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:41.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:42.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:41.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:42.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:41.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:42.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:41.786+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:42.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:42.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:42.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:42.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:42.051+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:43.060 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:54:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:54:42.791+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45431 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:54:43.060 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:42.781+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:43.061 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:42.781+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:43.061 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:42.781+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:43.061 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:42.782+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:43.061 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:42.789+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:43.061 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:42.789+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:43.061 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:42.789+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:43.061 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:42.790+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:43.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:43.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:43.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:43.059+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:44.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:43.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:44.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:43.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:44.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:43.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:44.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:43.765+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:44.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:43.769+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:44.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:43.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:44.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:43.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:44.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:43.770+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:44.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:44.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:44.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:44.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:44.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:44.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:44.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:44.046+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:45.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:44.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:44.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:44.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:44.779+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:45.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:44.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:45.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:44.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:45.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:44.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:45.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:44.751+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:45.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:45.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:45.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:45.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:45.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:45.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:45.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:45.037+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:46.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:45.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:46.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:45.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:46.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:45.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:46.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:45.772+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:46.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:45.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:46.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:45.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:46.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:45.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:46.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:45.718+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:46.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:46.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:46.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:46.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:46.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:46.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:46.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:46.021+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:46.747+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:46.747+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:46.747+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:47.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:46.747+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:47.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:46.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:47.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:46.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:47.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:46.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:47.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:46.761+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:47.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:47.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:47.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:47.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:47.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:47.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:47.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:47.066+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:48.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:54:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:54:47.797+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45477 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:54:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:47.743+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:47.743+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:47.743+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:47.744+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:47.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:47.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:47.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:47.777+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:48.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:48.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:48.544 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:48.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:48.544 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:48.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:48.545 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:48.065+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:49.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:48.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:48.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:48.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:48.784+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:48.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:48.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:49.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:48.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:49.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:48.753+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:49.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:49.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:49.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:49.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:49.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:49.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:49.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:49.091+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:49.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:49.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:49.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:49.793+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:50.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:49.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:50.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:49.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:50.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:49.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:50.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:49.764+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:50.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:50.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:50.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:50.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:50.128+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:51.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:50.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:50.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:50.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:50.755+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:51.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:50.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:51.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:50.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:51.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:50.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:51.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:50.759+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:51.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:51.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:51.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:51.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:51.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:51.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:51.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:51.099+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:52.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:51.763+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:51.763+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:51.763+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:51.765+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:52.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:51.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:52.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:51.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:52.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:51.762+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:52.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:51.763+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:52.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:52.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:52.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:52.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:52.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:52.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:52.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:52.126+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:53.110 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:54:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:54:52.802+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45523 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:54:53.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:52.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:53.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:52.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:53.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:52.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:53.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:52.812+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:53.111 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:52.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:53.111 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:52.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:53.112 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:52.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:53.112 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:52.796+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:53.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:53.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:53.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:53.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:53.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:53.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:53.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:53.110+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:54.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:53.810+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:53.810+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:53.810+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:53.811+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:54.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:53.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:54.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:53.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:54.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:53.778+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:54.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:53.779+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:54.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:54.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:54.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:54.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:54.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:54.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:54.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:54.151+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:54.755+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:54.755+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:54.755+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:54.756+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:55.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:54.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:55.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:54.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:55.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:54.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:55.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:54.795+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:55.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:55.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:55.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:55.160+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:56.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:55.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:56.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:55.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:56.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:55.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:56.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:55.726+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:56.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:55.837+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:56.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:55.837+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:56.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:55.837+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:56.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:55.838+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:56.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:56.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:56.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:56.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:56.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:56.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:56.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:56.161+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:56.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:56.704+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:56.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:56.704+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:56.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:56.704+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:56.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:56.705+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:57.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:56.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:57.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:56.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:57.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:56.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:57.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:56.854+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:57.460 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:57.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:57.461 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:57.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:57.461 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:57.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:57.461 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:57.140+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:57.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:57.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:57.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:57.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:57.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:57.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:57.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:57.702+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:58.141 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:54:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:54:57.809+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45570 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:54:58.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:57.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:58.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:57.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:58.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:57.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:58.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:57.840+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:58.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:58.140+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:58.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:58.140+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:58.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:58.140+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:58.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:58.141+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:58.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:58.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:58.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:54:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:58.673+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:54:59.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:58.801+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:59.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:58.801+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:59.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:58.801+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:54:59.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:58.801+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:54:59.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:59.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:59.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:59.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:59.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:59.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:54:59.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:54:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:54:59.154+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:00.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:59.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:00.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:59.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:00.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:59.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:00.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:54:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:54:59.707+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:59.837+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:59.837+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:00.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:59.837+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:00.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:54:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:54:59.838+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:00.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:00.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:00.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:00.184+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:01.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:00.808+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:01.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:00.808+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:01.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:00.808+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:01.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:00.809+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:01.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:00.673+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:01.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:00.673+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:01.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:00.673+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:01.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:00.674+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:01.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:01.226+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:01.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:01.226+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:01.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:01.226+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:01.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:01.226+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:02.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:01.828+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:02.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:01.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:02.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:01.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:02.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:01.829+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:01.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:01.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:01.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:02.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:01.699+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:02.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:02.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:02.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:02.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:02.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:02.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:02.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:02.180+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:02.938 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:55:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:55:02.815+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45616 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:55:02.938 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:02.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:02.938 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:02.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:02.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:02.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:02.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:02.749+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:02.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:02.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:02.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:02.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:02.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:02.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:02.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:02.813+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:03.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:03.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:03.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:03.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:03.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:03.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:03.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:03.132+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:04.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:03.729+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:04.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:03.729+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:04.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:03.729+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:04.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:03.730+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:03.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:03.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:03.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:04.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:03.804+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:04.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:04.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:04.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:04.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:04.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:04.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:04.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:04.159+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:05.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:04.740+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:05.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:04.740+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:05.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:04.740+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:05.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:04.741+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:05.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:04.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:05.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:04.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:05.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:04.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:05.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:04.854+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:05.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:05.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:05.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:05.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:05.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:05.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:05.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:05.203+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:06.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:05.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:06.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:05.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:06.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:05.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:06.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:05.712+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:05.816+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:06.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:05.816+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:06.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:05.816+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:06.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:05.817+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:06.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:06.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:06.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:06.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:06.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:06.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:06.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:06.184+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:07.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:06.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:07.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:06.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:07.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:06.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:07.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:06.755+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:07.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:06.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:07.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:06.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:07.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:06.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:07.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:06.813+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:07.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:07.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:07.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:07.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:07.175+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:08.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:55:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:55:07.821+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45662 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:55:08.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:07.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:08.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:07.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:08.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:07.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:08.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:07.708+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:08.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:07.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:08.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:07.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:08.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:07.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:08.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:07.812+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:08.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:08.214+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:08.214+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:08.214+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:08.214+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:09.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:08.741+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:09.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:08.741+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:09.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:08.741+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:09.026 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:08.742+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:09.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:08.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:09.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:08.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:09.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:08.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:09.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:08.861+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:09.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:09.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:09.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:09.168+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:09.714+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:09.714+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:09.714+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:09.715+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:10.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:09.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:09.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:09.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:09.820+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:10.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:10.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:10.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:10.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:10.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:10.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:10.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:10.148+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:10.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:10.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:10.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:10.765+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:10.815+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:10.815+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:10.815+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:10.815+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:11.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:11.173+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:11.173+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:11.173+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:11.173+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:12.125 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:11.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:12.125 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:11.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:12.125 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:11.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:12.126 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:11.802+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:12.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:11.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:12.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:11.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:12.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:11.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:12.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:11.842+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:12.127 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:12.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:12.127 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:12.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:12.127 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:12.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:12.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:12.123+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:13.151 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:55:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:55:12.826+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45708 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:55:13.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:12.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:13.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:12.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:13.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:12.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:13.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:12.850+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:13.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:12.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:13.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:12.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:13.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:12.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:13.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:12.890+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:13.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:13.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:13.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:13.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:13.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:13.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:13.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:13.150+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:13.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:13.812+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:13.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:13.812+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:13.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:13.812+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:13.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:13.813+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:14.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:13.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:13.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:13.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:13.920+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:14.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:14.165+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:14.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:14.165+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:14.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:14.165+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:14.165+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:15.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:14.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:15.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:14.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:15.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:14.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:15.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:14.851+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:15.129 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:14.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:15.129 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:14.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:15.129 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:14.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:15.129 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:14.879+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:15.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:15.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:15.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:15.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:15.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:15.127+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:15.474 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:15.127+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:16.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:15.857+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:16.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:15.857+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:16.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:15.857+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:16.133 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:15.857+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:16.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:15.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:16.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:15.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:16.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:15.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:16.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:15.834+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:16.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:16.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:16.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:16.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:16.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:16.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:16.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:16.131+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:17.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:16.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:17.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:16.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:17.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:16.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:17.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:16.843+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:17.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:16.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:17.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:16.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:17.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:16.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:17.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:16.851+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:17.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:17.162+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:17.162+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:17.162+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:17.162+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:18.168 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:55:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:55:17.831+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45755 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:55:18.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:17.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:18.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:17.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:18.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:17.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:18.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:17.799+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:18.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:17.895+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:18.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:17.895+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:18.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:17.895+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:18.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:17.895+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:18.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:18.167+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:18.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:18.167+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:18.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:18.167+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:18.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:18.167+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:19.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:18.824+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:19.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:18.824+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:19.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:18.824+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:19.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:18.825+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:19.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:18.932+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:19.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:18.932+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:19.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:18.932+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:19.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:18.933+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:19.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:19.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:19.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:19.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:19.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:19.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:19.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:19.126+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:20.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:19.937+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:20.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:19.937+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:20.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:19.937+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:20.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:19.937+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:20.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:19.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:20.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:19.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:20.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:19.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:20.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:19.800+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:20.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:20.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:20.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:20.077+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:21.051 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:20.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:21.051 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:20.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:21.051 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:20.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:21.051 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:20.901+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:21.052 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:20.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:21.052 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:20.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:21.052 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:20.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:21.052 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:20.787+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:21.053 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:21.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:21.053 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:21.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:21.053 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:21.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:21.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:21.050+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:22.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:21.745+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:22.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:21.745+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:22.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:21.745+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:22.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:21.745+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:22.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:21.939+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:22.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:21.939+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:22.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:21.939+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:22.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:21.940+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:22.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:22.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:22.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:22.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:22.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:22.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:22.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:22.011+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:23.051 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:55:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:55:22.837+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45801 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:55:23.051 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:22.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:23.052 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:22.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:23.052 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:22.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:23.052 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:22.788+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:23.052 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:22.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:23.052 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:22.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:23.053 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:22.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:23.053 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:22.937+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:23.053 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:23.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:23.053 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:23.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:23.053 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:23.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:23.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:23.050+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:23.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:23.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:23.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:23.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:23.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:23.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:23.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:23.829+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:23.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:23.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:23.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:23.920+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:24.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:24.011+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:24.011+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:24.011+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:24.011+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:25.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:24.824+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:25.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:24.824+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:25.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:24.824+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:25.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:24.825+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:24.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:24.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:24.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:24.902+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:25.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:25.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:25.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:25.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:25.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:25.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:25.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:25.002+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:25.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:25.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:25.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:25.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:25.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:25.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:25.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:25.786+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:25.939+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:25.939+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:25.939+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:25.940+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:26.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:25.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:26.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:25.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:26.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:25.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:26.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:25.975+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:26.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:26.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:26.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:26.820+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:26.976+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:26.976+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:26.976+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:26.977+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:26.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:26.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:26.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:26.931+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:28.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:55:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:55:27.843+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45847 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:55:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:27.918+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:27.918+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:27.918+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:27.919+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:27.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:27.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:27.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:27.798+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:28.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:27.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:28.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:27.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:28.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:27.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:28.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:27.965+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:28.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:28.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:28.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:28.832+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:28.944+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:28.944+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:28.944+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:28.945+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:29.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:28.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:29.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:28.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:29.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:28.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:29.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:28.903+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:30.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:29.822+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:30.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:29.822+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:30.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:29.822+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:30.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:29.823+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:30.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:29.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:30.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:29.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:30.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:29.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:30.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:29.921+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:29.942+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:29.942+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:29.942+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:30.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:29.943+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:30.949 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:30.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:30.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:30.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:30.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:30.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:30.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:30.862+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:30.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:30.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:30.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:30.981+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:31.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:30.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:30.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:30.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:30.948+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:32.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:31.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:32.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:31.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:32.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:31.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:32.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:31.980+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:32.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:31.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:32.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:31.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:32.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:31.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:32.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:31.879+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:31.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:31.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:32.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:31.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:32.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:31.954+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:33.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:55:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:55:32.849+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45893 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:55:33.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:32.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:33.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:32.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:33.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:32.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:32.838+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:33.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:32.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:33.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:32.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:33.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:32.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:33.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:32.983+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:33.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:33.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:33.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:33.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:33.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:33.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:33.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:33.003+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:34.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:33.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:33.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:33.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:33.989+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:34.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:33.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:34.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:33.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:34.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:33.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:34.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:33.982+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:33.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:33.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:33.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:33.856+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:35.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:34.956+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:35.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:34.956+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:35.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:34.956+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:35.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:34.957+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:34.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:34.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:34.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:34.904+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:35.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:35.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:35.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:35.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:35.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:35.013+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:36.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:35.932+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:36.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:35.932+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:36.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:35.932+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:36.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:35.932+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:36.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:35.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:36.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:35.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:36.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:35.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:36.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:35.927+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:36.012+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:36.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:36.012+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:36.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:36.012+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:36.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:36.013+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:36.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:36.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:36.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:36.938+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:37.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:37.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:37.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:37.040+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:36.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:36.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:36.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:36.920+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:38.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:55:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:55:37.855+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45940 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:55:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:37.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:37.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:37.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:38.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:37.950+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:37.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:37.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:37.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:37.926+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:38.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:38.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:38.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:38.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:38.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:38.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:38.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:38.037+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:38.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:38.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:38.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:38.972+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:39.085+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:39.085+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:39.085+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:39.086+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:38.911+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:38.911+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:38.911+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:39.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:38.911+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:40.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:39.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:40.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:39.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:40.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:39.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:40.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:39.993+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:40.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:40.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:40.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:40.076+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:40.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:39.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:40.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:39.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:40.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:39.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:40.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:39.932+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:41.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:40.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:41.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:40.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:41.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:40.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:41.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:40.967+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:41.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:41.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:41.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:41.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:41.092+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:41.013+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:41.013+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:41.013+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:41.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:41.015+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:42.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:42.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:42.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:42.097+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:41.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:41.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:41.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:41.990+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:42.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:42.003+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:42.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:42.003+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:42.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:42.003+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:42.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:42.003+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:43.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:55:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:55:42.861+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 45986 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:55:43.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:43.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:43.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:43.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:43.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:43.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:43.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:43.019+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:43.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:43.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:43.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:43.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:43.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:43.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:43.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:43.079+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:43.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:43.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:43.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:43.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:43.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:43.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:43.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:43.017+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:44.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:44.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:44.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:44.062+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:44.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:44.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:44.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:44.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:44.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:44.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:44.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:44.059+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:44.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:43.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:44.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:43.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:44.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:43.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:44.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:43.998+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:45.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:45.103+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:45.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:45.103+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:45.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:45.103+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:45.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:45.104+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:45.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:45.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:45.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:45.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:45.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:45.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:45.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:45.087+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:45.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:45.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:45.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:45.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:45.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:45.045+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:45.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:45.046+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:46.148+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:46.148+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:46.148+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:46.148+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:46.080+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:46.080+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:46.080+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:46.081+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:46.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:46.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:46.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:46.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:46.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:46.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:46.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:46.016+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:47.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:47.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:47.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:47.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:47.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:47.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:47.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:47.070+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:47.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:47.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:47.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:47.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:47.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:47.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:47.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:47.175+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:47.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:47.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:47.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:47.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:47.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:47.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:47.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:47.037+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:48.158 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:55:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:55:47.866+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46032 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:55:48.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:48.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:48.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:48.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:48.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:48.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:48.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:48.028+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:48.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:47.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:48.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:47.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:48.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:47.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:48.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:47.988+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:48.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:48.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:48.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:48.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:48.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:48.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:48.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:48.157+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:49.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:49.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:49.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:49.110+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:49.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:49.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:49.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:49.005+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:49.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:48.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:49.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:48.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:49.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:48.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:49.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:48.941+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:50.130+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:50.130+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:50.131+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:50.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:50.131+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:50.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:50.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:50.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:50.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:50.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:50.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:50.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:50.029+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:50.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:49.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:50.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:49.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:50.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:49.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:50.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:49.963+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:51.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:51.049+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:51.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:51.049+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:51.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:51.049+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:51.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:51.049+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:51.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:50.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:51.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:50.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:51.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:50.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:51.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:50.989+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:51.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:51.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:51.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:51.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:51.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:51.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:51.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:51.160+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:52.085 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:52.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:52.085 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:52.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:52.085 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:52.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:52.085 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:52.036+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:52.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:52.188+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:52.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:52.189+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:52.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:52.189+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:52.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:52.189+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:52.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:52.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:52.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:52.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:52.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:52.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:52.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:52.084+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:53.160 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:55:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:55:52.872+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46078 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:55:53.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:53.071+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:53.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:53.071+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:53.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:53.071+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:53.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:53.072+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:53.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:53.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:53.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:53.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:53.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:53.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:53.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:53.045+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:53.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:53.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:53.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:53.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:53.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:53.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:53.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:53.159+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:54.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:54.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:54.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:54.036+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:54.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:54.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:54.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:54.042+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:54.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:54.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:54.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:54.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:54.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:54.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:54.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:54.137+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:55.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:55.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:55.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:55.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:55.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:55.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:55.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:55.069+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:55.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:55.034+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:55.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:55.034+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:55.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:55.034+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:55.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:55.035+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:55.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:55.139+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:55.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:55.139+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:55.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:55.139+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:55.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:55.140+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:56.124+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:56.124+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:56.124+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:56.125+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:56.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:56.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:56.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:56.001+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:56.102+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:56.102+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:56.102+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:56.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:56.102+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:57.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:57.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:57.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:57.156+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:57.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:57.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:57.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:57.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:57.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:57.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:57.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:57.105+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:57.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:56.970+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:57.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:56.970+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:57.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:56.970+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:57.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:56.970+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:58.155 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:55:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:55:57.878+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46125 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:55:58.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:58.123+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:58.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:58.123+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:58.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:58.123+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:58.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:58.124+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:58.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:57.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:58.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:57.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:58.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:57.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:58.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:57.942+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:58.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:58.153+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:58.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:58.153+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:58.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:58.153+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:58.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:58.153+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:59.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:59.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:59.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:59.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:59.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:59.149+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:55:59.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:55:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:55:59.150+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:55:59.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:59.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:59.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:59.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:59.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:59.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:55:59.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:55:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:55:59.153+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:55:59.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:58.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:59.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:58.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:59.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:58.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:55:59.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:58.944+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:00.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:59.917+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:00.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:59.917+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:00.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:59.917+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:00.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:55:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:55:59.918+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:00.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:00.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:00.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:00.178+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:00.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:00.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:00.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:00.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:00.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:00.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:00.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:00.179+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:01.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:01.132+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:01.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:01.132+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:01.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:01.132+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:01.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:01.133+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:01.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:00.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:01.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:00.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:01.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:00.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:01.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:00.890+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:01.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:01.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:01.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:01.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:01.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:01.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:01.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:01.181+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:02.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:01.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:02.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:01.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:02.130 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:01.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:02.130 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:01.872+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:02.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:02.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:02.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:02.129+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:02.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:02.171+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:02.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:02.171+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:02.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:02.171+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:02.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:02.171+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:03.140 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:56:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:56:02.884+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46171 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:56:03.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:02.914+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:03.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:02.914+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:03.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:02.914+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:03.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:02.914+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:03.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:03.142+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:03.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:03.142+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:03.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:03.142+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:03.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:03.143+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:03.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:03.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:03.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:03.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:03.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:03.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:03.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:03.139+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:04.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:04.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:04.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:04.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:04.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:04.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:04.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:04.109+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:04.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:04.161+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:04.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:04.161+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:04.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:04.161+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:04.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:04.162+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:03.939+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:03.939+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:03.939+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:04.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:03.940+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:05.067+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:05.067+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:05.067+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:05.068+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:05.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:05.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:05.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:05.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:05.155+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:05.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:04.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:05.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:04.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:05.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:04.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:05.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:04.963+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:06.058+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:06.058+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:06.058+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:06.059+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:06.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:06.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:06.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:06.014+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:06.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:06.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:06.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:06.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:06.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:06.124+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:07.108+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:07.108+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:07.108+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:07.108+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:07.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:07.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:07.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:07.013+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:07.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:07.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:07.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:07.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:07.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:07.159+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:08.153 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:56:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:56:07.891+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46217 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:56:08.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:08.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:08.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:08.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:08.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:08.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:08.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:08.122+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:08.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:08.005+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:08.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:08.005+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:08.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:08.005+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:08.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:08.006+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:08.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:08.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:08.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:08.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:08.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:08.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:08.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:08.153+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:09.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:09.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:09.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:09.004+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:09.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:09.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:09.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:09.118+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:09.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:09.185+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:09.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:09.185+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:09.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:09.185+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:09.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:09.185+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:10.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:10.120+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:10.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:10.120+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:10.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:10.120+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:10.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:10.121+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:10.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:10.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:10.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:10.139+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:10.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:09.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:10.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:09.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:10.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:09.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:10.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:09.988+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:11.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:11.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:11.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:11.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:11.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:11.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:11.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:11.145+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:11.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:11.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:11.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:11.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:11.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:11.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:11.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:11.149+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:11.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:11.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:11.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:11.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:11.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:11.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:11.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:11.026+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:12.480 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:12.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:12.481 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:12.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:12.481 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:12.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:12.481 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:12.112+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:12.481 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:12.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:12.481 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:12.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:12.482 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:12.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:12.482 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:12.141+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:12.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:12.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:12.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:12.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:12.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:12.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:12.482 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:12.019+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:13.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:56:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:56:12.897+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46263 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:56:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:13.132+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:13.132+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:13.132+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:13.133+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:13.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:13.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:13.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:13.114+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:13.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:13.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:13.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:13.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:13.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:13.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:13.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:13.020+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:14.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:14.127+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:14.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:14.127+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:14.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:14.127+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:14.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:14.128+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:14.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:14.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:14.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:14.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:14.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:14.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:14.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:14.063+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:14.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:14.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:14.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:14.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:14.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:14.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:14.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:14.084+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:15.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:15.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:15.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:15.086+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:15.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:15.110+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:15.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:15.110+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:15.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:15.110+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:15.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:15.111+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:15.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:15.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:15.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:15.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:15.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:15.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:15.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:15.107+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:16.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:16.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:16.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:16.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:16.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:16.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:16.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:16.158+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:16.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:16.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:16.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:16.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:16.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:16.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:16.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:16.059+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:16.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:16.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:16.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:16.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:16.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:16.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:16.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:16.090+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:17.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:17.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:17.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:17.054+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:17.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:17.123+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:17.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:17.123+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:17.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:17.123+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:17.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:17.124+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:17.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:17.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:17.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:17.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:17.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:17.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:17.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:17.071+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:18.155 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:56:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:56:17.903+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46310 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:56:18.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:18.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:18.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:18.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:18.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:18.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:18.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:18.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:18.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:18.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:18.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:18.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:18.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:18.055+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:18.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:18.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:18.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:18.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:18.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:18.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:18.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:18.032+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:18.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:18.154+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:19.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:19.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:19.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:19.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:19.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:19.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:19.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:19.029+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:19.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:19.148+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:19.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:19.148+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:19.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:19.148+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:19.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:19.149+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:19.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:19.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:19.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:19.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:19.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:19.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:19.043+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:20.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:20.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:20.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:20.042+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:20.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:20.182+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:20.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:20.182+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:20.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:20.182+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:20.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:20.183+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:20.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:19.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:20.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:19.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:20.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:19.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:20.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:19.982+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:21.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:21.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:21.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:21.050+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:21.166+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:21.166+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:21.166+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:21.168+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:20.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:20.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:20.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:21.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:20.962+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:22.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:21.939+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:22.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:21.939+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:22.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:21.939+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:22.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:21.939+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:22.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:22.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:22.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:22.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:22.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:22.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:22.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:22.009+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:22.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:22.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:22.516 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:22.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:22.516 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:22.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:22.516 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:22.204+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:23.182 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:56:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:56:22.909+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46356 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:56:23.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:22.939+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:23.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:22.939+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:23.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:22.939+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:23.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:22.940+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:23.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:22.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:23.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:22.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:23.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:22.989+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:23.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:22.989+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:23.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:23.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:23.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:23.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:23.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:23.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:23.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:23.182+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:24.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:24.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:24.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:24.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:24.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:24.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:24.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:24.152+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:23.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:23.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:23.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:23.946+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:24.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:23.976+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:24.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:23.976+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:24.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:23.976+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:24.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:23.977+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:25.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:25.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:25.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:25.025+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:25.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:24.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:25.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:24.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:25.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:24.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:25.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:24.909+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:25.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:25.131+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:25.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:25.131+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:25.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:25.131+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:25.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:25.132+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:26.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:26.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:26.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:26.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:26.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:26.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:26.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:26.053+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:26.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:25.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:26.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:25.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:26.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:25.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:26.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:25.900+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:26.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:26.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:26.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:26.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:26.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:26.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:26.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:26.174+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:27.206 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:27.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:27.207 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:27.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:27.207 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:27.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:27.207 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:27.005+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:27.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:26.923+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:27.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:26.923+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:27.207 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:26.923+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:27.208 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:26.924+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:27.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:27.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:27.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:27.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:27.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:27.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:27.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:27.206+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:28.186 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:56:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:56:27.914+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46402 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:56:28.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:27.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:28.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:27.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:28.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:27.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:28.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:27.921+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:28.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:28.025+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:28.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:28.025+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:28.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:28.025+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:28.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:28.026+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:28.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:28.185+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:28.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:28.185+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:28.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:28.185+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:28.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:28.185+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:28.961+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:28.961+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:28.961+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:28.961+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:29.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:29.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:29.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:29.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:29.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:29.180+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:29.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:29.180+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:29.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:29.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:29.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:29.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:29.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:29.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:29.060+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:30.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:29.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:30.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:29.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:30.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:29.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:30.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:29.914+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:30.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:30.148+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:30.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:30.148+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:30.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:30.148+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:30.275 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:30.149+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:30.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:30.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:30.275 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:30.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:30.276 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:30.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:30.276 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:30.052+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:31.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:31.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:31.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:31.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:31.240 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:31.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:31.240 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:31.182+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:31.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:31.042+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:31.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:31.042+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:31.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:31.042+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:31.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:31.043+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:31.241 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:30.959+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:31.241 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:30.959+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:31.241 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:30.959+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:31.241 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:30.959+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:31.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:31.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:31.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:31.963+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:32.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:32.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:32.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:32.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:32.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:32.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:32.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:32.038+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:32.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:32.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:32.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:32.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:32.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:32.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:32.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:32.154+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:33.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:56:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:56:32.921+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46448 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:56:33.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:33.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:33.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:33.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:33.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:33.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:33.110+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:33.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:33.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:33.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:33.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:33.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:33.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:33.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:33.020+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:33.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:32.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:33.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:32.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:33.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:32.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:33.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:32.946+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:34.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:34.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:34.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:34.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:34.025+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:34.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:34.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:34.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:34.119+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:34.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:33.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:34.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:33.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:34.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:33.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:34.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:33.936+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:35.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:35.033+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:35.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:35.033+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:35.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:35.033+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:35.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:35.034+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:35.084 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:34.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:35.084 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:34.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:35.084 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:34.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:35.084 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:34.904+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:35.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:35.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:35.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:35.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:35.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:35.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:35.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:35.082+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:36.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:36.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:36.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:36.076+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:35.984+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:35.984+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:35.984+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:35.985+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:36.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:35.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:36.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:35.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:36.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:35.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:36.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:35.869+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:37.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:37.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:37.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:37.057+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:36.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:36.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:36.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:36.995+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:36.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:36.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:36.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:36.893+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:38.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:56:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:56:37.926+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46495 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:56:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:38.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:38.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:38.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:38.074+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:37.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:37.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:37.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:37.966+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:38.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:37.923+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:38.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:37.923+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:38.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:37.923+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:38.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:37.924+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:38.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:38.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:38.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:38.955+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:38.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:38.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:38.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:38.948+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:39.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:39.095+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:39.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:39.095+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:39.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:39.095+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:39.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:39.096+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:40.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:40.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:40.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:40.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:40.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:40.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:40.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:40.048+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:39.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:39.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:39.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:39.967+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:40.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:39.927+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:40.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:39.927+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:40.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:39.927+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:40.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:39.928+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:41.096+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:41.096+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:41.096+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:41.097+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:40.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:41.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:40.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:41.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:40.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:41.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:40.936+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:41.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:40.951+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:41.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:40.951+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:41.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:40.951+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:41.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:40.952+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:42.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:42.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:42.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:42.067+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:41.922+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:41.922+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:41.922+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:41.923+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:41.914+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:41.914+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:41.914+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:42.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:41.914+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:43.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:56:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:56:42.933+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46541 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:56:43.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:43.060+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:43.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:43.060+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:43.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:43.060+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:43.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:43.061+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:43.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:42.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:43.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:42.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:43.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:42.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:43.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:42.881+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:43.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:42.929+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:43.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:42.929+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:43.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:42.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:43.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:42.930+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:44.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:44.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:44.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:44.109+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:44.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:43.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:44.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:43.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:44.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:43.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:44.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:43.862+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:43.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:43.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:43.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:44.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:43.905+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:45.145 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:44.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:45.146 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:44.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:45.146 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:44.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:45.146 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:44.878+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:45.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:44.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:45.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:44.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:45.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:44.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:45.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:44.883+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:45.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:45.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:45.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:45.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:45.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:45.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:45.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:45.143+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:46.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:45.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:46.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:45.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:46.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:45.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:46.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:45.887+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:46.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:45.844+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:46.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:45.844+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:46.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:45.844+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:46.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:45.845+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:46.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:46.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:46.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:46.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:46.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:46.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:46.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:46.155+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:47.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:46.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:47.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:46.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:47.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:46.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:47.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:46.842+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:47.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:46.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:47.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:46.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:47.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:46.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:47.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:46.851+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:47.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:47.167+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:47.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:47.167+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:47.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:47.167+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:47.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:47.168+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:48.156 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:56:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:56:47.938+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46586 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:56:48.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:47.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:48.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:47.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:48.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:47.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:48.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:47.884+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:48.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:47.842+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:48.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:47.842+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:48.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:47.842+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:48.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:47.842+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:48.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:48.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:48.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:48.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:48.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:48.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:48.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:48.156+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:49.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:48.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:49.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:48.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:49.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:48.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:49.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:48.885+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:49.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:48.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:49.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:48.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:49.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:48.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:49.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:48.819+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:49.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:49.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:49.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:49.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:49.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:49.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:49.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:49.141+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:50.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:49.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:50.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:49.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:50.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:49.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:50.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:49.817+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:50.135 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:49.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:50.135 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:49.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:50.135 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:49.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:50.135 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:49.920+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:50.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:50.132+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:50.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:50.132+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:50.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:50.132+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:50.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:50.133+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:51.116 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:50.918+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:51.116 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:50.918+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:51.116 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:50.918+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:51.117 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:50.919+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:51.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:50.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:51.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:50.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:51.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:50.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:51.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:50.838+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:51.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:51.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:51.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:51.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:51.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:51.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:51.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:51.115+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:52.114 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:51.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:52.114 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:51.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:52.114 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:51.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:52.114 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:51.910+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:52.115 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:51.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:52.115 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:51.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:52.115 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:51.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:52.115 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:51.831+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:52.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:52.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:52.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:52.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:52.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:52.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:52.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:52.113+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:53.063 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:56:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:56:52.945+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46633 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:56:53.063 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:52.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:53.063 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:52.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:53.063 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:52.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:53.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:52.942+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:53.064 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:52.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:53.064 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:52.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:53.064 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:52.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:53.064 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:52.811+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:53.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:53.128+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:53.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:53.128+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:53.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:53.128+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:53.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:53.129+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:54.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:53.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:54.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:53.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:54.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:53.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:54.127 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:53.839+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:54.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:53.983+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:54.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:53.983+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:54.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:53.983+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:54.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:53.984+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:54.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:54.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:54.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:54.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:54.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:54.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:54.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:54.125+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:55.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:55.102+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:55.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:55.102+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:55.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:55.102+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:55.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:55.103+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:54.992+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:54.992+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:54.992+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:54.993+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:55.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:54.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:55.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:54.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:55.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:54.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:55.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:54.880+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:56.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:56.064+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:56.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:56.064+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:56.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:56.064+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:56.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:56.065+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:56.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:55.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:56.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:55.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:56.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:55.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:56.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:55.848+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:56.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:55.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:56.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:55.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:56.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:55.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:56.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:55.955+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:57.078 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:56.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:57.079 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:56.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:57.079 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:56.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:57.079 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:56.820+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:57.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:56.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:57.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:56.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:57.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:56.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:57.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:56.985+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:57.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:57.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:57.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:57.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:57.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:57.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:57.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:57.078+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:58.070 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:56:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:56:57.951+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46679 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:56:58.071 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:57.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:58.071 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:57.946+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:58.071 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:57.946+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:58.071 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:57.947+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:58.072 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:57.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:58.072 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:57.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:58.072 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:57.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:58.072 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:57.771+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:58.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:58.104+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:58.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:58.104+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:58.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:58.104+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:58.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:58.105+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:59.066 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:58.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:59.066 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:58.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:59.066 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:58.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:56:59.066 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:58.921+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:56:59.067 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:58.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:59.067 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:58.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:59.067 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:58.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:56:59.067 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:58.809+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:56:59.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:59.064+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:59.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:59.064+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:59.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:59.064+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:56:59.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:56:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:56:59.065+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:59.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:59.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:59.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:56:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:56:59.969+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:00.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:59.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:00.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:59.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:00.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:59.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:00.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:56:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:56:59.855+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:00.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:00.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:00.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:00.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:00.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:00.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:00.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:00.032+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:01.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:01.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:01.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:01.032+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:00.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:00.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:00.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:00.995+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:01.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:00.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:01.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:00.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:01.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:00.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:01.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:00.841+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:02.065+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:02.065+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:02.065+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:02.066+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:01.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:01.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:01.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:01.969+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:02.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:01.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:02.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:01.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:02.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:01.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:02.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:01.880+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:03.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:57:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:57:02.957+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46726 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:57:03.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:03.107+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:03.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:03.107+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:03.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:03.107+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:03.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:03.108+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:02.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:02.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:02.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:02.949+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:03.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:02.902+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:03.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:02.902+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:03.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:02.902+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:03.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:02.902+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:03.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:03.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:03.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:03.982+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:03.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:03.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:03.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:03.910+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:04.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:04.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:04.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:04.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:04.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:04.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:04.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:04.089+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:05.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:05.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:05.114+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:05.115+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:04.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:04.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:04.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:04.999+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:05.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:04.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:05.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:04.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:05.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:04.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:05.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:04.867+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:06.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:06.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:06.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:06.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:06.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:06.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:06.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:06.026+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:06.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:05.843+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:06.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:05.843+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:06.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:05.843+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:06.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:05.844+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:06.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:06.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:06.153+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:06.154+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:07.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:06.817+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:07.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:06.817+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:07.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:06.817+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:07.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:06.817+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:07.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:07.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:07.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:07.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:07.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:07.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:07.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:07.148+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:07.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:07.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:07.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:07.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:07.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:07.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:07.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:07.075+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:08.115 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:57:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:57:07.963+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46771 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:57:08.115 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:08.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:08.115 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:08.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:08.116 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:08.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:08.116 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:08.061+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:08.116 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:07.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:08.116 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:07.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:08.116 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:07.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:08.116 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:07.854+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:08.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:08.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:08.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:08.114+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:09.133 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:09.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:09.133 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:09.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:09.133 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:09.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:09.133 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:09.087+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:09.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:08.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:09.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:08.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:09.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:08.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:09.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:08.859+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:09.131+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:09.131+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:09.131+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:09.132+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:10.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:09.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:10.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:09.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:10.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:09.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:10.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:09.813+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:10.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:10.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:10.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:10.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:10.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:10.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:10.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:10.155+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:10.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:10.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:10.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:10.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:10.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:10.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:10.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:10.120+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:11.130 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:10.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:11.130 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:10.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:11.131 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:10.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:11.131 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:10.821+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:11.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:11.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:11.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:11.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:11.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:11.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:11.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:11.134+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:11.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:11.129+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:11.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:11.129+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:11.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:11.129+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:11.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:11.130+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:12.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:11.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:12.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:11.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:12.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:11.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:12.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:11.803+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:12.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:12.098+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:12.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:12.098+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:12.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:12.098+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:12.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:12.099+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:12.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:12.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:12.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:12.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:12.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:12.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:12.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:12.109+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:13.090 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:57:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:57:12.968+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46818 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:57:13.090 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:13.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:13.090 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:13.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:13.091 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:13.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:13.091 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:13.077+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:13.091 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:12.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:13.091 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:12.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:13.091 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:12.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:13.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:12.830+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:13.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:13.141+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:13.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:13.141+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:13.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:13.141+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:13.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:13.142+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:14.042+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:14.042+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:14.042+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:14.043+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:13.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:13.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:13.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:13.858+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:14.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:14.095+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:14.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:14.095+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:14.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:14.095+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:14.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:14.096+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:15.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:14.884+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:15.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:14.884+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:15.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:14.884+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:15.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:14.885+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:15.061+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:15.061+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:15.061+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:15.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:15.062+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:15.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:15.052+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:15.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:15.052+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:15.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:15.052+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:15.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:15.053+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:16.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:16.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:16.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:16.080+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:16.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:16.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:16.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:16.037+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:15.862+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:16.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:15.862+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:16.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:15.862+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:16.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:15.862+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:16.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:16.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:16.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:16.901+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:17.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:17.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:17.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:17.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:17.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:17.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:17.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:17.054+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:17.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:17.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:17.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:17.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:17.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:17.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:17.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:17.066+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:18.110 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:57:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:57:17.974+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46864 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:57:18.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:18.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:18.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:18.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:18.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:18.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:18.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:18.052+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:18.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:17.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:18.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:17.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:18.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:17.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:18.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:17.855+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:18.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:18.109+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:18.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:18.109+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:18.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:18.109+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:18.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:18.109+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:19.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:19.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:19.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:19.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:19.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:19.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:19.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:19.031+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:19.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:18.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:19.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:18.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:19.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:18.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:19.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:18.880+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:19.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:19.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:19.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:19.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:19.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:19.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:19.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:19.140+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:20.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:19.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:20.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:19.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:20.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:19.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:20.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:19.887+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:20.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:20.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:20.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:20.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:20.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:20.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:20.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:20.048+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:20.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:20.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:20.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:20.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:20.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:20.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:20.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:20.163+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:20.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:20.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:20.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:20.908+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:21.015+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:21.015+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:21.015+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:21.016+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:21.115+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:21.115+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:21.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:21.115+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:21.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:21.116+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:22.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:22.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:22.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:22.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:22.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:22.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:22.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:22.011+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:22.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:21.865+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:22.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:21.865+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:22.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:21.865+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:22.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:21.865+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:22.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:22.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:22.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:22.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:22.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:22.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:22.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:22.091+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:23.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:57:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:57:22.980+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46911 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:57:23.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:22.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:23.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:22.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:23.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:22.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:23.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:22.900+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:23.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:22.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:23.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:22.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:23.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:22.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:23.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:22.977+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:23.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:23.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:23.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:23.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:23.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:23.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:23.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:23.134+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:24.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:24.004+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:24.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:24.004+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:24.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:24.004+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:24.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:24.005+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:24.128 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:23.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:24.128 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:23.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:24.128 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:23.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:24.128 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:23.868+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:24.129 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:24.125+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:24.129 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:24.125+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:24.129 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:24.125+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:24.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:24.126+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:25.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:24.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:25.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:24.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:25.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:24.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:25.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:24.959+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:25.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:24.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:25.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:24.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:25.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:24.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:25.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:24.867+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:25.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:25.145+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:25.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:25.145+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:25.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:25.145+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:25.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:25.146+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:26.185 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:25.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:26.185 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:25.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:26.185 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:25.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:26.185 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:25.923+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:26.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:25.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:26.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:25.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:26.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:25.868+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:26.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:25.868+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:26.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:26.183+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:26.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:26.183+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:26.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:26.183+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:26.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:26.184+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:27.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:26.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:27.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:26.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:27.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:26.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:27.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:26.932+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:27.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:26.864+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:27.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:26.864+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:27.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:26.864+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:27.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:26.864+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:27.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:27.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:27.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:27.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:27.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:27.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:27.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:27.162+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:28.106 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:57:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:57:27.986+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 46956 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:57:28.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:27.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:28.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:27.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:28.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:27.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:28.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:27.982+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:28.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:27.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:28.108 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:27.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:28.108 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:27.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:28.108 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:27.839+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:28.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:28.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:28.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:28.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:28.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:28.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:28.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:28.165+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:29.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:29.014+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:29.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:29.014+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:29.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:29.014+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:29.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:29.015+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:29.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:28.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:29.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:28.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:29.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:28.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:29.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:28.889+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:29.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:29.180+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:29.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:29.180+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:29.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:29.180+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:29.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:29.181+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:30.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:29.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:30.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:29.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:30.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:29.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:30.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:29.981+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:30.194 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:29.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:30.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:29.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:30.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:29.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:30.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:29.867+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:30.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:30.192+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:30.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:30.192+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:30.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:30.192+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:30.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:30.193+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:31.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:31.013+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:31.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:31.013+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:31.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:31.013+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:31.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:31.015+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:31.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:30.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:31.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:30.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:31.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:30.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:31.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:30.885+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:31.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:31.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:31.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:31.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:31.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:31.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:31.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:31.179+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:32.188 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:32.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:32.189 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:32.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:32.189 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:32.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:32.189 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:32.002+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:32.189 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:31.887+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:32.189 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:31.887+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:32.189 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:31.887+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:32.190 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:31.888+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:32.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:32.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:32.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:32.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:32.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:32.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:32.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:32.188+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:33.171 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:57:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:57:32.992+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47003 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:57:33.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:32.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:33.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:32.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:33.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:32.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:33.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:32.972+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:33.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:32.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:33.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:32.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:33.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:32.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:33.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:32.899+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:33.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:33.212+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:33.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:33.212+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:33.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:33.212+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:33.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:33.213+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:34.222 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:33.953+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:34.222 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:33.953+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:34.222 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:33.953+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:34.222 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:33.954+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:34.223 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:33.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:34.223 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:33.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:34.223 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:33.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:34.223 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:33.851+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:34.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:34.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:34.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:34.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:34.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:34.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:34.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:34.221+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:35.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:34.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:35.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:34.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:35.195 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:34.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:35.195 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:34.907+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:35.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:34.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:35.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:34.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:35.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:34.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:35.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:34.808+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:35.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:35.192+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:35.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:35.192+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:35.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:35.193+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:35.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:35.193+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:36.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:35.877+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:36.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:35.877+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:36.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:35.877+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:36.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:35.877+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:36.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:35.774+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:36.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:35.774+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:36.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:35.774+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:36.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:35.774+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:36.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:36.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:36.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:36.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:36.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:36.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:36.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:36.234+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:37.214 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:36.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:37.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:36.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:37.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:36.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:37.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:36.842+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:37.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:36.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:37.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:36.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:37.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:36.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:37.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:36.788+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:37.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:37.212+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:37.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:37.213+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:37.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:37.213+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:37.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:37.213+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:38.000 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:37.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:38.000 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:37.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:38.001 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:37.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:38.001 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:37.836+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:38.001 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:37.746+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:38.001 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:37.746+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:38.001 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:37.746+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:38.001 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:37.747+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:38.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:57:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:57:37.998+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47049 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:57:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:38.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:38.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:38.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:38.171+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:38.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:38.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:38.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:38.857+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:39.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:38.712+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:39.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:38.712+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:39.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:38.712+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:39.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:38.713+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:39.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:39.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:39.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:39.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:39.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:39.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:39.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:39.159+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:40.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:39.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:40.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:39.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:40.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:39.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:40.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:39.695+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:40.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:39.823+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:40.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:39.823+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:40.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:39.823+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:40.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:39.824+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:40.183+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:40.183+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:40.183+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:40.184+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:41.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:40.671+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:41.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:40.671+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:41.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:40.671+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:40.672+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:41.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:40.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:41.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:40.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:41.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:40.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:41.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:40.836+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:41.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:41.218+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:41.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:41.218+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:41.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:41.218+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:41.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:41.219+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:41.976 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:41.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:41.976 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:41.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:41.976 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:41.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:41.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:41.832+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:41.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:41.640+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:41.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:41.640+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:41.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:41.640+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:41.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:41.641+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:42.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:42.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:42.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:42.217+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:43.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:42.880+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:43.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:42.880+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:43.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:42.880+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:43.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:42.881+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:43.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:42.654+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:43.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:42.654+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:43.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:42.654+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:43.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:42.655+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:43.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:57:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:57:43.004+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47096 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:57:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:43.198+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:43.198+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:43.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:43.198+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:43.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:43.199+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:43.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:43.689+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:43.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:43.689+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:43.887 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:43.689+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:43.887 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:43.690+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:44.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:43.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:44.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:43.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:44.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:43.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:44.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:43.885+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:44.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:44.210+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:44.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:44.210+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:44.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:44.210+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:44.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:44.211+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:44.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:44.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:44.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:44.870+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:45.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:44.667+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:45.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:44.667+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:45.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:44.667+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:45.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:44.668+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:45.460 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:45.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:45.461 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:45.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:45.461 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:45.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:45.461 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:45.170+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:45.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:45.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:45.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:46.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:45.718+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:45.863+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:45.863+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:45.863+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:45.863+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:46.131+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:46.131+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:46.131+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:46.132+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:46.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:46.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:46.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:46.910+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:47.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:46.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:47.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:46.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:47.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:46.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:47.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:46.727+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:47.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:47.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:47.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:47.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:47.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:47.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:47.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:47.157+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:48.028 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:57:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:57:48.010+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47141 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:57:48.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:47.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:48.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:47.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:48.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:47.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:48.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:47.890+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:48.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:47.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:48.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:47.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:48.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:47.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:48.030 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:47.772+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:48.124+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:48.124+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:48.124+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:48.125+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:49.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:48.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:49.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:48.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:49.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:48.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:49.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:48.864+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:49.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:48.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:49.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:48.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:49.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:48.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:49.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:48.796+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:49.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:49.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:49.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:49.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:49.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:49.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:49.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:49.139+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:49.866+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:49.866+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:49.866+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:50.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:49.867+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:50.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:49.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:50.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:49.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:50.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:49.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:50.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:49.764+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:50.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:50.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:50.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:50.188+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:51.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:50.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:51.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:50.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:51.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:50.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:51.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:50.855+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:51.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:50.792+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:51.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:50.792+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:51.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:50.792+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:51.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:50.793+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:51.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:51.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:51.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:51.164+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:51.863 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:51.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:51.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:51.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:51.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:51.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:51.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:51.782+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:52.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:51.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:52.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:51.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:52.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:51.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:52.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:51.863+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:52.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:52.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:52.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:52.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:52.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:52.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:52.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:52.165+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:53.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:52.890+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:53.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:52.890+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:53.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:52.890+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:53.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:52.891+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:53.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:52.739+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:53.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:52.739+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:53.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:52.739+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:53.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:52.739+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:53.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:57:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:57:53.016+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47188 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:57:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:53.212+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:53.212+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:53.212+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:53.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:53.213+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:53.931 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:53.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:53.931 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:53.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:53.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:53.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:53.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:53.722+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:54.228 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:53.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:54.229 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:53.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:54.229 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:53.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:54.229 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:53.930+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:54.227+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:54.227+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:54.227+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:54.227+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:54.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:54.764+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:54.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:54.764+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:54.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:54.764+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:54.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:54.765+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:55.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:54.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:55.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:54.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:55.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:54.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:55.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:54.963+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:55.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:55.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:55.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:55.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:55.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:55.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:55.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:55.257+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:55.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:55.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:55.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:55.971+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:55.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:55.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:55.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:55.813+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:56.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:56.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:56.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:56.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:56.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:56.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:56.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:56.297+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:57.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:56.985+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:57.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:56.985+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:57.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:56.985+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:57.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:56.987+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:57.131 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:56.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:57.131 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:56.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:57.131 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:56.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:57.131 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:56.796+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:57.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:57.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:57.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:57.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:57.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:57.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:57.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:57.343+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:58.041 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:57:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:57:58.022+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47234 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:57:58.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:57.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:58.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:57.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:58.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:57.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:58.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:57.946+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:58.042 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:57.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:58.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:57.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:58.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:57.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:58.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:57.788+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:58.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:58.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:58.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:58.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:58.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:58.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:58.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:58.321+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:57:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:58.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:58.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:58.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:57:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:58.940+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:59.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:58.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:59.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:58.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:59.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:58.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:57:59.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:58.757+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:57:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:59.281+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:59.281+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:59.281+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:57:59.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:57:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:57:59.281+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:00.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:59.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:00.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:59.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:00.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:59.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:00.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:57:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:57:59.985+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:00.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:59.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:00.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:59.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:00.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:59.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:00.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:57:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:57:59.776+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:00.242+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:00.242+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:00.242+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:00.243+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:01.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:00.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:01.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:00.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:01.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:00.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:01.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:00.983+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:01.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:00.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:01.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:00.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:01.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:00.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:01.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:00.735+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:01.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:01.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:01.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:01.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:01.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:01.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:01.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:01.200+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:01.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:01.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:01.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:01.983+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:02.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:01.742+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:02.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:01.742+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:02.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:01.742+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:02.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:01.742+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:02.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:02.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:02.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:02.170+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:03.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:02.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:03.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:02.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:03.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:02.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:03.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:02.757+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:02.979+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:02.979+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:02.979+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:02.980+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:03.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:58:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:58:03.028+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47281 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:58:03.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:03.191+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:03.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:03.191+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:03.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:03.191+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:03.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:03.192+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:03.893 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:03.748+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:03.893 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:03.748+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:03.893 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:03.748+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:03.893 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:03.749+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:04.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:03.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:04.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:03.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:04.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:03.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:04.225 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:03.976+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:04.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:04.222+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:04.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:04.222+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:04.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:04.222+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:04.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:04.223+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:05.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:05.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:05.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:05.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:05.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:05.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:05.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:05.011+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:05.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:04.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:05.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:04.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:05.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:04.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:05.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:04.778+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:05.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:05.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:05.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:05.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:05.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:05.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:05.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:05.174+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:06.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:05.729+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:06.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:05.729+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:06.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:05.729+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:06.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:05.729+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:06.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:06.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:06.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:06.050+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:06.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:06.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:06.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:06.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:06.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:06.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:06.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:06.157+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:07.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:06.749+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:07.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:06.749+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:07.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:06.749+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:07.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:06.750+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:07.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:07.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:07.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:07.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:07.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:07.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:07.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:07.034+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:07.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:07.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:07.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:07.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:07.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:07.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:07.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:07.159+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:08.053 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:58:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:58:08.034+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47322 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:58:08.054 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:07.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:08.054 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:07.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:08.054 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:07.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:08.054 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:07.800+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:08.054 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:08.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:08.055 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:08.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:08.055 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:08.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:08.055 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:08.023+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:08.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:08.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:08.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:08.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:08.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:08.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:08.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:08.178+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:09.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:08.777+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:09.036 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:08.777+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:09.037 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:08.777+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:09.037 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:08.778+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:09.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:09.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:09.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:09.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:09.035+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:09.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:09.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:09.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:09.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:09.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:09.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:09.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:09.152+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:10.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:10.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:10.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:10.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:10.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:10.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:10.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:10.023+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:10.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:09.815+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:10.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:09.815+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:10.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:09.815+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:10.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:09.816+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:10.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:10.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:10.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:10.173+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:11.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:11.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:11.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:11.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:11.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:11.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:11.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:11.017+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:11.128 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:10.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:11.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:10.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:11.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:10.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:11.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:10.800+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:11.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:11.126+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:11.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:11.126+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:11.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:11.126+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:11.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:11.127+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:12.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:12.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:12.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:12.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:12.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:12.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:12.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:12.049+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:12.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:11.842+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:12.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:11.842+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:12.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:11.842+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:12.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:11.843+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:12.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:12.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:12.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:12.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:12.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:12.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:12.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:12.164+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:13.145 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:58:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:58:13.040+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47369 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:58:13.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:13.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:13.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:13.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:13.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:13.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:13.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:13.019+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:13.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:12.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:13.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:12.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:13.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:12.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:13.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:12.886+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:13.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:13.144+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:13.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:13.144+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:13.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:13.144+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:13.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:13.144+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:14.055+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:14.055+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:14.055+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:14.056+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:14.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:14.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:14.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:14.103+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:13.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:14.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:13.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:14.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:13.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:14.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:13.897+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:15.076+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:15.076+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:15.076+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:15.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:15.077+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:15.125+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:15.125+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:15.125+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:15.126+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:15.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:14.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:15.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:14.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:15.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:14.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:15.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:14.910+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:16.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:15.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:16.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:15.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:16.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:15.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:16.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:15.950+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:16.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:16.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:16.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:16.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:16.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:16.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:16.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:16.073+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:16.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:16.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:16.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:16.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:16.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:16.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:16.098+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:17.115+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:17.115+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:17.115+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:17.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:17.116+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:16.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:16.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:16.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:16.993+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:17.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:17.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:17.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:17.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:17.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:17.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:17.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:17.082+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:18.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:58:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:58:18.045+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47416 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:58:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:18.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:18.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:18.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:18.092+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:18.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:18.083+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:18.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:18.083+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:18.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:18.083+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:18.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:18.084+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:18.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:17.971+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:18.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:17.971+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:18.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:17.971+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:18.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:17.971+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:18.953+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:18.953+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:18.953+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:18.953+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:19.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:19.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:19.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:19.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:19.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:19.112+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:19.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:19.113+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:19.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:19.126+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:19.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:19.126+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:19.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:19.126+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:19.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:19.127+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:20.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:20.084+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:20.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:20.084+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:20.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:20.084+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:20.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:20.085+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:20.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:20.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:20.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:20.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:20.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:20.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:20.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:20.163+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:20.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:19.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:20.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:19.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:20.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:19.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:20.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:19.965+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:21.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:21.037+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:21.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:21.037+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:21.037+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:21.038+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:21.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:21.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:21.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:21.157+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:21.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:20.978+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:21.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:20.978+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:21.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:20.978+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:21.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:20.979+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:22.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:21.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:22.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:21.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:22.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:21.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:22.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:21.987+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:22.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:22.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:22.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:22.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:22.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:22.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:22.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:22.080+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:22.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:22.125+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:22.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:22.125+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:22.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:22.125+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:22.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:22.126+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:23.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:58:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:58:23.052+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47463 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:58:23.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:22.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:23.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:22.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:23.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:22.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:23.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:22.984+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:23.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:23.092+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:23.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:23.092+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:23.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:23.092+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:23.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:23.093+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:23.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:23.087+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:23.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:23.087+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:23.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:23.087+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:23.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:23.088+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:24.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:24.104+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:24.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:24.104+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:24.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:24.104+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:24.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:24.105+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:24.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:24.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:24.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:24.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:24.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:24.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:24.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:24.046+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:24.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:24.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:24.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:24.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:24.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:24.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:24.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:24.025+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:25.395 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:25.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:25.396 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:25.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:25.396 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:25.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:25.396 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:25.038+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:25.396 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:25.067+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:25.397 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:25.067+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:25.397 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:25.067+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:25.397 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:25.068+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:25.397 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:25.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:25.397 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:25.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:25.397 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:25.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:25.397 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:25.022+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:26.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:26.061+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:26.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:26.061+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:26.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:26.061+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:26.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:26.061+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:26.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:26.052+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:26.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:26.052+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:26.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:26.052+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:26.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:26.053+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:26.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:26.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:26.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:26.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:26.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:26.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:26.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:26.037+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:27.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:27.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:27.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:27.050+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:27.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:27.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:27.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:27.075+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:26.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:27.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:26.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:27.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:26.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:27.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:26.994+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:28.521 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:58:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:58:28.059+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47508 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:58:28.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:28.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:28.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:28.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:28.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:28.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:28.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:28.036+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:28.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:28.055+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:28.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:28.055+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:28.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:28.055+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:28.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:28.057+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:28.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:28.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:28.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:28.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:28.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:28.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:28.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:28.031+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:29.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:29.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:29.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:29.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:29.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:29.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:29.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:29.029+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:29.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:29.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:29.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:29.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:29.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:29.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:29.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:29.104+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:29.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:29.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:29.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:29.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:29.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:29.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:29.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:29.025+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:30.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:30.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:30.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:30.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:30.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:30.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:30.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:30.030+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:30.301 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:30.087+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:30.302 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:30.087+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:30.302 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:30.087+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:30.302 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:30.087+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:30.302 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:30.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:30.302 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:30.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:30.302 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:30.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:30.302 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:30.060+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:31.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:31.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:31.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:31.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:31.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:31.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:31.108+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:31.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:31.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:31.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:31.017+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:31.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:31.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:31.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:31.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:31.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:31.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:31.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:31.077+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:32.488 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:32.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:32.489 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:32.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:32.489 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:32.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:32.489 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:32.057+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:32.490 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:32.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:32.490 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:32.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:32.490 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:32.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:32.490 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:32.062+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:32.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:32.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:32.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:32.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:32.491 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:32.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:32.491 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:32.105+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:33.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:58:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:58:33.065+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47555 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:58:33.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:33.027+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:33.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:33.027+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:33.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:33.027+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:33.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:33.028+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:33.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:33.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:33.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:33.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:33.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:33.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:33.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:33.037+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:33.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:33.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:33.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:33.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:33.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:33.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:33.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:33.097+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:34.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:34.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:34.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:34.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:34.008+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:34.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:34.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:34.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:34.009+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:34.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:34.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:34.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:34.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:34.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:34.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:34.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:34.076+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:35.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:35.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:35.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:35.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:35.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:35.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:35.017+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:35.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:35.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:35.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:35.007+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:35.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:35.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:35.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:35.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:35.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:35.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:35.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:35.080+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:36.012+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:36.012+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:36.012+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:36.013+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:36.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:36.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:36.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:36.052+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:36.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:36.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:36.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:36.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:36.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:36.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:36.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:36.083+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:36.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:36.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:36.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:37.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:36.984+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:37.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:37.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:37.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:37.014+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:37.117+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:37.117+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:37.117+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:37.117+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:38.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:58:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:58:38.071+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47601 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:58:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:37.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:37.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:37.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:37.964+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:37.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:37.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:38.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:37.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:38.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:37.992+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:38.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:38.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:38.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:38.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:38.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:38.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:38.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:38.151+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:38.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:38.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:38.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:39.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:38.944+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:39.025+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:39.025+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:39.025+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:39.026+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:39.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:39.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:39.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:39.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:39.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:39.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:39.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:39.110+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:40.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:39.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:40.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:39.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:40.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:39.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:40.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:39.926+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:40.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:40.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:40.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:40.002+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:40.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:40.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:40.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:40.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:40.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:40.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:40.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:40.087+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:40.966+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:40.966+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:41.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:40.966+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:41.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:40.967+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:40.974+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:40.974+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:40.974+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:40.974+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:41.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:41.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:41.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:41.096+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:42.023+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:42.023+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:42.023+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:42.024+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:42.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:42.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:42.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:42.142+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:42.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:41.993+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:42.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:41.993+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:42.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:41.993+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:42.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:41.994+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:43.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:58:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:58:43.077+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47648 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:58:43.317 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:43.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:43.317 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:43.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:43.317 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:43.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:43.317 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:43.192+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:43.318 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:43.013+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:43.318 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:43.013+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:43.318 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:43.013+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:43.318 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:43.014+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:43.318 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:43.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:43.318 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:43.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:43.318 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:43.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:43.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:43.052+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:44.004+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:44.004+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:44.004+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:44.005+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:44.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:44.050+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:44.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:44.050+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:44.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:44.050+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:44.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:44.051+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:44.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:44.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:44.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:44.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:44.172+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:44.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:44.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:44.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:45.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:44.990+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:45.063+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:45.063+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:45.063+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:45.064+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:45.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:45.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:45.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:45.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:45.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:45.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:45.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:45.125+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:46.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:46.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:46.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:46.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:46.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:46.121+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:46.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:46.121+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:45.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:45.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:45.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:45.986+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:46.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:46.021+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:46.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:46.021+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:46.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:46.021+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:46.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:46.022+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:47.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:47.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:47.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:47.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:47.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:47.072+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:47.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:47.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:47.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:47.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:47.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:47.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:47.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:47.030+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:47.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:46.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:47.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:46.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:47.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:46.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:47.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:46.959+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:48.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:58:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:58:48.083+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47693 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:58:48.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:48.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:48.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:48.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:48.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:48.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:48.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:48.092+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:48.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:48.060+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:48.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:48.060+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:48.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:48.060+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:48.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:48.061+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:48.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:48.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:48.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:48.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:48.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:48.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:48.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:48.002+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:49.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:49.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:49.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:49.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:49.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:49.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:49.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:49.113+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:49.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:49.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:49.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:49.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:49.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:49.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:49.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:49.089+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:49.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:49.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:49.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:49.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:49.007+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:49.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:49.008+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:50.069+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:50.069+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:50.069+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:50.070+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:50.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:50.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:50.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:50.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:50.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:50.144+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:50.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:50.144+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:50.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:49.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:50.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:49.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:50.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:49.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:50.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:49.979+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:51.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:51.021+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:51.021+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:51.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:51.021+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:51.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:51.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:51.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:51.004+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:51.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:51.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:51.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:51.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:51.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:51.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:51.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:51.126+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:52.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:52.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:52.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:52.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:52.057+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:51.987+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:51.987+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:51.987+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:51.988+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:52.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:52.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:52.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:52.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:52.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:52.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:52.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:52.105+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:53.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:58:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:58:53.089+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47739 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:58:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:52.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:52.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:53.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:52.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:53.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:52.965+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:53.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:53.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:53.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:53.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:53.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:53.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:53.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:53.029+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:53.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:53.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:53.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:53.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:53.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:53.146+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:53.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:53.147+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:54.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:54.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:54.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:54.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:54.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:54.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:54.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:54.193+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:54.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:54.014+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:54.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:54.014+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:54.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:54.014+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:54.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:54.015+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:54.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:53.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:54.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:53.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:54.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:53.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:54.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:53.999+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:55.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:54.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:55.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:54.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:55.211 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:54.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:55.211 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:54.957+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:55.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:54.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:55.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:54.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:55.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:54.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:55.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:54.990+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:55.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:55.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:55.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:55.209+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:55.940+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:55.940+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:55.940+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:55.941+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:55.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:55.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:55.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:55.962+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:56.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:56.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:56.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:56.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:56.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:56.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:56.183+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:56.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:56.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:56.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:57.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:56.939+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:57.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:57.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:57.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:57.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:57.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:57.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:57.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:57.156+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:57.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:56.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:57.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:56.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:57.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:56.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:57.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:56.967+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:57.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:57.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:58.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:57.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:58.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:57.986+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:57.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:57.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:57.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:57.933+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:58.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:58:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:58:58.095+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47778 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:58:58.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:58.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:58.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:58.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:58.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:58.202+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:58.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:58.203+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:59.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:58.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:59.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:58.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:59.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:58.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:58:59.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:58.957+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:58:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:59.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:59.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:59.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:58:59.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:58:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:58:59.036+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:58:59.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:59.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:59.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:59.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:59.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:59.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:58:59.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:58:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:58:59.214+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:00.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:00.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:00.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:00.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:00.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:00.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:00.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:00.005+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:00.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:59.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:00.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:59.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:00.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:59.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:00.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:58:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:58:59.923+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:00.207+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:00.207+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:00.207+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:00.208+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:00.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:00.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:00.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:00.927+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:01.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:01.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:01.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:01.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:01.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:01.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:01.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:01.163+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:01.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:01.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:01.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:01.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:01.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:01.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:01.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:01.047+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:02.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:01.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:02.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:01.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:02.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:01.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:02.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:01.937+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:02.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:01.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:02.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:01.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:02.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:01.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:02.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:02.000+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:02.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:02.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:02.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:02.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:02.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:02.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:02.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:02.144+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:03.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:02.961+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:03.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:02.961+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:03.104 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:02.961+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:03.104 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:02.962+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:03.104 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:02.996+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:03.104 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:02.996+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:03.104 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:02.996+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:03.104 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:02.997+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:03.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:59:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:59:03.101+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47820 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:59:03.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:03.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:03.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:03.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:03.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:03.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:03.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:03.142+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:04.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:03.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:04.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:03.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:04.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:03.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:04.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:03.960+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:04.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:04.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:04.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:04.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:04.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:04.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:04.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:04.005+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:04.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:04.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:04.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:04.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:04.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:04.166+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:04.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:04.167+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:04.996+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:05.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:04.996+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:05.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:04.996+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:05.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:04.997+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:05.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:04.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:05.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:04.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:05.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:04.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:05.273 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:04.979+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:05.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:05.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:05.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:05.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:05.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:05.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:05.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:05.216+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:06.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:06.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:06.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:06.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:06.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:06.026+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:06.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:06.040+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:06.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:06.040+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:06.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:06.040+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:06.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:06.041+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:06.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:06.173+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:06.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:06.173+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:06.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:06.173+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:06.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:06.174+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:06.992+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:06.992+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:06.992+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:06.993+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:07.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:07.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:07.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:07.006+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:07.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:07.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:07.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:07.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:07.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:07.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:07.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:07.126+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:08.128 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:59:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:59:08.107+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47859 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:59:08.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:08.009+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:08.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:08.009+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:08.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:08.009+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:08.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:08.010+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:08.130 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:07.946+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:08.130 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:07.946+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:08.130 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:07.946+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:08.130 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:07.947+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:08.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:08.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:08.132+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:08.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:08.132+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:09.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:09.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:09.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:09.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:09.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:09.123+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:08.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:08.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:08.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:08.971+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:09.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:08.932+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:09.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:08.932+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:09.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:08.932+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:09.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:08.933+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:10.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:10.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:10.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:10.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:10.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:10.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:10.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:10.117+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:10.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:10.013+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:10.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:10.013+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:10.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:10.013+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:10.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:10.014+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:10.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:09.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:10.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:09.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:10.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:09.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:10.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:09.981+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:11.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:11.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:11.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:11.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:11.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:11.112+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:11.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:10.993+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:11.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:10.993+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:11.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:10.993+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:11.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:10.994+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:11.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:10.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:11.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:10.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:11.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:10.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:11.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:10.933+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:12.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:12.092+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:12.092+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:12.092+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:12.092+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:12.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:12.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:12.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:12.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:12.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:12.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:12.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:12.032+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:12.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:11.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:12.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:11.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:12.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:11.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:12.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:11.968+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:13.234 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:59:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:59:13.113+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47900 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:59:13.235 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:13.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:13.235 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:13.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:13.235 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:13.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:13.235 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:13.011+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:13.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:13.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:13.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:13.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:13.236 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:13.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:13.236 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:13.003+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:13.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:13.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:13.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:13.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:13.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:13.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:13.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:13.111+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:14.008+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:14.008+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:14.008+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:14.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:14.010+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:13.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:13.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:13.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:14.000+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:14.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:14.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:14.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:14.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:14.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:14.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:14.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:14.135+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:15.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:15.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:15.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:15.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:15.049+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:15.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:15.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:15.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:15.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:15.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:15.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:15.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:15.030+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:15.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:15.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:15.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:15.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:15.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:15.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:15.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:15.105+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:16.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:16.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:16.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:16.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:16.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:16.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:16.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:16.078+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:16.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:16.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:16.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:16.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:16.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:16.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:16.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:16.021+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:16.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:16.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:16.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:16.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:16.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:16.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:16.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:16.152+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:17.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:17.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:17.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:17.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:17.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:17.006+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:17.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:17.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:17.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:17.114+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:17.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:17.111+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:17.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:17.111+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:17.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:17.111+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:17.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:17.112+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:18.238 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:59:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:59:18.119+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47939 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:59:18.239 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:17.974+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:18.239 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:17.974+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:18.239 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:17.974+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:18.239 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:17.975+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:18.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:18.131+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:18.240 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:18.132+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:18.240 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:18.132+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:18.240 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:18.132+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:18.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:18.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:18.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:18.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:18.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:18.110+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:18.241 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:18.110+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:19.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:19.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:19.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:19.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:19.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:19.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:19.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:19.159+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:19.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:19.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:19.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:19.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:19.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:19.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:19.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:19.151+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:19.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:19.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:19.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:19.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:19.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:19.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:19.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:19.011+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:20.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:20.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:20.305 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:20.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:20.305 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:20.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:20.305 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:20.192+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:20.305 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:20.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:20.305 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:20.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:20.305 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:20.030+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:20.305 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:20.031+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:20.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:20.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:20.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:20.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:20.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:20.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:20.306 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:20.116+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:21.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:21.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:21.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:21.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:21.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:21.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:21.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:21.064+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:21.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:21.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:21.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:21.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:21.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:21.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:21.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:21.097+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:21.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:21.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:21.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:21.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:21.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:21.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:21.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:21.161+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:22.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:22.115+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:22.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:22.115+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:22.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:22.115+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:22.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:22.116+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:22.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:22.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:22.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:22.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:22.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:22.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:22.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:22.047+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:22.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:22.093+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:22.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:22.094+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:22.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:22.094+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:22.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:22.094+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:23.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:59:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:59:23.125+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 47980 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:59:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:23.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:23.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:23.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:23.065+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:23.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:23.134+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:23.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:23.134+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:23.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:23.134+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:23.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:23.135+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:23.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:23.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:23.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:23.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:23.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:23.056+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:23.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:23.056+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:24.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:24.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:24.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:24.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:24.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:24.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:24.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:24.044+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:24.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:24.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:24.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:24.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:24.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:24.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:24.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:24.044+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:24.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:24.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:24.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:24.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:24.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:24.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:24.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:24.179+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:25.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:25.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:25.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:25.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:25.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:25.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:25.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:25.029+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:25.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:25.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:25.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:25.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:25.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:25.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:25.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:25.074+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:25.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:25.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:25.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:25.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:25.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:25.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:25.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:25.210+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:26.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:26.187+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:26.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:26.187+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:26.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:26.187+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:26.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:26.187+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:26.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:26.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:26.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:26.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:26.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:26.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:26.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:26.044+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:26.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:26.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:26.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:26.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:26.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:26.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:26.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:26.014+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:27.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:27.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:27.159+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:27.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:27.160+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:27.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:27.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:27.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:27.081+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:27.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:26.984+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:27.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:26.984+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:27.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:26.984+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:27.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:26.985+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:28.254 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:59:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:59:28.132+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48019 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:59:28.254 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:28.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:28.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:28.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:28.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:28.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:28.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:28.101+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:28.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:27.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:28.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:27.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:28.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:27.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:28.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:27.961+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:28.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:28.164+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:28.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:28.164+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:28.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:28.164+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:28.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:28.164+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:28.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:28.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:28.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:28.999+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:29.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:29.126+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:29.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:29.126+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:29.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:29.126+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:29.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:29.127+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:29.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:29.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:29.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:29.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:29.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:29.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:29.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:29.060+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:29.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:29.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:29.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:29.979+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:30.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:30.095+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:30.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:30.095+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:30.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:30.095+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:30.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:30.096+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:30.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:30.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:30.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:30.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:30.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:30.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:30.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:30.076+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:30.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:30.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:30.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:31.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:30.978+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:31.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:31.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:31.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:31.062+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:31.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:31.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:31.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:31.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:31.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:31.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:31.063+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:32.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:32.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:32.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:32.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:32.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:32.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:32.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:32.093+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:32.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:32.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:32.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:32.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:32.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:32.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:32.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:32.022+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:32.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:32.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:32.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:32.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:32.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:32.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:32.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:32.023+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:33.261 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:59:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:59:33.138+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48060 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:59:33.262 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:33.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:33.262 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:33.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:33.262 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:33.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:33.262 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:33.048+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:33.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:33.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:33.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:33.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:33.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:33.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:33.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:33.057+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:33.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:32.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:33.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:32.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:33.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:32.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:33.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:32.994+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:34.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:34.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:34.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:34.027+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:34.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:34.027+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:34.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:34.027+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:34.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:34.089+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:34.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:34.089+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:34.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:34.089+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:34.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:34.090+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:34.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:34.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:34.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:34.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:34.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:34.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:34.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:34.020+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:35.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:35.027+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:35.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:35.027+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:35.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:35.027+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:35.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:35.028+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:35.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:35.099+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:35.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:35.099+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:35.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:35.099+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:35.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:35.100+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:35.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:35.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:35.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:35.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:35.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:35.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:35.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:35.030+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:36.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:36.130+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:36.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:36.130+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:36.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:36.130+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:36.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:36.131+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:36.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:36.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:36.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:36.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:36.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:36.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:36.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:36.045+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:36.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:36.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:36.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:36.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:36.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:36.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:36.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:36.028+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:37.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:37.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:37.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:37.065+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:37.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:37.108+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:37.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:37.108+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:37.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:37.108+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:37.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:37.108+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:36.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:36.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:36.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:36.980+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:38.265 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:59:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:59:38.144+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48099 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:59:38.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:38.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:38.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:38.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:38.266 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:38.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:38.266 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:38.121+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:38.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:38.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:38.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:38.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:38.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:38.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:38.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:38.107+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:38.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:37.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:38.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:37.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:38.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:37.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:38.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:37.942+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:39.085+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:39.085+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:39.085+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:39.085+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:39.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:39.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:39.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:39.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:39.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:39.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:39.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:39.158+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:38.932+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:39.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:38.932+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:39.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:38.932+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:39.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:38.933+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:40.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:40.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:40.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:40.124+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:40.138+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:40.138+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:40.138+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:40.140+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:40.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:39.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:40.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:39.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:40.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:39.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:40.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:39.944+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:41.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:41.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:41.170+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:41.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:41.171+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:41.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:41.139+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:41.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:41.139+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:41.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:41.139+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:41.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:41.140+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:41.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:40.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:41.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:40.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:41.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:40.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:41.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:40.972+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:42.146+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:42.146+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:42.146+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:42.147+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:42.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:42.105+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:42.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:42.105+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:42.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:42.105+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:42.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:42.106+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:42.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:42.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:42.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:42.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:42.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:42.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:42.013+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:43.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:59:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:59:43.150+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48141 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:59:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:43.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:43.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:43.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:43.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:43.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:43.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:43.049+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:43.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:43.058+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:43.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:43.058+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:43.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:43.058+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:43.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:43.059+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:43.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:43.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:43.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:43.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:43.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:43.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:43.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:43.118+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:44.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:44.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:44.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:44.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:44.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:44.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:44.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:44.074+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:44.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:44.100+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:44.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:44.100+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:44.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:44.100+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:44.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:44.101+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:44.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:44.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:44.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:44.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:44.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:44.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:44.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:44.044+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:45.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:45.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:45.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:45.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:45.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:45.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:45.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:45.094+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:45.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:45.086+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:45.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:45.086+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:45.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:45.086+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:45.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:45.086+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:45.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:45.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:45.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:45.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:45.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:45.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:45.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:45.090+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:46.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:46.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:46.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:46.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:46.091+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:46.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:46.129+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:46.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:46.129+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:46.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:46.129+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:46.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:46.130+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:46.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:46.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:46.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:46.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:46.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:46.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:46.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:46.097+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:47.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:47.116+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:47.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:47.116+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:47.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:47.116+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:47.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:47.117+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:47.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:47.120+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:47.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:47.120+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:47.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:47.120+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:47.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:47.121+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:47.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:47.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:47.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:47.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:47.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:47.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:47.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:47.098+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:48.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:59:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:59:48.156+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48180 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:59:48.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:48.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:48.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:48.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:48.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:48.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:48.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:48.085+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:48.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:48.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:48.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:48.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:48.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:48.134+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:48.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:48.134+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:48.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:48.069+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:48.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:48.069+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:48.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:48.069+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:48.523 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:48.070+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:49.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:49.083+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:49.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:49.083+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:49.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:49.083+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:49.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:49.084+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:49.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:49.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:49.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:49.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:49.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:49.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:49.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:49.083+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:49.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:49.140+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:49.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:49.140+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:49.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:49.140+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:49.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:49.141+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:50.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:50.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:50.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:50.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:50.119+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:50.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:50.107+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:50.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:50.107+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:50.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:50.107+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:50.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:50.107+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:50.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:50.114+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:50.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:50.114+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:50.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:50.114+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:50.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:50.115+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:51.412 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:51.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:51.412 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:51.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:51.412 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:51.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:51.412 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:51.149+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:51.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:51.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:51.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:51.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:51.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:51.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:51.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:51.160+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:51.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:51.146+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:51.413 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:51.146+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:51.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:51.146+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:51.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:51.146+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:52.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:52.123+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:52.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:52.123+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:52.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:52.123+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:52.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:52.124+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:52.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:52.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:52.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:52.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:52.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:52.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:52.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:52.120+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:52.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:52.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:52.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:52.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:52.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:52.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:52.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:52.191+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:53.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:59:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:59:53.163+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48221 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:59:53.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:53.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:53.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:53.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:53.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:53.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:53.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:53.173+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:53.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:53.159+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:53.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:53.159+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:53.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:53.159+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:53.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:53.161+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:53.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:53.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:53.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:53.157+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:53.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:53.157+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:53.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:53.157+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:54.485 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:54.133+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:54.485 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:54.133+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:54.485 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:54.133+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:54.485 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:54.134+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:54.486 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:54.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:54.486 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:54.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:54.486 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:54.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:54.486 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:54.159+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:54.486 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:54.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:54.486 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:54.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:54.487 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:54.177+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:54.487 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:54.178+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:55.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:55.144+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:55.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:55.144+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:55.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:55.144+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:55.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:55.145+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:55.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:55.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:55.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:55.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:55.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:55.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:55.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:55.163+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:55.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:55.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:55.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:55.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:55.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:55.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:55.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:55.204+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:56.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:56.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:56.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:56.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:56.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:56.147+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:56.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:56.148+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:56.423 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:56.167+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:56.423 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:56.167+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:56.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:56.167+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:56.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:56.167+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:56.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:56.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:56.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:56.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:56.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:56.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:56.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:56.185+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:57.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:57.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:57.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:57.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:57.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:57.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:57.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:57.141+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:57.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:57.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:57.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:57.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:57.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:57.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:57.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:57.130+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:57.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:57.196+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:57.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:57.196+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:57.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:57.196+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:57.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:57.197+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:58.377 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 19:59:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T19:59:58.170+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48260 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T19:59:58.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:58.185+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:58.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:58.185+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:58.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:58.185+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:58.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:58.186+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:58.378 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:58.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:58.379 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:58.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:58.379 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:58.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:58.379 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:58.129+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:58.379 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:58.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:58.379 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:58.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:58.379 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:58.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:58.379 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:58.110+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:59.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:59.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:59.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:59.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:59.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:59.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T19:59:59.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 19:59:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T19:59:59.065+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T19:59:59.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:59.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:59.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:59.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:59.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:59.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T19:59:59.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 19:59:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T19:59:59.102+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T19:59:59.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:59.196+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:59.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:59.196+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:59.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:59.196+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T19:59:59.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 19:59:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T19:59:59.196+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:00.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:00.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:00.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:00.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:00.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:00.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:00.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:00.222+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:00.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:00.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:00.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:00.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:00.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:00.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:00.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:00.059+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:00.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:00.115+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:00.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:00.115+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:00.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:00.115+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:00.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:00.115+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:01.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:01.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:01.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:01.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:01.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:01.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:01.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:01.039+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:01.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:01.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:01.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:01.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:01.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:01.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:01.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:01.223+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:01.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:01.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:01.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:01.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:01.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:01.124+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:01.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:01.124+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:01.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:01.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:01.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:01.992+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:02.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:02.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:02.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:02.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:02.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:02.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:02.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:02.237+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:02.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:02.092+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:02.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:02.092+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:02.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:02.092+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:02.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:02.093+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:03.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:00:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:00:03.176+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48301 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:00:03.287 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:03.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:03.287 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:03.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:03.287 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:03.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:03.287 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:03.052+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:03.287 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:03.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:03.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:03.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:03.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:03.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:03.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:03.030+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:03.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:03.284+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:03.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:03.284+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:03.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:03.284+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:03.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:03.285+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:04.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:04.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:04.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:04.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:04.003+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:04.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:04.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:04.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:04.009+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:04.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:04.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:04.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:04.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:04.325+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:05.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:05.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:05.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:05.048+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:05.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:04.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:05.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:04.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:05.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:04.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:05.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:04.994+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:05.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:05.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:05.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:05.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:05.373+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:06.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:05.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:06.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:05.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:06.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:05.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:05.980+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:06.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:06.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:06.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:06.032+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:06.401+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:06.401+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:06.401+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:06.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:06.401+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:07.067 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:07.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:07.067 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:07.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:07.067 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:07.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:07.068 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:07.015+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:07.383 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:07.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:07.383 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:07.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:07.383 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:07.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:07.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:07.066+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:07.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:07.381+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:07.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:07.381+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:07.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:07.381+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:07.382+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:08.304 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:00:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:00:08.182+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48340 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:00:08.304 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:08.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:08.305 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:08.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:08.305 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:08.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:08.305 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:08.051+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:08.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:08.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:08.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:08.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:08.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:08.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:08.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:08.039+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:08.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:08.394+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:08.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:08.394+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:08.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:08.394+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:08.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:08.395+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:09.366 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:09.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:09.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:09.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:09.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:09.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:09.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:09.077+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:09.367 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:09.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:09.367 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:09.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:09.367 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:09.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:09.368 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:09.020+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:09.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:09.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:09.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:09.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:09.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:09.365+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:10.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:10.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:10.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:10.114+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:10.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:09.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:10.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:09.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:10.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:09.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:10.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:09.977+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:10.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:10.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:10.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:10.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:10.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:10.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:10.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:10.345+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:11.146+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:11.146+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:11.146+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:11.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:11.147+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:10.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:10.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:10.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:10.932+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:11.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:11.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:11.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:11.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:11.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:11.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:11.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:11.298+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:12.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:11.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:12.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:11.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:12.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:11.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:12.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:11.916+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:12.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:12.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:12.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:12.135+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:12.320+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:12.320+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:12.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:12.320+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:12.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:12.321+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:13.190 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:13.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:13.191 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:13.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:13.191 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:13.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:13.191 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:13.151+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:13.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:12.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:13.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:12.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:13.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:12.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:13.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:12.906+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:13.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:00:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:00:13.188+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48381 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:00:13.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:13.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:13.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:13.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:13.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:13.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:13.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:13.291+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:14.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:13.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:13.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:13.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:13.872+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:14.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:14.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:14.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:14.118+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:14.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:14.339+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:14.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:14.339+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:14.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:14.339+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:14.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:14.339+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:14.918+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:14.918+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:14.918+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:14.919+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:15.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:15.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:15.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:15.142+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:15.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:15.307+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:15.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:15.307+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:15.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:15.307+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:15.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:15.308+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:16.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:16.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:16.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:16.169+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:15.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:15.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:15.952+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:15.952+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:16.352+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:16.352+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:16.352+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:16.353+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:17.135+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:17.135+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:17.135+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:17.135+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:16.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:16.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:16.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:16.931+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:17.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:17.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:17.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:17.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:17.311+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:18.196 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:18.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:18.196 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:18.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:18.196 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:18.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:18.197 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:18.090+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:18.197 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:17.890+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:18.197 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:17.890+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:18.197 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:17.890+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:18.197 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:17.890+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:18.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:00:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:00:18.194+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48420 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:00:18.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:18.336+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:18.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:18.336+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:18.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:18.336+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:18.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:18.337+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:19.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:19.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:19.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:19.080+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:19.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:18.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:19.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:18.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:19.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:18.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:19.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:18.846+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:19.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:19.369+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:19.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:19.369+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:19.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:19.369+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:19.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:19.370+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:20.085+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:20.085+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:20.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:20.085+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:20.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:20.086+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:20.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:19.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:20.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:19.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:20.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:19.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:20.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:19.887+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:20.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:20.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:20.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:20.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:20.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:20.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:20.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:20.328+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:21.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:21.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:21.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:21.094+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:21.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:20.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:21.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:20.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:21.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:20.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:21.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:20.896+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:21.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:21.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:21.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:21.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:21.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:21.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:21.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:21.353+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:22.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:22.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:22.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:22.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:22.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:22.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:22.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:22.067+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:22.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:21.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:22.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:21.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:22.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:21.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:22.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:21.862+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:22.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:22.401+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:22.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:22.401+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:22.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:22.401+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:22.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:22.402+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:23.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:23.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:23.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:23.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:23.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:23.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:23.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:23.062+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:23.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:22.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:23.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:22.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:23.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:22.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:23.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:22.880+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:23.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:00:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:00:23.200+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48460 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:00:23.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:23.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:23.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:23.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:23.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:23.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:23.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:23.417+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:24.104+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:24.104+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:24.104+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:24.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:24.104+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:23.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:23.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:23.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:23.926+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:24.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:24.377+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:24.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:24.377+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:24.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:24.377+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:24.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:24.378+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:25.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:25.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:25.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:25.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:25.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:25.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:25.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:25.114+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:25.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:24.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:25.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:24.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:25.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:24.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:25.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:24.919+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:25.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:25.419+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:25.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:25.419+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:25.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:25.419+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:25.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:25.420+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:26.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:26.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:26.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:26.120+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:26.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:25.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:26.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:25.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:26.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:25.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:26.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:25.906+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:26.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:26.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:26.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:26.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:26.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:26.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:26.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:26.373+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:27.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:26.881+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:27.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:26.881+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:27.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:26.881+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:27.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:26.881+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:27.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:27.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:27.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:27.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:27.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:27.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:27.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:27.167+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:27.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:27.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:27.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:27.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:27.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:27.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:27.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:27.380+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:28.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:27.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:28.135 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:27.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:28.135 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:27.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:28.135 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:27.855+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:28.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:00:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:00:28.206+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48500 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:00:28.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:28.133+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:28.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:28.133+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:28.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:28.133+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:28.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:28.133+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:28.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:28.374+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:28.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:28.374+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:28.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:28.374+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:28.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:28.375+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:28.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:28.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:28.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:29.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:28.833+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:29.406 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:29.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:29.407 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:29.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:29.407 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:29.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:29.407 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:29.142+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:29.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:29.405+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:29.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:29.405+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:29.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:29.405+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:29.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:29.405+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:30.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:29.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:30.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:29.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:30.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:29.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:30.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:29.814+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:30.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:30.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:30.429 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:30.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:30.429 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:30.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:30.429 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:30.104+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:30.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:30.427+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:30.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:30.427+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:30.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:30.427+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:30.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:30.428+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:31.080+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:31.080+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:31.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:31.080+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:31.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:31.081+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:30.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:30.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:30.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:30.861+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:31.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:31.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:31.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:31.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:31.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:31.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:31.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:31.461+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:32.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:31.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:32.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:31.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:32.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:31.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:32.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:31.840+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:32.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:32.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:32.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:32.084+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:32.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:32.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:32.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:32.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:32.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:32.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:32.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:32.476+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:33.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:33.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:33.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:33.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:33.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:33.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:33.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:33.104+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:33.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:32.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:33.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:32.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:33.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:32.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:33.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:32.883+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:33.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:00:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:00:33.212+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48540 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:00:33.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:33.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:33.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:33.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:33.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:33.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:33.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:33.519+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:34.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:34.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:34.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:34.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:34.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:34.075+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:34.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:33.917+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:34.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:33.917+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:34.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:33.917+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:34.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:33.918+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:34.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:34.470+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:34.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:34.470+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:34.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:34.470+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:34.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:34.470+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:35.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:35.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:35.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:35.059+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:35.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:34.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:35.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:34.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:35.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:34.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:35.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:34.906+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:35.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:35.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:35.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:35.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:35.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:35.493+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:36.014+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:36.014+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:36.014+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:36.014+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:36.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:35.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:36.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:35.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:36.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:35.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:36.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:35.892+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:36.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:36.497+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:36.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:36.497+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:36.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:36.497+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:36.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:36.498+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:37.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:36.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:37.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:36.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:37.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:36.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:36.883+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:37.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:37.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:37.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:37.028+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:37.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:37.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:37.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:37.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:37.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:37.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:37.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:37.528+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:38.220 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:37.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:38.220 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:37.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:38.220 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:37.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:38.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:37.985+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:38.221 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:37.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:38.221 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:37.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:38.221 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:37.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:38.221 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:37.923+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:38.494 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:00:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:00:38.218+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48580 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:00:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:38.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:38.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:38.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:38.493+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:38.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:38.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:38.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:38.898+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:38.995+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:38.995+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:38.995+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:38.996+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:39.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:39.466+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:39.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:39.466+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:39.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:39.466+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:39.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:39.467+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:40.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:39.884+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:40.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:39.884+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:40.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:39.884+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:40.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:39.885+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:39.962+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:39.962+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:39.962+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:39.962+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:40.564 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:40.473+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:40.564 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:40.473+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:40.564 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:40.473+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:40.564 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:40.474+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:41.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:41.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:41.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:41.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:41.009+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:41.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:40.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:41.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:40.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:41.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:40.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:41.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:40.851+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:41.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:41.515+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:41.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:41.515+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:41.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:41.515+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:41.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:41.516+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:42.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:41.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:42.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:41.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:41.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:41.807+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:42.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:42.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:42.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:42.029+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:43.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:42.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:43.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:42.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:43.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:42.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:43.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:42.554+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:43.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:42.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:43.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:42.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:43.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:42.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:43.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:42.761+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:43.348 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:00:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:00:43.224+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48620 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:00:43.349 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:43.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:43.349 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:43.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:43.349 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:43.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:43.349 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:43.046+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:43.729 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:43.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:43.729 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:43.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:43.729 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:43.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:43.729 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:43.506+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:44.002 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:43.777+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:44.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:43.777+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:44.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:43.777+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:44.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:43.777+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:44.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:44.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:44.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:44.002+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:44.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:44.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:44.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:44.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:44.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:44.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:44.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:44.468+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:45.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:44.794+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:45.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:44.794+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:45.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:44.794+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:45.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:44.795+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:45.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:44.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:45.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:44.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:45.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:44.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:45.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:44.968+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:45.429+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:45.429+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:45.429+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:45.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:45.430+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:45.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:45.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:45.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:45.946+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:46.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:45.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:46.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:45.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:46.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:45.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:46.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:45.792+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:46.708 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:46.380+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:46.708 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:46.380+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:46.708 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:46.380+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:46.708 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:46.381+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:47.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:46.744+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:47.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:46.744+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:47.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:46.744+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:47.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:46.745+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:47.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:46.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:47.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:46.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:47.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:46.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:47.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:46.906+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:47.738 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:47.402+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:47.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:47.402+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:47.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:47.402+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:47.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:47.403+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:47.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:47.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:47.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:47.926+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:47.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:47.737+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:47.737+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:47.737+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:48.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:00:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:00:48.230+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48660 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:00:48.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:48.369+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:48.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:48.369+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:48.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:48.369+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:48.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:48.370+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:49.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:48.937+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:48.937+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:48.937+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:48.938+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:48.784+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:48.784+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:48.784+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:49.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:48.785+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:49.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:49.365+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:49.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:49.365+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:49.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:49.365+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:49.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:49.366+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:50.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:49.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:50.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:49.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:50.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:49.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:50.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:49.960+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:50.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:49.817+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:50.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:49.817+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:50.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:49.817+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:50.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:49.817+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:50.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:50.379+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:50.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:50.379+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:50.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:50.379+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:50.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:50.380+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:50.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:50.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:51.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:50.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:51.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:50.835+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:50.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:50.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:50.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:50.958+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:51.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:51.404+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:51.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:51.404+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:51.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:51.404+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:51.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:51.405+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:51.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:51.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:51.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:51.824+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:51.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:51.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:51.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:51.980+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:52.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:52.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:52.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:52.360+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:53.031 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:52.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:53.031 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:52.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:53.031 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:52.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:53.031 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:52.776+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:53.351 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:00:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:00:53.236+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48700 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:00:53.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:53.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:53.352 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:53.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:53.352 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:53.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:53.352 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:53.030+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:53.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:53.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:53.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:53.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:53.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:53.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:53.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:53.350+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:54.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:53.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:54.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:53.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:54.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:53.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:54.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:53.802+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:54.342 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:54.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:54.342 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:54.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:54.342 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:54.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:54.342 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:54.037+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:54.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:54.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:54.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:54.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:54.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:54.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:54.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:54.341+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:55.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:54.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:55.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:54.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:55.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:54.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:55.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:54.779+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:54.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:54.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:54.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:54.991+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:55.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:55.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:55.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:55.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:55.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:55.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:55.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:55.306+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:56.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:55.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:56.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:55.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:55.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:55.816+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:56.020+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:56.020+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:56.020+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:56.021+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:56.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:56.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:56.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:56.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:56.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:56.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:56.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:56.345+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:57.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:56.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:57.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:56.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:57.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:56.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:57.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:56.824+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:57.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:56.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:57.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:56.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:57.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:56.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:57.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:56.991+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:57.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:57.381+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:57.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:57.381+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:57.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:57.381+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:57.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:57.382+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:58.244 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:58.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:58.244 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:58.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:58.245 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:58.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:58.245 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:58.007+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:58.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:57.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:58.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:57.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:58.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:57.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:58.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:57.800+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:58.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:00:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:00:58.242+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48740 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:00:58.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:58.414+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:58.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:58.414+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:58.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:58.414+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:58.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:58.414+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:59.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:58.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:59.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:58.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:59.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:58.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:00:59.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:58.772+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:00:59.419 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:59.041+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:59.419 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:59.041+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:59.419 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:59.041+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:00:59.419 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:00:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:00:59.042+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:00:59.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:59.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:59.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:59.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:59.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:59.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:00:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:00:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:00:59.418+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:00.055 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:59.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:00.056 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:59.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:00.056 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:59.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:00.056 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:00:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:00:59.796+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:00.311 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:00.053+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:00.311 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:00.053+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:00.311 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:00.053+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:00.312 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:00.054+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:00.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:00.397+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:00.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:00.397+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:00.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:00.397+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:00.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:00.398+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:01.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:00.754+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:01.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:00.754+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:01.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:00.754+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:01.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:00.755+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:01.018+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:01.018+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:01.018+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:01.019+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:01.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:01.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:01.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:01.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:01.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:01.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:01.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:01.355+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:02.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:01.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:02.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:01.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:02.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:01.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:02.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:01.985+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:02.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:01.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:02.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:01.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:02.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:01.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:02.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:01.761+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:02.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:02.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:02.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:02.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:02.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:02.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:02.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:02.348+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:03.250 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:02.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:03.251 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:02.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:03.251 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:02.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:03.251 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:02.992+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:03.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:02.780+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:03.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:02.780+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:03.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:02.780+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:03.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:02.780+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:03.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:01:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:01:03.248+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48779 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:01:03.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:03.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:03.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:03.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:03.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:03.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:03.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:03.338+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:03.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:03.794+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:03.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:03.794+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:03.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:03.794+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:03.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:03.794+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:03.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:03.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:03.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:04.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:03.979+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:04.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:04.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:04.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:04.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:04.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:04.339+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:05.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:05.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:05.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:05.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:05.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:05.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:05.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:05.027+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:05.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:04.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:05.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:04.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:05.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:04.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:05.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:04.823+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:05.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:05.383+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:05.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:05.383+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:05.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:05.383+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:05.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:05.384+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:05.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:05.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:05.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:05.982+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:05.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:05.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:05.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:06.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:05.841+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:06.380+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:06.380+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:06.380+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:06.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:06.381+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:06.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:06.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:06.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:06.983+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:06.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:06.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:06.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:06.811+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:07.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:07.377+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:07.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:07.377+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:07.377+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:07.377+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:08.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:08.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:08.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:08.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:08.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:08.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:08.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:08.030+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:08.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:07.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:08.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:07.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:08.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:07.846+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:08.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:07.847+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:08.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:01:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:01:08.255+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48812 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:01:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:08.395+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:08.395+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:08.395+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:08.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:08.396+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:08.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:08.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:08.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:09.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:08.873+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:09.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:09.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:09.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:09.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:09.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:09.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:09.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:09.045+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:09.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:09.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:09.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:09.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:09.441+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:09.996+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:09.996+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:09.996+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:09.997+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:10.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:09.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:10.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:09.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:10.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:09.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:10.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:09.872+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:10.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:10.428+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:10.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:10.428+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:10.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:10.428+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:10.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:10.428+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:10.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:10.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:10.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:10.961+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:10.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:10.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:10.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:10.899+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:11.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:11.409+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:11.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:11.409+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:11.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:11.409+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:11.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:11.410+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:11.972+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:11.972+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:11.972+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:11.973+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:11.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:11.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:11.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:12.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:11.859+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:12.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:12.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:12.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:12.419+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:13.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:12.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:13.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:12.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:13.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:12.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:13.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:12.968+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:13.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:12.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:13.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:12.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:13.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:12.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:13.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:12.829+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:13.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:01:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:01:13.261+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48852 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:01:13.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:13.374+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:13.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:13.374+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:13.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:13.374+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:13.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:13.375+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:13.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:13.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:13.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:13.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:13.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:13.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:13.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:13.823+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:13.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:13.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:13.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:13.980+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:14.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:14.388+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:14.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:14.388+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:14.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:14.388+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:14.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:14.389+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:14.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:14.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:15.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:14.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:14.801+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:14.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:14.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:14.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:14.981+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:15.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:15.368+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:15.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:15.368+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:15.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:15.368+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:15.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:15.368+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:16.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:15.811+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:16.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:15.811+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:16.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:15.811+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:15.811+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:15.947+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:15.947+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:15.947+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:15.948+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:16.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:16.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:16.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:16.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:16.418+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:16.922+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:16.922+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:16.922+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:17.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:16.923+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:16.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:16.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:16.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:16.846+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:17.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:17.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:17.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:17.459+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:17.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:17.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:18.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:17.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:18.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:17.920+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:18.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:17.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:18.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:17.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:18.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:17.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:18.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:17.856+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:18.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:01:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:01:18.267+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48892 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:01:18.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:18.493+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:18.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:18.493+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:18.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:18.493+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:18.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:18.494+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:19.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:18.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:19.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:18.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:19.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:18.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:19.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:18.934+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:19.221 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:18.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:19.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:18.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:19.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:18.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:19.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:18.886+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:19.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:19.485+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:19.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:19.485+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:19.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:19.485+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:19.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:19.486+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:19.959+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:19.959+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:19.959+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:20.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:19.959+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:20.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:19.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:20.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:19.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:20.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:19.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:45.639483+0000 (oldest deadline 2024-02-19T19:51:05.639483+0000) 2024-02-19T20:01:20.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:19.886+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:20.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:20.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:20.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:20.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:20.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:20.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:20.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:20.502+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:20.855+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:20.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:20.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:20.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:20.988+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:21.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:21.500+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:21.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:21.500+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:21.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:21.500+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:21.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:21.501+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:22.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:21.826+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:22.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:22.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:22.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:22.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:22.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:22.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:22.007+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:22.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:22.515+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:22.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:22.515+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:22.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:22.515+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:22.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:22.516+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:23.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:22.852+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:22.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:22.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:22.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:54.062025+0000 (oldest deadline 2024-02-19T19:51:14.062025+0000) 2024-02-19T20:01:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:22.987+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:23.581 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:01:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:01:23.273+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48932 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:01:23.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:23.519+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:23.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:23.519+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:23.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:23.519+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:23.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:23.520+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:24.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:24.033+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:24.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:23.891+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:24.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:24.533+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:24.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:24.533+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:24.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:24.533+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:24.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:24.534+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:24.907+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:25.006+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:25.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:25.486+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:25.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:25.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:25.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:25.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:25.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:25.487+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:26.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:25.887+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:26.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:26.040+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:26.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:26.452+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:26.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:26.452+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:26.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:26.452+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:26.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:26.453+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:27.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:26.932+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:27.069+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:27.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:27.424+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:27.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:27.424+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:27.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:27.424+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:27.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:27.425+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:27.966+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:28.056+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:28.770 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:01:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:01:28.280+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 48972 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:01:28.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:28.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:28.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:28.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:50:56.778232+0000 (oldest deadline 2024-02-19T19:51:16.778232+0000) 2024-02-19T20:01:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:28.397+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:29.082+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:29.015+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:29.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:29.401+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:30.111+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:30.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:29.995+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:30.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:30.367+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:31.392 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:31.067+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:31.393 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:31.029+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:31.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:31.391+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:32.085+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:32.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:31.995+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:32.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:32.408+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:33.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:32.974+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:33.408 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:01:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:01:33.286+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49012 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:01:33.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:33.095+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:33.732 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:33.429+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:34.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:33.947+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:34.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:34.118+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:34.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:34.427+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:35.157+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:35.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:34.935+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:35.427+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:36.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:35.937+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:36.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:36.172+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:36.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:36.463+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:37.169+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:37.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:36.972+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:37.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:37.487+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:38.140+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:38.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:37.977+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:38.770 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:01:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:01:38.293+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49052 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:01:38.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:38.499+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:39.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:38.943+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:39.175+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:39.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:39.526+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:40.185+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:40.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:39.966+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:40.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:40.540+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:41.141+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:41.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:40.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:41.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:40.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:41.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:40.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:41.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:40.996+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:42.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:41.571+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:42.134+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:42.021+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:42.021+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:42.021+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:42.022+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:42.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:42.607+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:43.142+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:43.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:42.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:43.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:42.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:43.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:42.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:43.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:42.976+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:43.582 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:01:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:01:43.299+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49092 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:01:44.000 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:43.626+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:44.180+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:44.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:43.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:44.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:43.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:43.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:43.999+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:44.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:44.593+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:45.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:45.132+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:45.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:45.132+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:45.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:45.132+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:45.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:45.133+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:45.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:45.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:45.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:45.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:45.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:45.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:45.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:45.023+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:46.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:45.608+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:46.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:46.111+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:46.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:46.111+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:46.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:46.111+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:46.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:46.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:46.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:46.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:46.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:46.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:46.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:46.023+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:46.982 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:46.584+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:47.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:47.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:47.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:47.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:47.099+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:47.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:46.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:47.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:46.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:47.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:46.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:47.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:46.981+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:47.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:47.599+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:48.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:48.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:48.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:48.071+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:48.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:47.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:48.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:47.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:48.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:47.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:48.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:47.993+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:48.630 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:01:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:01:48.305+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49132 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:01:49.000 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:48.628+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:49.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:48.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:49.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:48.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:49.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:48.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:49.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:48.999+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:49.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:49.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:49.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:49.064+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:49.638+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:50.314 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:50.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:50.315 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:50.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:50.315 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:50.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:50.315 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:50.047+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:50.315 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:50.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:50.315 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:50.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:50.315 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:50.049+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:50.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:50.049+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:50.671+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:51.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:51.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:51.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:51.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:51.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:51.019+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:51.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:51.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:51.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:51.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:51.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:51.069+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:51.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:51.070+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:51.671+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:52.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:52.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:52.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:52.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:52.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:52.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:52.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:52.051+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:52.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:52.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:52.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:52.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:52.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:52.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:52.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:52.052+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:53.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:52.660+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:53.431 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:01:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:01:53.311+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49172 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:01:53.431 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:53.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:53.431 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:53.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:53.432 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:53.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:53.432 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:53.084+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:53.432 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:53.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:53.432 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:53.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:53.432 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:53.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:53.432 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:53.080+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:54.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:53.686+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:54.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:54.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:54.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:54.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:54.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:54.096+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:54.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:54.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:54.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:54.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:54.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:54.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:54.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:54.061+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:54.662+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:01:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:54.662+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:01:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:54.662+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:01:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:54.663+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:55.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:55.049+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:55.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:55.049+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:55.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:55.049+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:55.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:55.050+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:55.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:55.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:55.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:55.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:55.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:55.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:55.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:55.100+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:56.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:55.634+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:01:56.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:55.634+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:01:56.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:55.634+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:01:56.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:55.634+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:56.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:56.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:56.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:56.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:56.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:56.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:56.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:56.083+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:56.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:56.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:56.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:56.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:56.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:56.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:56.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:56.100+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:57.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:56.608+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:01:57.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:56.608+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:01:57.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:56.608+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:01:57.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:56.608+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:57.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:57.102+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:57.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:57.102+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:57.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:57.102+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:57.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:57.103+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:57.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:57.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:57.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:57.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:57.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:57.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:57.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:57.091+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:58.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:57.563+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:01:58.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:57.563+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:01:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:57.563+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:01:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:57.564+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:58.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:58.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:58.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:58.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:58.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:58.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:58.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:58.087+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:58.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:58.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:58.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:58.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:58.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:58.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:58.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:58.043+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:58.572 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:01:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:01:58.317+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49212 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:01:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:58.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:01:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:58.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:01:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:58.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:01:59.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:58.571+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:01:59.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:59.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:59.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:59.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:59.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:59.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:01:59.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:01:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:01:59.044+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:01:59.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:59.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:59.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:59.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:59.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:59.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:01:59.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:01:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:01:59.065+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:00.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:59.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:00.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:59.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:00.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:59.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:00.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:01:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:01:59.603+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:00.316 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:00.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:00.316 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:00.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:00.316 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:00.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:00.316 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:00.030+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:00.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:00.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:00.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:00.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:00.317 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:00.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:00.317 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:00.035+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:00.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:00.618+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:00.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:00.618+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:00.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:00.618+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:00.995 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:00.619+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:00.992+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:00.992+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:00.992+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:00.993+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:01.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:01.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:01.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:01.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:01.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:01.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:01.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:01.015+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:01.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:01.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:01.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:01.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:01.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:01.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:01.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:01.588+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:02.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:01.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:02.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:01.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:02.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:01.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:02.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:01.982+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:01.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:01.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:01.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:01.950+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:02.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:02.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:02.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:02.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:02.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:02.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:02.918 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:02.568+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:03.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:03.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:03.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:03.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:03.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:03.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:03.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:03.020+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:02.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:02.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:02.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:03.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:02.916+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:03.582 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:02:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:02:03.323+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49252 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:02:03.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:03.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:03.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:03.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:03.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:03.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:03.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:03.553+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:03.947+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:03.947+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:03.947+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:04.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:03.948+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:04.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:04.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:04.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:04.038+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:04.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:04.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:04.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:04.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:04.773 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:04.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:04.773 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:04.518+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:05.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:05.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:05.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:05.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:05.012+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:04.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:04.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:04.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:04.995+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:05.497+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:05.497+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:05.497+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:05.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:05.498+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:06.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:06.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:06.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:06.003+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:06.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:06.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:06.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:06.043+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:06.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:06.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:06.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:06.473+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:07.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:07.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:07.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:07.010+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:07.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:07.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:07.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:07.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:07.062+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:07.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:07.483+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:07.483+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:07.483+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:07.484+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:07.987+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:07.987+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:07.987+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:07.987+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:08.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:08.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:08.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:08.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:08.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:08.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:08.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:08.064+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:08.770 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:02:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:02:08.330+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49292 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:02:08.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:08.470+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:08.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:08.470+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:08.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:08.470+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:08.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:08.471+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:09.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:09.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:09.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:09.063+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:09.064+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:09.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:09.034+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:09.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:09.034+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:09.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:09.034+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:09.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:09.035+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:09.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:09.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:09.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:09.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:09.518+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:10.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:10.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:10.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:10.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:10.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:10.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:10.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:10.016+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:10.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:10.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:10.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:10.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:10.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:10.067+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:10.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:10.482+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:10.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:10.482+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:10.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:10.482+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:10.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:10.483+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:11.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:11.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:11.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:11.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:11.059+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:11.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:11.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:11.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:11.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:11.001+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:11.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:11.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:11.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:11.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:11.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:11.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:11.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:11.458+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:12.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:11.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:12.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:11.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:12.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:11.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:11.997+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:12.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:12.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:12.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:12.071+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:12.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:12.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:12.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:12.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:12.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:12.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:12.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:12.419+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:13.338 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:13.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:13.338 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:13.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:13.339 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:13.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:13.339 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:13.023+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:13.339 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:13.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:13.339 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:13.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:13.339 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:13.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:13.339 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:13.044+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:13.735 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:02:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:02:13.336+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49332 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:02:13.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:13.386+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:13.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:13.386+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:13.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:13.386+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:13.736 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:13.387+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:14.433 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:14.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:14.434 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:14.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:14.434 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:14.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:14.434 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:14.039+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:14.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:14.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:14.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:14.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:14.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:14.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:14.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:14.027+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:14.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:14.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:14.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:14.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:14.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:14.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:14.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:14.432+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:15.012+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:15.012+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:15.012+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:15.013+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:15.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:15.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:15.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:15.037+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:15.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:15.466+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:15.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:15.466+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:15.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:15.466+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:15.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:15.467+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:16.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:16.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:16.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:16.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:16.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:16.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:16.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:16.040+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:16.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:16.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:16.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:16.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:16.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:16.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:16.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:16.082+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:16.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:16.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:16.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:16.456+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:17.482 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:17.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:17.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:17.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:17.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:17.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:17.483 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:17.036+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:17.483 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:17.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:17.483 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:17.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:17.483 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:17.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:17.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:17.112+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:17.480+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:17.480+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:17.480+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:17.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:17.481+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:17.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:17.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:17.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:17.987+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:18.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:18.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:18.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:18.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:18.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:18.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:18.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:18.105+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:18.770 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:02:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:02:18.342+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49372 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:02:18.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:18.519+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:18.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:18.519+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:18.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:18.519+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:18.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:18.521+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:19.070+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:19.070+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:19.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:19.070+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:19.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:19.071+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:19.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:18.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:19.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:18.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:19.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:18.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:19.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:18.967+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:19.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:19.480+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:19.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:19.480+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:19.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:19.480+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:19.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:19.481+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:20.040+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:20.040+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:20.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:20.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:20.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:20.041+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:20.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:20.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:20.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:20.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:20.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:20.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:20.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:20.009+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:20.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:20.446+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:20.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:20.446+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:20.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:20.446+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:20.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:20.447+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:21.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:21.034+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:21.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:21.034+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:21.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:21.034+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:21.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:21.035+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:21.492 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:21.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:21.492 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:21.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:21.492 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:21.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:21.492 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:21.026+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:21.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:21.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:21.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:21.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:21.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:21.487+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:21.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:21.488+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:22.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:22.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:22.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:22.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:22.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:22.036+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:22.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:22.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:22.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:22.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:22.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:22.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:22.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:22.047+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:22.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:22.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:22.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:22.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:22.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:22.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:22.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:22.502+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:23.350 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:23.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:23.350 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:23.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:23.350 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:23.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:23.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:23.068+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:23.351 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:23.047+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:23.351 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:23.047+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:23.351 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:23.047+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:23.351 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:23.047+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:23.735 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:02:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:02:23.348+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49412 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:02:23.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:23.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:23.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:23.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:23.736 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:23.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:23.736 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:23.524+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:24.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:24.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:24.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:24.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:24.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:24.030+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:24.007+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:24.007+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:24.007+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:24.008+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:24.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:24.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:24.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:24.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:24.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:24.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:24.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:24.532+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:24.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:24.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:25.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:24.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:25.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:24.981+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:25.000+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:25.000+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:25.000+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:25.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:25.001+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:25.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:25.499+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:25.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:25.499+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:25.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:25.499+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:25.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:25.500+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:26.000+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:26.000+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:26.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:26.000+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:26.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:26.001+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:26.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:26.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:26.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:26.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:26.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:26.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:26.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:26.005+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:26.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:26.470+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:26.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:26.470+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:26.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:26.470+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:26.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:26.471+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:26.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:26.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:26.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:27.000+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:26.985+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:26.985+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:26.985+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:27.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:26.986+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:27.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:27.512+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:27.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:27.512+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:27.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:27.512+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:27.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:27.513+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:28.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:28.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:28.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:28.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:28.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:28.011+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:28.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:28.034+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:28.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:28.034+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:28.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:28.034+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:28.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:28.034+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:28.770 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:02:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:02:28.354+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49452 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:02:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:28.547+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:28.547+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:28.547+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:28.548+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:29.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:29.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:29.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:29.009+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:29.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:28.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:29.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:28.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:29.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:28.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:29.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:28.996+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:29.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:29.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:29.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:29.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:29.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:29.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:29.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:29.574+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:30.040+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:30.040+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:30.040+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:30.040+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:30.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:29.956+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:30.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:29.956+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:30.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:29.956+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:30.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:29.957+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:30.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:30.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:30.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:30.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:30.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:30.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:30.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:30.526+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:31.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:30.914+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:30.914+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:30.914+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:30.914+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:31.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:31.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:31.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:31.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:31.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:31.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:31.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:31.002+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:31.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:31.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:31.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:31.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:31.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:31.501+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:31.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:31.502+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:32.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:32.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:32.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:32.009+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:32.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:31.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:32.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:31.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:32.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:31.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:32.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:31.959+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:32.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:32.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:32.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:32.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:32.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:32.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:32.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:32.507+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:33.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:32.924+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:33.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:32.924+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:33.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:32.924+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:33.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:32.924+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:33.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:32.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:33.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:32.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:33.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:32.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:33.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:32.993+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:33.735 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:02:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:02:33.360+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49492 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:02:33.736 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:33.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:33.736 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:33.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:33.736 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:33.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:33.736 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:33.464+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:34.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:33.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:34.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:33.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:34.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:33.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:34.014 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:33.892+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:34.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:34.011+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:34.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:34.011+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:34.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:34.011+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:34.012+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:34.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:34.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:34.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:34.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:34.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:34.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:34.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:34.419+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:35.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:34.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:35.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:34.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:35.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:34.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:35.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:34.893+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:35.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:35.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:35.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:35.046+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:35.430+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:35.430+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:35.430+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:35.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:35.431+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:36.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:36.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:36.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:36.038+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:36.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:35.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:36.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:35.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:36.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:35.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:36.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:35.851+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:36.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:36.434+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:36.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:36.434+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:36.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:36.434+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:36.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:36.435+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:37.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:37.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:37.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:37.060+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:36.817+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:36.817+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:36.817+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:36.817+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:37.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:37.414+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:37.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:37.414+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:37.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:37.414+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:37.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:37.415+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:38.017+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:38.017+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:38.017+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:38.018+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:38.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:37.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:38.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:37.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:38.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:37.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:38.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:37.840+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:38.770 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:02:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:02:38.366+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49528 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:02:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:38.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:38.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:38.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:38.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:38.455+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:38.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:38.455+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:39.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:39.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:39.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:39.033+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:38.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:38.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:38.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:38.799+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:39.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:39.441+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:39.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:39.441+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:39.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:39.441+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:39.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:39.442+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:40.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:39.754+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:40.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:39.754+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:40.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:39.754+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:40.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:39.755+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:40.318 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:40.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:40.318 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:40.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:40.318 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:40.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:40.318 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:40.071+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:40.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:40.406+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:40.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:40.406+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:40.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:40.406+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:40.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:40.407+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:41.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:40.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:41.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:40.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:40.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:40.723+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:41.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:41.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:41.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:41.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:41.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:41.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:41.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:41.092+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:41.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:41.384+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:41.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:41.384+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:41.677 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:41.384+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:41.677 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:41.385+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:42.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:41.674+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:42.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:41.674+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:42.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:41.674+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:42.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:41.675+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:42.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:42.099+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:42.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:42.099+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:42.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:42.099+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:42.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:42.100+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:42.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:42.398+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:42.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:42.398+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:42.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:42.398+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:42.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:42.399+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:43.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:42.649+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:43.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:42.650+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:43.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:42.650+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:43.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:42.650+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:43.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:43.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:43.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:43.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:43.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:43.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:43.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:43.092+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:43.667 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:02:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:02:43.372+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49568 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:02:43.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:43.423+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:43.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:43.423+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:43.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:43.423+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:43.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:43.423+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:44.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:43.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:44.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:43.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:44.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:43.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:44.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:43.666+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:44.472 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:44.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:44.472 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:44.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:44.473 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:44.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:44.473 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:44.055+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:44.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:44.712+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:44.712+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:44.712+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:44.712+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:44.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:44.470+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:44.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:44.470+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:44.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:44.470+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:44.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:44.471+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:45.477 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:45.087+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:45.477 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:45.087+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:45.477 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:45.087+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:45.477 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:45.088+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:45.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:45.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:45.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:45.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:45.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:45.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:45.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:45.476+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:45.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:45.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:45.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:45.757+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:46.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:46.055+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:46.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:46.055+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:46.464 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:46.055+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:46.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:46.056+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:46.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:46.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:46.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:46.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:46.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:46.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:46.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:46.463+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:47.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:46.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:47.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:46.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:47.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:46.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:47.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:46.760+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:47.420 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:47.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:47.420 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:47.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:47.421 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:47.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:47.421 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:47.036+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:47.722 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:47.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:47.723 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:47.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:47.723 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:47.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:47.723 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:47.419+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:48.000 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:47.720+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:48.000 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:47.720+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:48.000 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:47.720+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:48.000 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:47.721+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:47.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:47.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:47.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:48.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:47.998+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:48.687 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:02:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:02:48.379+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49609 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:02:48.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:48.454+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:48.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:48.454+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:48.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:48.454+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:48.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:48.455+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:48.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:48.685+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:48.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:48.685+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:48.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:48.685+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:48.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:48.686+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:49.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:48.984+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:48.984+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:48.984+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:48.984+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:49.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:49.674+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:49.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:49.674+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:49.674+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:49.675+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:49.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:49.454+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:49.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:49.454+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:49.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:49.454+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:49.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:49.455+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:50.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:49.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:50.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:49.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:50.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:49.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:50.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:49.994+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:50.648 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:50.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:50.648 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:50.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:50.648 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:50.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:50.649 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:50.461+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:51.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:50.646+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:51.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:50.646+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:51.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:50.646+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:51.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:50.646+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:51.505 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:51.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:51.506 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:51.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:51.506 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:51.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:51.506 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:51.039+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:51.681+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:51.681+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:51.681+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:51.681+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:51.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:51.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:51.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:51.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:51.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:51.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:51.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:51.504+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:52.516 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:52.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:52.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:52.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:52.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:52.089+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:52.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:52.090+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:52.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:52.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:52.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:52.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:52.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:52.516+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:52.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:52.673+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:52.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:52.673+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:52.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:52.673+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:52.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:52.674+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:53.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:53.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:53.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:53.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:53.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:53.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:53.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:53.092+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:53.697 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:02:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:02:53.385+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49648 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:02:53.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:53.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:53.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:53.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:53.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:53.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:53.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:53.469+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:54.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:53.695+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:54.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:53.695+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:54.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:53.695+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:54.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:53.695+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:54.447 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:54.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:54.448 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:54.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:54.448 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:54.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:54.448 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:54.044+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:54.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:54.445+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:54.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:54.445+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:54.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:54.445+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:54.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:54.446+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:54.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:54.670+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:54.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:54.670+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:54.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:54.670+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:54.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:54.670+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:55.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:55.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:55.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:55.003+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:55.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:55.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:55.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:55.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:55.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:55.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:55.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:55.682+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:55.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:55.486+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:55.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:55.486+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:55.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:55.486+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:55.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:55.486+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:56.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:56.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:56.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:56.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:56.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:56.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:56.492 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:56.036+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:56.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:56.699+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:56.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:56.699+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:56.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:56.699+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:56.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:56.699+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:56.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:56.489+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:56.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:56.489+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:56.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:56.489+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:56.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:56.490+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:57.465 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:57.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:57.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:57.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:57.466 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:57.031+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:57.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:57.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:57.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:57.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:57.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:57.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:57.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:57.464+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:57.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:57.651+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:57.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:57.651+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:57.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:57.651+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:57.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:57.651+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:58.393 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:58.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:58.393 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:58.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:58.393 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:58.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:58.393 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:58.079+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:58.700 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:02:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:02:58.390+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49689 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:02:58.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:58.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:58.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:58.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:58.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:58.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:58.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:58.452+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:59.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:58.696+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:59.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:58.696+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:59.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:58.696+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:59.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:58.697+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:59.495 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:59.100+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:59.496 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:59.100+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:59.496 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:59.100+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:02:59.496 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:02:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:02:59.100+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:02:59.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:59.493+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:59.493+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:59.493+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:02:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:02:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:02:59.494+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:02:59.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:59.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:59.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:59.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:59.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:59.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:02:59.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:02:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:02:59.695+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:00.115 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:00.111+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:00.115 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:00.111+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:00.509 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:00.111+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:00.509 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:00.112+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:00.672+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:00.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:00.672+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:00.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:00.672+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:00.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:00.672+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:00.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:00.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:00.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:00.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:00.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:00.507+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:00.772 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:00.508+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:01.488 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:01.140+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:01.489 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:01.140+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:01.489 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:01.140+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:01.489 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:01.140+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:01.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:01.486+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:01.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:01.486+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:01.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:01.486+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:01.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:01.487+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:01.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:01.628+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:01.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:01.628+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:01.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:01.628+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:01.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:01.629+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:02.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:02.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:02.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:02.117+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:02.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:02.596+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:02.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:02.597+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:02.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:02.597+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:02.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:02.597+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:02.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:02.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:02.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:02.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:02.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:02.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:02.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:02.475+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:03.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:03.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:03.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:03.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:03.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:03.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:03.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:03.130+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:03.738 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:03:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:03:03.397+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49728 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:03:03.738 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:03.509+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:03.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:03.509+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:03.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:03.509+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:03.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:03.510+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:03.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:03.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:03.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:03.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:03.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:03.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:03.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:03.560+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:04.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:04.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:04.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:04.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:04.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:04.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:04.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:04.111+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:04.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:04.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:04.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:04.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:04.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:04.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:04.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:04.576+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:04.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:04.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:04.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:04.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:04.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:04.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:04.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:04.506+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:05.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:05.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:05.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:05.086+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:05.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:05.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:05.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:05.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:05.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:05.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:05.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:05.554+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:05.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:05.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:05.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:05.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:05.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:05.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:05.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:05.556+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:06.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:06.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:06.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:06.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:06.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:06.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:06.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:06.053+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:06.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:06.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:06.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:06.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:06.554+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:06.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:06.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:06.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:06.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:06.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:06.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:06.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:06.511+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:07.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:07.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:07.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:07.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:07.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:07.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:07.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:07.099+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:07.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:07.584+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:07.584+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:07.584+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:07.585+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:07.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:07.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:07.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:07.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:07.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:07.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:07.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:07.511+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:08.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:08.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:08.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:08.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:08.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:08.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:08.406 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:08.075+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:08.770 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:03:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:03:08.403+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49770 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:03:08.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:08.588+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:08.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:08.588+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:08.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:08.588+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:08.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:08.588+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:08.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:08.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:08.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:08.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:08.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:08.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:08.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:08.540+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:09.506 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:09.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:09.506 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:09.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:09.506 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:09.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:09.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:09.092+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:09.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:09.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:09.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:09.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:09.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:09.617+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:09.504+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:09.504+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:09.504+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:09.505+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:10.392 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:10.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:10.392 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:10.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:10.393 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:10.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:10.393 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:10.140+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:10.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:10.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:10.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:10.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:10.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:10.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:10.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:10.514+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:10.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:10.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:10.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:10.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:10.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:10.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:10.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:10.617+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:11.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:11.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:11.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:11.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:11.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:11.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:11.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:11.106+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:12.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:11.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:12.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:11.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:12.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:11.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:12.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:11.610+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:12.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:11.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:12.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:11.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:12.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:11.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:12.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:11.519+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:12.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:12.109+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:12.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:12.109+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:12.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:12.109+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:12.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:12.109+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:13.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:12.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:13.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:12.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:13.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:12.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:13.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:12.587+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:13.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:12.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:13.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:12.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:13.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:12.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:13.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:12.560+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:13.412 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:13.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:13.412 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:13.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:13.412 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:13.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:13.412 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:13.148+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:13.739 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:03:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:03:13.409+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49809 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:03:13.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:13.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:13.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:13.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:13.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:13.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:13.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:13.635+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:13.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:13.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:13.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:13.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:13.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:13.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:13.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:13.569+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:14.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:14.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:14.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:14.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:14.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:14.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:14.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:14.158+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:14.613+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:14.613+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:15.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:14.613+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:15.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:14.614+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:15.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:14.564+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:15.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:14.564+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:15.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:14.564+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:15.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:14.564+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:15.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:15.184+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:15.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:15.184+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:15.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:15.184+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:15.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:15.185+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:15.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:15.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:15.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:15.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:15.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:15.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:15.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:15.623+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:15.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:15.520+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:15.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:15.520+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:15.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:15.520+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:15.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:15.521+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:16.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:16.211+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:16.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:16.211+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:16.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:16.211+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:16.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:16.211+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:17.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:16.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:17.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:16.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:17.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:16.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:17.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:16.658+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:17.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:16.534+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:17.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:16.534+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:17.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:16.534+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:17.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:16.535+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:17.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:17.237+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:17.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:17.237+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:17.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:17.237+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:17.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:17.237+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:17.627+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:17.627+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:17.627+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:17.628+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:18.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:17.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:18.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:17.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:18.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:17.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:18.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:17.529+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:18.514 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:03:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:03:18.416+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49850 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:03:18.514 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:18.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:18.514 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:18.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:18.514 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:18.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:18.514 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:18.225+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:18.773 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:18.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:18.773 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:18.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:18.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:18.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:18.774 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:18.632+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:18.774 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:18.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:18.774 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:18.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:18.774 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:18.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:18.774 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:18.512+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:19.490 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:19.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:19.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:19.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:19.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:19.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:19.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:19.235+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:19.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:19.488+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:19.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:19.488+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:19.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:19.488+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:19.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:19.489+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:19.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:19.673+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:19.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:19.673+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:19.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:19.673+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:19.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:19.674+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:20.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:20.195+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:20.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:20.195+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:20.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:20.195+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:20.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:20.196+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:20.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:20.715+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:20.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:20.715+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:20.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:20.715+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:20.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:20.716+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:20.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:20.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:20.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:20.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:20.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:20.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:20.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:20.489+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:21.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:21.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:21.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:21.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:21.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:21.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:21.469 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:21.164+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:21.466+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:21.466+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:21.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:21.466+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:21.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:21.467+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:21.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:21.704+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:21.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:21.704+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:21.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:21.704+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:21.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:21.704+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:22.474 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:22.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:22.474 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:22.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:22.474 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:22.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:22.474 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:22.142+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:22.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:22.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:22.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:22.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:22.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:22.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:22.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:22.473+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:23.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:22.724+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:23.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:22.724+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:23.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:22.724+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:23.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:22.724+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:23.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:23.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:23.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:23.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:23.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:23.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:23.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:23.117+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:23.739 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:03:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:03:23.423+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49889 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:03:23.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:23.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:23.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:23.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:23.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:23.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:23.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:23.520+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:24.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:23.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:24.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:23.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:24.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:23.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:24.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:23.766+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:24.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:24.137+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:24.492 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:24.137+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:24.492 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:24.137+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:24.492 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:24.138+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:24.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:24.730+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:24.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:24.730+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:24.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:24.730+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:24.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:24.731+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:24.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:24.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:24.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:24.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:24.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:24.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:24.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:24.490+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:25.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:25.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:25.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:25.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:25.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:25.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:25.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:25.098+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:25.763+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:26.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:25.763+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:26.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:25.763+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:26.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:25.764+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:26.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:25.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:26.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:25.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:26.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:25.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:26.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:25.532+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:26.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:26.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:26.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:26.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:26.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:26.123+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:26.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:26.124+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:26.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:26.545+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:26.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:26.545+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:26.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:26.545+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:26.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:26.546+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:27.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:26.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:27.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:26.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:27.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:26.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:27.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:26.784+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:27.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:27.100+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:27.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:27.100+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:27.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:27.100+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:27.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:27.101+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:27.818 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:27.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:27.818 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:27.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:27.818 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:27.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:27.818 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:27.550+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:28.150 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:27.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:28.150 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:27.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:28.150 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:27.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:28.150 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:27.817+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:28.431 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:28.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:28.431 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:28.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:28.431 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:28.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:28.432 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:28.148+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:28.770 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:03:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:03:28.429+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49930 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:03:28.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:28.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:28.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:28.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:28.573+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:29.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:28.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:29.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:28.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:29.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:28.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:29.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:28.813+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:29.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:29.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:29.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:29.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:29.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:29.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:29.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:29.178+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:29.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:29.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:29.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:29.768+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:30.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:29.577+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:30.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:29.577+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:30.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:29.577+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:30.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:29.578+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:30.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:30.226+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:30.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:30.226+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:30.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:30.226+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:30.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:30.227+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:30.800 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:30.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:30.801 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:30.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:30.801 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:30.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:30.801 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:30.529+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:31.237 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:30.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:31.238 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:30.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:31.238 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:30.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:31.238 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:30.798+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:31.492 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:31.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:31.492 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:31.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:31.492 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:31.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:31.492 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:31.236+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:31.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:31.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:31.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:31.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:31.490+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:32.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:31.776+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:32.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:31.776+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:32.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:31.776+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:32.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:31.777+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:32.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:32.252+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:32.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:32.252+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:32.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:32.252+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:32.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:32.252+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:32.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:32.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:32.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:32.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:32.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:32.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:32.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:32.462+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:32.824+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:33.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:32.824+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:33.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:32.824+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:33.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:32.825+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:33.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:03:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:03:33.435+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 49969 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:03:33.556 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:33.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:33.556 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:33.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:33.556 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:33.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:33.556 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:33.300+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:33.557 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:33.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:33.557 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:33.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:33.557 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:33.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:33.557 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:33.468+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:33.821+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:33.821+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:33.821+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:34.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:33.822+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:34.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:34.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:34.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:34.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:34.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:34.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:34.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:34.307+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:34.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:34.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:34.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:34.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:34.474+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:35.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:34.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:34.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:34.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:34.801+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:35.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:35.291+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:35.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:35.291+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:35.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:35.291+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:35.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:35.292+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:35.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:35.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:35.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:35.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:35.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:35.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:35.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:35.489+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:35.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:35.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:35.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:36.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:35.757+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:36.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:36.338+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:36.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:36.338+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:36.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:36.338+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:36.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:36.339+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:36.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:36.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:36.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:36.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:36.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:36.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:36.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:36.509+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:36.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:36.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:36.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:37.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:36.797+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:37.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:37.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:37.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:37.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:37.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:37.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:37.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:37.320+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:37.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:37.463+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:37.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:37.463+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:37.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:37.463+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:37.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:37.463+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:37.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:37.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:37.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:37.807+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:38.770 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:03:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:03:38.442+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50010 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:03:38.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:38.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:38.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:38.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:38.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:38.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:38.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:38.332+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:38.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:38.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:38.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:38.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:38.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:38.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:38.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:38.487+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:38.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:38.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:39.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:38.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:39.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:38.793+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:39.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:39.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:39.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:39.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:39.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:39.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:39.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:39.306+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:39.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:39.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:39.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:39.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:39.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:39.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:39.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:39.455+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:40.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:39.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:40.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:39.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:39.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:39.826+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:40.746 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:40.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:40.747 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:40.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:40.747 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:40.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:40.747 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:40.312+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:40.747 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:40.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:40.747 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:40.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:40.747 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:40.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:40.747 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:40.459+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:40.866+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:40.866+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:40.866+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:41.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:40.867+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:41.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:41.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:41.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:41.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:41.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:41.494+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:41.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:41.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:41.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:41.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:41.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:41.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:41.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:41.319+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:41.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:41.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:42.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:41.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:42.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:41.886+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:42.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:42.354+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:42.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:42.354+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:42.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:42.354+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:42.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:42.354+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:42.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:42.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:42.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:42.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:42.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:42.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:42.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:42.462+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:43.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:42.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:43.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:42.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:43.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:42.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:43.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:42.855+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:43.571 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:03:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:03:43.448+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50049 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:03:43.571 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:43.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:43.571 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:43.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:43.571 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:43.306+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:43.571 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:43.306+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:43.572 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:43.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:43.572 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:43.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:43.572 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:43.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:43.572 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:43.416+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:43.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:43.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:44.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:43.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:44.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:43.877+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:44.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:44.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:44.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:44.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:44.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:44.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:44.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:44.318+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:44.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:44.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:44.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:44.429+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:44.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:44.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:44.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:44.897+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:45.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:45.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:45.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:45.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:45.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:45.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:45.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:45.356+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:45.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:45.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:45.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:45.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:45.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:45.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:45.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:45.428+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:45.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:45.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:45.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:46.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:45.914+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:46.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:46.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:46.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:46.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:46.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:46.314+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:46.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:46.315+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:46.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:46.433+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:46.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:46.433+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:46.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:46.433+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:46.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:46.434+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:47.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:46.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:47.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:46.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:47.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:46.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:47.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:46.908+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:47.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:47.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:47.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:47.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:47.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:47.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:47.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:47.341+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:47.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:47.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:47.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:47.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:47.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:47.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:47.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:47.413+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:48.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:47.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:48.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:47.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:48.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:47.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:48.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:47.911+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:48.577 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:03:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:03:48.455+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50090 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:03:48.577 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:48.303+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:48.577 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:48.303+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:48.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:48.303+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:48.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:48.304+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:48.578 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:48.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:48.578 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:48.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:48.578 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:48.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:48.578 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:48.427+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:48.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:48.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:48.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:49.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:48.894+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:49.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:49.290+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:49.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:49.290+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:49.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:49.290+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:49.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:49.291+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:49.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:49.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:49.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:49.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:49.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:49.457+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:49.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:49.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:49.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:50.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:49.904+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:50.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:50.310+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:50.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:50.310+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:50.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:50.310+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:50.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:50.311+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:50.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:50.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:50.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:50.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:50.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:50.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:50.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:50.409+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:50.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:50.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:51.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:50.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:51.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:50.933+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:51.403+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:51.403+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:51.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:51.403+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:51.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:51.403+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:51.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:51.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:51.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:51.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:51.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:51.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:51.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:51.284+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:52.250 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:51.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:52.250 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:51.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:52.250 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:51.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:52.250 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:51.942+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:52.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:52.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:52.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:52.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:52.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:52.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:52.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:52.452+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:52.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:52.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:52.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:52.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:52.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:52.248+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:52.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:52.249+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:53.267 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:52.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:53.267 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:52.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:53.267 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:52.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:53.267 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:52.944+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:53.520 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:03:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:03:53.461+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50129 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:03:53.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:53.265+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:53.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:53.265+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:53.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:53.265+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:53.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:53.266+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:53.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:53.417+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:53.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:53.417+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:53.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:53.417+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:53.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:53.417+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:54.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:53.905+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:54.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:53.905+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:54.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:53.905+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:54.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:53.906+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:54.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:54.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:54.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:54.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:54.263+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:54.398+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:54.398+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:54.398+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:54.398+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:55.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:54.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:55.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:54.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:55.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:54.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:55.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:54.923+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:55.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:55.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:55.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:55.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:55.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:55.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:55.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:55.441+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:55.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:55.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:55.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:55.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:55.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:55.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:55.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:55.309+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:55.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:55.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:56.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:55.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:56.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:55.879+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:56.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:56.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:56.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:56.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:56.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:56.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:56.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:56.423+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:56.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:56.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:56.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:56.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:56.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:56.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:56.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:56.299+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:56.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:56.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:56.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:57.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:56.858+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:57.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:57.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:57.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:57.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:57.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:57.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:57.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:57.459+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:57.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:57.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:57.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:57.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:57.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:57.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:57.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:57.328+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:58.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:57.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:58.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:57.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:58.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:57.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:58.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:57.837+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:58.588 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:03:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:03:58.468+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50169 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:03:58.588 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:58.313+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:58.588 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:58.313+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:58.588 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:58.313+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:58.589 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:58.314+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:58.589 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:58.444+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:58.589 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:58.444+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:58.589 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:58.444+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:58.589 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:58.445+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:59.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:58.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:59.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:58.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:59.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:58.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:03:59.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:58.852+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:03:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:59.279+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:59.279+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:59.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:59.279+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:03:59.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:03:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:03:59.280+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:03:59.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:59.410+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:59.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:59.410+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:59.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:59.410+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:03:59.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:03:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:03:59.410+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:00.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:59.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:00.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:59.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:00.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:59.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:00.250 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:03:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:03:59.849+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:00.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:00.246+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:00.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:00.246+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:00.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:00.247+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:00.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:00.247+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:00.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:00.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:00.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:00.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:00.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:00.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:00.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:00.408+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:00.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:01.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:00.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:01.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:00.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:01.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:00.862+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:01.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:01.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:01.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:01.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:01.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:01.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:01.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:01.298+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:01.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:01.453+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:01.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:01.453+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:01.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:01.453+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:01.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:01.454+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:02.262 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:01.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:02.262 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:01.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:02.263 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:01.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:02.263 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:01.876+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:02.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:02.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:02.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:02.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:02.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:02.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:02.261+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:02.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:02.448+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:02.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:02.448+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:02.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:02.448+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:02.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:02.449+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:03.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:02.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:03.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:02.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:03.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:02.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:03.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:02.919+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:03.581 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:04:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:04:03.474+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50209 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:04:03.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:03.414+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:03.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:03.414+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:03.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:03.414+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:03.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:03.414+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:03.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:03.273+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:03.583 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:03.273+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:03.583 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:03.273+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:03.583 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:03.275+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:04.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:03.897+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:04.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:03.897+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:04.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:03.897+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:04.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:03.898+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:04.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:04.444+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:04.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:04.444+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:04.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:04.444+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:04.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:04.445+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:04.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:04.253+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:04.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:04.253+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:04.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:04.253+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:04.522 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:04.254+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:04.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:04.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:04.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:04.884+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:05.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:05.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:05.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:05.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:05.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:05.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:05.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:05.474+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:05.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:05.302+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:05.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:05.302+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:05.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:05.302+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:05.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:05.302+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:05.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:05.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:05.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:06.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:05.929+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:06.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:06.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:06.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:06.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:06.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:06.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:06.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:06.333+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:06.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:06.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:06.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:06.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:06.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:06.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:06.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:06.502+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:06.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:06.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:06.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:06.976+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:07.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:07.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:07.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:07.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:07.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:07.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:07.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:07.300+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:07.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:07.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:07.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:07.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:07.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:07.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:07.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:07.460+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:07.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:07.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:07.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:07.944+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:08.602 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:04:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:04:08.480+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50249 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:04:08.603 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:08.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:08.603 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:08.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:08.603 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:08.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:08.604 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:08.320+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:08.604 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:08.503+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:08.604 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:08.503+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:08.604 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:08.503+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:08.604 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:08.504+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:09.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:08.994+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:08.994+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:08.994+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:08.995+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:09.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:09.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:09.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:09.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:09.473+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:09.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:09.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:09.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:09.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:09.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:09.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:09.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:09.367+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:10.325 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:10.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:10.325 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:10.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:10.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:10.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:10.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:10.036+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:10.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:10.398+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:10.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:10.398+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:10.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:10.398+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:10.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:10.399+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:10.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:10.517+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:10.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:10.517+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:10.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:10.517+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:10.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:10.517+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:11.357 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:11.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:11.357 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:11.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:11.357 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:11.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:11.357 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:11.025+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:11.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:11.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:11.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:11.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:11.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:11.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:11.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:11.356+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:11.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:11.501+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:11.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:11.501+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:11.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:11.501+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:11.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:11.501+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:12.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:12.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:12.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:12.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:12.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:12.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:12.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:12.006+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:12.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:12.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:12.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:12.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:12.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:12.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:12.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:12.382+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:12.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:12.453+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:12.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:12.453+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:12.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:12.453+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:12.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:12.454+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:12.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:12.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:12.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:12.978+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:13.742 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:04:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:04:13.487+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50289 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:04:13.743 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:13.379+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:13.743 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:13.379+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:13.743 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:13.379+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:13.743 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:13.380+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:13.744 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:13.499+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:13.744 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:13.499+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:13.744 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:13.499+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:13.744 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:13.499+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:13.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:13.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:14.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:13.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:14.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:13.993+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:14.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:14.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:14.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:14.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:14.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:14.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:14.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:14.531+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:14.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:14.424+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:14.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:14.424+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:14.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:14.424+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:14.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:14.425+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:14.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:15.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:14.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:15.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:14.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:15.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:14.993+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:15.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:15.396+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:15.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:15.396+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:15.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:15.396+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:15.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:15.396+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:15.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:15.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:15.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:15.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:15.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:15.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:15.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:15.569+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:15.994+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:15.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:16.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:15.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:16.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:15.995+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:16.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:16.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:16.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:16.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:16.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:16.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:16.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:16.391+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:16.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:16.552+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:16.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:16.552+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:16.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:16.552+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:16.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:16.553+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:16.951+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:17.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:16.951+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:17.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:16.951+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:17.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:16.952+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:17.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:17.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:17.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:17.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:17.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:17.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:17.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:17.416+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:17.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:17.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:17.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:17.549+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:17.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:17.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:18.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:17.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:18.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:17.911+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:18.770 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:04:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:04:18.494+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50329 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:04:18.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:18.462+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:18.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:18.462+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:18.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:18.462+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:18.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:18.462+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:18.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:18.595+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:18.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:18.595+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:18.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:18.595+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:18.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:18.595+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:19.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:18.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:19.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:18.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:19.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:18.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:19.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:18.958+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:19.451+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:19.451+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:19.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:19.451+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:19.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:19.451+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:19.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:19.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:19.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:19.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:19.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:19.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:19.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:19.572+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:20.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:19.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:20.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:19.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:20.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:19.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:20.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:19.970+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:20.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:20.497+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:20.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:20.497+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:20.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:20.497+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:20.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:20.498+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:20.935 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:20.607+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:20.935 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:20.607+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:20.935 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:20.607+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:20.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:20.607+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:21.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:20.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:21.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:20.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:21.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:20.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:21.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:20.934+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:21.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:21.544+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:21.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:21.544+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:21.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:21.544+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:21.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:21.545+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:21.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:21.655+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:21.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:21.655+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:21.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:21.655+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:21.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:21.655+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:21.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:21.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:21.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:21.934+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:22.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:22.580+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:22.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:22.580+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:22.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:22.580+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:22.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:22.581+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:22.962 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:22.621+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:22.962 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:22.621+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:22.962 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:22.621+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:22.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:22.622+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:22.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:22.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:23.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:22.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:23.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:22.960+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:23.770 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:04:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:04:23.501+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50369 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:04:23.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:23.557+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:23.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:23.557+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:23.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:23.557+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:23.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:23.558+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:23.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:23.630+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:23.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:23.630+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:23.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:23.630+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:23.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:23.631+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:24.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:23.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:24.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:23.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:24.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:23.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:24.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:23.949+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:24.820 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:24.550+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:24.820 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:24.550+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:24.820 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:24.550+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:24.820 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:24.551+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:24.821 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:24.650+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:24.821 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:24.650+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:24.821 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:24.650+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:24.821 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:24.650+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:25.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:24.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:25.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:24.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:25.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:24.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:25.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:24.949+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:25.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:25.578+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:25.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:25.578+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:25.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:25.578+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:25.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:25.578+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:25.993 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:25.684+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:25.993 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:25.684+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:25.993 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:25.684+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:25.993 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:25.685+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:26.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:25.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:26.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:25.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:26.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:25.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:26.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:25.990+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:27.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:26.542+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:27.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:26.542+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:27.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:26.542+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:27.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:26.543+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:27.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:26.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:27.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:26.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:27.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:26.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:27.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:26.659+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:27.506 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:27.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:27.506 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:27.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:27.506 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:27.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:27.506 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:27.032+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:27.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:27.504+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:27.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:27.504+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:27.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:27.504+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:27.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:27.505+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:27.690+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:27.690+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:27.690+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:27.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:27.691+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:28.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:28.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:28.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:28.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:28.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:28.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:28.510 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:28.082+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:28.770 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:04:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:04:28.507+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50409 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:04:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:28.740+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:28.740+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:28.740+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:28.741+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:28.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:28.543+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:28.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:28.543+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:28.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:28.543+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:28.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:28.544+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:29.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:29.094+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:29.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:29.095+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:29.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:29.095+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:29.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:29.096+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:30.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:29.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:30.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:29.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:30.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:29.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:30.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:29.570+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:30.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:29.785+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:30.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:29.785+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:30.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:29.785+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:30.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:29.785+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:30.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:30.067+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:30.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:30.067+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:30.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:30.067+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:30.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:30.068+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:30.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:30.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:30.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:30.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:30.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:30.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:30.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:30.801+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:30.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:30.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:30.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:30.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:30.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:30.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:30.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:30.591+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:31.050+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:31.050+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:31.050+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:31.051+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:31.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:31.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:32.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:31.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:32.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:31.813+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:31.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:31.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:31.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:31.570+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:32.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:32.059+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:32.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:32.059+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:32.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:32.059+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:32.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:32.060+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:33.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:32.789+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:33.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:32.789+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:33.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:32.789+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:33.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:32.790+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:32.568+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:32.568+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:32.568+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:32.569+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:33.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:33.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:33.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:33.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:33.515 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:33.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:33.516 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:33.054+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:33.771 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:04:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:04:33.513+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50449 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:04:33.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:33.595+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:33.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:33.595+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:33.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:33.595+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:33.773 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:33.596+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:34.074 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:33.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:34.075 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:33.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:34.075 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:33.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:34.075 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:33.788+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:34.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:34.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:34.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:34.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:34.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:34.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:34.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:34.074+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:34.577+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:35.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:34.577+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:35.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:34.577+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:35.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:34.578+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:35.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:34.806+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:35.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:34.806+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:35.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:34.806+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:35.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:34.806+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:35.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:35.083+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:35.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:35.083+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:35.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:35.083+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:35.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:35.084+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:36.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:35.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:36.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:35.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:35.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:35.581+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:36.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:35.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:36.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:35.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:36.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:35.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:36.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:35.809+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:36.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:36.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:36.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:36.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:36.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:36.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:36.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:36.094+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:37.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:36.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:37.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:36.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:37.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:36.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:37.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:36.850+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:37.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:36.618+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:37.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:36.618+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:37.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:36.618+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:37.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:36.619+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:37.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:37.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:37.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:37.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:37.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:37.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:37.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:37.050+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:37.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:37.602+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:37.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:37.602+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:37.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:37.602+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:37.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:37.603+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:38.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:38.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:38.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:38.032+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:38.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:37.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:38.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:37.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:38.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:37.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:38.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:37.889+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:38.908 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:04:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:04:38.519+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50489 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:04:38.909 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:38.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:38.909 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:38.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:38.909 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:38.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:38.909 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:38.650+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:39.024+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:39.024+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:39.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:39.024+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:39.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:39.025+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:38.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:38.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:38.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:38.907+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:40.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:39.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:40.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:39.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:40.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:39.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:40.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:39.894+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:39.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:39.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:39.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:39.696+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:40.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:40.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:40.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:40.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:40.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:40.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:40.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:40.020+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:40.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:40.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:40.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:40.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:40.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:40.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:40.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:40.680+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:40.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:40.844+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:40.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:40.844+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:40.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:40.844+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:40.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:40.845+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:40.985+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:40.985+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:41.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:40.985+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:41.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:40.986+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:42.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:41.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:42.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:41.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:42.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:41.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:42.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:41.710+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:42.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:41.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:42.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:41.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:42.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:41.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:42.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:41.862+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:42.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:42.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:42.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:42.006+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:42.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:42.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:42.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:42.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:42.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:42.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:42.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:42.714+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:42.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:42.876+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:42.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:42.876+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:42.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:42.876+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:42.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:42.877+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:43.252 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:42.985+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:43.252 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:42.985+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:43.252 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:42.985+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:43.252 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:42.985+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:43.848 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:04:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:04:43.525+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50529 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:04:43.849 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:43.722+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:43.849 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:43.722+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:43.849 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:43.722+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:43.849 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:43.723+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:43.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:43.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:43.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:44.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:43.964+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:43.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:43.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:43.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:43.848+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:44.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:44.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:44.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:44.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:44.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:44.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:44.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:44.830+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:44.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:44.681+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:44.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:44.681+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:44.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:44.681+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:44.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:44.682+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:44.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:44.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:44.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:44.985+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:46.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:45.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:46.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:45.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:46.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:45.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:46.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:45.677+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:46.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:45.840+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:46.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:45.840+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:46.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:45.840+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:46.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:45.840+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:46.324 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:46.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:46.324 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:46.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:46.324 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:46.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:46.324 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:46.034+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:47.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:46.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:47.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:46.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:47.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:46.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:47.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:46.655+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:47.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:46.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:47.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:46.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:47.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:46.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:47.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:46.838+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:47.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:47.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:47.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:47.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:47.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:47.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:47.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:47.006+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:48.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:47.618+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:48.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:47.618+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:48.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:47.618+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:48.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:47.619+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:48.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:47.834+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:48.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:47.834+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:48.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:47.834+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:48.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:47.835+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:48.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:48.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:48.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:48.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:48.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:48.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:48.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:48.006+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:48.881 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:04:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:04:48.532+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50569 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:04:48.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:48.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:48.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:48.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:48.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:48.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:48.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:48.658+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:48.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:48.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:49.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:48.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:49.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:48.961+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:48.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:48.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:48.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:49.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:48.879+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:49.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:49.682+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:49.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:49.682+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:49.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:49.682+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:49.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:49.683+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:49.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:49.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:49.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:49.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:49.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:49.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:49.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:49.907+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:49.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:49.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:49.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:50.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:49.975+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:50.947 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:50.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:50.947 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:50.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:50.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:50.900+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:50.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:50.901+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:50.948 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:50.683+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:50.949 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:50.683+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:50.949 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:50.683+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:50.949 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:50.684+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:50.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:50.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:50.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:50.946+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:51.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:51.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:51.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:51.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:51.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:51.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:51.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:51.641+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:51.933+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:51.933+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:52.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:51.933+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:52.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:51.934+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:52.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:51.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:52.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:51.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:52.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:51.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:52.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:51.915+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:52.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:52.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:52.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:52.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:52.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:52.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:52.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:52.661+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:53.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:52.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:53.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:52.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:53.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:52.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:53.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:52.869+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:53.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:52.940+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:53.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:52.940+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:53.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:52.940+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:53.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:52.941+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:53.907 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:04:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:04:53.538+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50609 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:04:53.907 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:53.710+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:53.908 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:53.710+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:53.908 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:53.710+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:53.908 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:53.710+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:54.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:53.905+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:54.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:53.905+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:54.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:53.905+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:54.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:53.906+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:53.951+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:53.951+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:53.951+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:53.951+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:54.997 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:54.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:54.997 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:54.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:54.997 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:54.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:54.998 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:54.915+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:54.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:54.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:54.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:54.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:54.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:54.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:54.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:54.687+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:55.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:54.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:55.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:54.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:55.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:54.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:55.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:54.996+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:55.885 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:55.674+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:55.885 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:55.674+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:55.885 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:55.674+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:55.885 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:55.675+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:56.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:55.946+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:56.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:55.947+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:55.947+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:55.947+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:56.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:55.882+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:56.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:55.882+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:56.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:55.882+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:56.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:55.883+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:56.853 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:56.722+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:56.854 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:56.722+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:56.854 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:56.722+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:56.854 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:56.723+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:56.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:57.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:56.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:57.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:56.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:57.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:56.852+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:57.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:56.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:57.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:56.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:57.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:56.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:57.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:56.905+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:58.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:57.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:58.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:57.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:58.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:57.857+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:58.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:57.857+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:57.843+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:57.843+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:57.843+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:57.844+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:58.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:57.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:58.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:57.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:58.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:57.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:58.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:57.713+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:58.801 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:04:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:04:58.544+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50649 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:04:58.802 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:58.697+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:58.802 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:58.697+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:58.802 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:58.697+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:04:58.802 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:58.697+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:04:59.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:58.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:59.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:58.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:59.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:58.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:04:59.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:58.800+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:04:59.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:58.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:59.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:58.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:59.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:58.821+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:04:59.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:58.821+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:00.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:59.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:00.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:59.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:00.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:59.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:00.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:04:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:04:59.803+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:00.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:59.822+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:00.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:59.822+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:00.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:59.822+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:00.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:04:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:04:59.823+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:00.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:59.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:00.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:59.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:00.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:59.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:00.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:04:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:04:59.743+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:00.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:00.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:01.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:00.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:01.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:00.849+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:00.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:00.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:00.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:00.769+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:01.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:00.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:01.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:00.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:01.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:00.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:01.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:00.778+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:01.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:02.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:01.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:02.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:01.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:02.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:01.827+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:02.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:01.723+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:02.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:01.723+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:02.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:01.723+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:02.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:01.724+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:02.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:01.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:02.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:01.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:02.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:01.816+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:02.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:01.816+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:03.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:02.844+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:02.844+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:02.844+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:02.845+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:03.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:02.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:03.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:02.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:03.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:02.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:03.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:02.693+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:03.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:02.862+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:03.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:02.862+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:03.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:02.862+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:03.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:02.863+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:03.827 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:05:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:05:03.551+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50689 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:05:03.827 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:03.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:03.827 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:03.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:03.827 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:03.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:03.827 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:03.712+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:04.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:03.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:04.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:03.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:04.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:03.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:04.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:03.890+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:03.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:03.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:03.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:04.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:03.825+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:05.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:04.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:05.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:04.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:05.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:04.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:05.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:04.797+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:05.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:04.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:05.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:04.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:05.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:04.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:05.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:04.889+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:05.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:04.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:05.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:04.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:05.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:04.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:05.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:04.699+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:05.912 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:05.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:05.913 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:05.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:05.913 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:05.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:05.913 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:05.761+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:05.913 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:05.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:05.913 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:05.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:05.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:05.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:05.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:05.650+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:05.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:05.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:05.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:06.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:05.911+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:06.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:06.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:06.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:06.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:06.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:06.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:06.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:06.644+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:06.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:06.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:06.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:06.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:06.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:06.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:06.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:06.776+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:06.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:06.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:06.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:06.934+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:07.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:07.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:07.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:07.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:07.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:07.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:07.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:07.647+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:07.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:07.781+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:07.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:07.781+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:07.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:07.781+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:07.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:07.782+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:07.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:07.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:08.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:07.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:08.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:07.984+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:08.996 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:05:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:05:08.558+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50729 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:05:08.996 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:08.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:08.997 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:08.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:08.997 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:08.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:08.997 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:08.601+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:08.997 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:08.758+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:08.997 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:08.758+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:08.997 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:08.758+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:08.998 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:08.759+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:09.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:08.994+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:09.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:08.994+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:09.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:08.994+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:08.995+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:09.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:09.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:09.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:09.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:09.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:09.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:09.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:09.625+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:09.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:09.746+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:09.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:09.746+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:09.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:09.746+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:09.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:09.746+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:10.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:09.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:10.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:09.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:10.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:09.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:10.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:09.988+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:10.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:10.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:10.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:10.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:10.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:10.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:10.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:10.607+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:10.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:10.779+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:10.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:10.779+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:10.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:10.779+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:10.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:10.780+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:11.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:11.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:11.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:11.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:11.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:11.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:11.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:11.037+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:12.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:11.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:11.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:11.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:11.640+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:12.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:11.815+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:12.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:11.815+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:12.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:11.815+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:12.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:11.815+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:12.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:12.039+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:12.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:12.039+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:12.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:12.039+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:12.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:12.041+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:13.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:12.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:13.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:12.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:13.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:12.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:13.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:12.772+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:13.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:12.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:13.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:12.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:13.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:12.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:13.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:12.689+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:13.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:13.055+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:13.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:13.055+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:13.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:13.055+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:13.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:13.056+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:13.972 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:05:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:05:13.564+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50769 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:05:13.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:13.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:13.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:13.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:13.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:13.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:13.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:13.663+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:13.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:13.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:13.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:13.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:13.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:13.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:13.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:13.807+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:14.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:14.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:14.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:14.048+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:15.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:14.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:15.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:14.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:15.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:14.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:15.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:14.661+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:15.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:14.805+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:15.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:14.805+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:15.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:14.805+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:15.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:14.806+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:15.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:15.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:15.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:15.008+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:16.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:15.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:16.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:15.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:16.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:15.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:16.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:15.697+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:16.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:15.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:16.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:15.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:16.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:15.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:16.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:15.779+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:16.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:16.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:16.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:16.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:16.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:16.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:16.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:16.025+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:16.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:16.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:16.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:16.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:16.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:16.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:16.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:16.710+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:16.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:16.762+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:16.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:16.762+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:16.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:16.762+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:16.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:16.763+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:16.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:16.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:17.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:16.979+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:17.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:16.979+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:17.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:17.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:17.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:17.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:17.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:17.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:17.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:17.695+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:17.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:17.789+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:17.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:17.789+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:17.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:17.789+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:17.991 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:17.790+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:17.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:17.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:17.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:17.988+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:18.975 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:05:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:05:18.571+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50809 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:05:18.976 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:18.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:18.976 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:18.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:18.976 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:18.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:18.976 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:18.809+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:18.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:18.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:18.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:18.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:18.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:18.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:18.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:18.665+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:19.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:18.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:19.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:18.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:19.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:18.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:19.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:18.974+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:19.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:19.666+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:19.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:19.666+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:19.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:19.666+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:19.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:19.666+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:19.960 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:19.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:19.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:19.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:19.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:19.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:19.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:19.855+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:20.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:19.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:20.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:19.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:20.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:19.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:20.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:19.959+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:20.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:20.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:20.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:20.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:20.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:20.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:20.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:20.706+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:20.962 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:20.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:20.962 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:20.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:20.962 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:20.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:20.962 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:20.880+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:21.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:20.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:21.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:20.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:21.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:20.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:20.959+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:21.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:21.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:21.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:21.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:21.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:21.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:21.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:21.718+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:21.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:21.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:21.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:21.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:21.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:21.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:21.990 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:21.889+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:21.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:21.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:21.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:22.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:21.988+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:22.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:22.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:22.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:22.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:22.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:22.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:22.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:22.728+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:22.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:22.849+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:22.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:22.849+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:22.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:22.849+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:22.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:22.849+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:22.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:22.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:23.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:22.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:23.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:22.984+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:23.898 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:05:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:05:23.577+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50849 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:05:23.899 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:23.755+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:23.899 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:23.755+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:23.899 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:23.755+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:23.899 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:23.755+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:24.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:23.990+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:24.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:23.990+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:24.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:23.990+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:24.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:23.991+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:23.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:23.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:23.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:23.897+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:25.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:24.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:25.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:24.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:25.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:24.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:25.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:24.963+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:24.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:24.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:24.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:25.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:24.760+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:25.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:24.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:25.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:24.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:25.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:24.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:25.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:24.911+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:26.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:26.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:26.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:26.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:26.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:26.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:26.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:26.008+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:26.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:25.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:26.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:25.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:26.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:25.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:26.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:25.778+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:26.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:25.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:26.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:25.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:26.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:25.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:26.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:25.883+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:26.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:26.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:26.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:26.815+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:27.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:27.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:27.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:27.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:27.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:27.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:27.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:27.039+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:27.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:26.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:27.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:26.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:27.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:26.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:27.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:26.928+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:27.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:27.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:28.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:27.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:28.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:27.949+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:28.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:28.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:28.016+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:28.017+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:28.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:27.810+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:28.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:27.810+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:28.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:27.810+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:28.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:27.810+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:28.836 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:05:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:05:28.584+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50889 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:05:28.837 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:28.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:28.837 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:28.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:28.837 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:28.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:28.837 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:28.834+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:28.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:28.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:29.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:28.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:29.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:28.972+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:29.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:28.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:29.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:28.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:29.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:28.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:29.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:28.977+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:30.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:30.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:30.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:30.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:30.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:30.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:30.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:30.023+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:30.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:29.924+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:30.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:29.924+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:30.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:29.924+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:30.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:29.925+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:30.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:29.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:30.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:29.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:30.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:29.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:30.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:29.873+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:31.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:30.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:31.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:30.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:30.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:30.999+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:30.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:30.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:31.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:30.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:31.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:30.958+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:31.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:30.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:31.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:30.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:31.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:30.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:31.273 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:30.873+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:31.929+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:31.929+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:31.929+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:32.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:31.931+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:31.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:31.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:31.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:31.917+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:32.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:31.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:32.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:31.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:32.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:31.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:32.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:31.955+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:32.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:32.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:32.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:33.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:32.946+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:33.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:32.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:33.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:32.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:33.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:32.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:33.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:32.989+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:33.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:32.953+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:33.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:32.953+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:33.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:32.953+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:33.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:32.954+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:33.917 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:05:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:05:33.590+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50929 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:05:34.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:33.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:34.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:33.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:33.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:33.916+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:34.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:33.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:34.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:33.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:34.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:33.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:34.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:33.998+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:33.994+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:33.994+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:33.994+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:34.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:33.995+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:35.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:34.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:35.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:34.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:35.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:34.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:35.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:34.997+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:34.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:34.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:34.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:34.966+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:34.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:35.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:34.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:35.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:34.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:35.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:34.976+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:35.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:35.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:35.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:35.968+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:36.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:35.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:36.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:35.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:36.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:35.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:36.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:35.961+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:36.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:36.039+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:36.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:36.039+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:36.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:36.039+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:36.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:36.040+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:36.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:36.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:36.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:36.941+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:37.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:37.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:37.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:37.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:37.081+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:37.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:37.009+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:37.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:37.009+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:37.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:37.009+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:37.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:37.010+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:38.108+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:38.108+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:38.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:38.108+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:38.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:38.109+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:37.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:37.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:37.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:37.908+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:38.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:38.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:38.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:38.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:38.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:38.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:38.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:38.036+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:38.872 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:05:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:05:38.596+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 50969 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:05:39.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:39.060+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:39.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:39.060+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:39.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:39.060+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:39.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:39.061+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:39.146 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:38.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:39.146 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:38.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:39.146 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:38.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:39.146 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:38.871+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:39.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:39.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:39.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:39.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:39.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:39.143+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:39.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:39.143+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:39.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:39.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:39.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:39.907+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:40.070+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:40.070+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:40.070+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:40.070+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:40.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:40.128+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:40.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:40.128+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:40.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:40.128+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:40.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:40.128+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:41.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:41.076+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:41.267 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:41.076+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:41.267 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:41.076+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:41.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:41.077+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:41.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:40.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:41.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:40.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:41.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:40.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:41.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:40.862+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:41.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:41.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:41.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:41.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:41.158+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:42.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:42.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:42.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:42.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:42.073+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:41.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:41.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:41.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:41.896+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:42.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:42.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:42.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:42.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:42.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:42.123+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:42.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:42.124+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:42.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:42.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:43.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:42.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:43.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:42.932+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:43.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:43.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:43.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:43.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:43.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:43.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:43.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:43.036+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:43.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:43.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:43.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:43.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:43.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:43.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:43.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:43.162+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:43.938 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:05:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:05:43.603+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51010 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:05:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:43.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:43.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:43.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:44.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:43.937+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:44.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:44.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:44.159+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:44.159+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:44.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:43.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:44.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:43.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:44.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:43.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:44.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:43.999+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:45.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:44.895+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:45.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:44.895+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:45.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:44.895+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:45.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:44.896+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:45.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:45.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:45.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:45.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:45.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:45.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:45.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:45.034+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:45.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:45.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:45.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:45.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:45.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:45.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:45.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:45.205+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:46.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:46.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:46.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:46.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:46.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:46.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:46.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:46.029+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:46.212 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:45.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:46.212 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:45.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:46.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:45.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:46.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:45.861+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:46.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:46.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:46.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:46.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:46.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:46.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:46.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:46.211+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:47.214 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:47.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:47.214 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:47.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:47.214 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:47.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:47.214 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:47.003+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:47.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:46.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:47.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:46.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:47.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:46.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:47.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:46.860+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:47.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:47.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:47.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:47.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:47.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:47.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:47.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:47.212+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:48.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:47.997+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:48.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:47.997+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:48.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:47.997+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:48.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:47.998+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:48.259 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:47.864+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:48.259 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:47.864+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:48.259 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:47.864+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:48.259 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:47.865+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:48.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:48.256+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:48.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:48.256+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:48.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:48.256+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:48.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:48.257+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:48.885 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:05:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:05:48.609+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51050 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:05:49.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:48.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:49.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:48.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:48.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:49.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:48.884+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:49.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:49.003+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:49.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:49.003+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:49.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:49.003+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:49.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:49.004+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:49.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:49.306+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:49.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:49.306+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:49.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:49.306+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:49.307+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:49.968+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:49.968+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:49.968+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:50.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:49.969+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:50.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:49.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:50.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:49.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:50.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:49.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:50.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:49.882+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:50.328+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:50.328+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:50.328+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:50.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:50.329+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:51.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:50.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:50.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:50.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:50.885+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:51.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:50.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:51.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:50.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:51.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:50.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:51.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:50.940+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:51.283+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:51.283+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:51.283+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:51.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:51.283+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:52.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:51.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:52.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:51.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:52.115 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:51.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:52.115 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:51.907+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:52.115 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:51.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:52.115 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:51.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:52.115 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:51.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:52.115 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:51.842+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:52.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:52.323+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:52.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:52.323+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:52.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:52.323+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:52.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:52.324+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:53.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:52.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:53.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:52.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:53.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:52.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:53.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:52.869+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:53.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:52.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:53.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:52.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:53.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:52.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:53.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:52.800+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:53.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:53.277+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:53.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:53.277+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:53.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:53.277+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:53.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:53.278+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:53.836 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:05:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:05:53.615+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51090 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:05:54.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:53.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:54.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:53.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:54.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:53.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:54.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:53.917+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:54.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:53.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:54.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:53.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:54.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:53.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:54.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:53.835+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:54.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:54.320+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:54.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:54.320+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:54.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:54.320+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:54.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:54.320+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:55.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:54.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:55.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:54.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:55.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:54.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:55.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:54.964+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:54.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:54.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:54.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:54.830+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:55.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:55.329+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:55.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:55.329+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:55.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:55.329+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:55.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:55.329+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:55.976+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:55.976+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:55.976+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:56.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:55.977+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:55.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:55.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:55.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:55.813+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:56.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:56.313+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:56.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:56.313+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:56.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:56.313+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:56.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:56.313+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:57.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:56.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:57.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:56.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:57.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:56.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:57.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:56.946+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:57.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:56.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:57.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:56.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:57.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:56.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:57.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:56.815+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:57.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:57.339+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:57.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:57.339+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:57.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:57.339+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:57.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:57.339+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:58.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:57.947+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:58.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:57.947+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:58.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:57.947+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:58.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:57.948+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:58.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:57.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:58.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:57.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:58.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:57.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:58.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:57.817+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:58.741 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:05:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:05:58.621+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51130 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:05:58.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:58.388+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:58.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:58.388+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:58.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:58.388+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:58.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:58.389+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:58.920+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:59.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:58.920+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:59.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:58.920+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:05:59.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:58.921+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:05:59.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:58.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:59.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:58.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:59.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:58.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:05:59.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:58.810+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:05:59.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:59.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:59.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:59.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:59.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:59.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:05:59.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:05:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:05:59.432+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:00.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:59.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:00.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:59.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:00.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:59.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:00.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:05:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:05:59.930+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:00.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:59.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:00.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:59.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:00.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:59.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:00.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:05:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:05:59.768+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:00.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:00.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:00.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:00.475+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:00.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:00.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:01.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:00.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:01.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:00.917+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:00.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:00.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:00.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:00.786+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:01.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:01.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:01.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:01.479+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:02.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:01.965+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:02.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:01.965+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:02.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:01.965+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:02.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:01.967+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:02.224 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:01.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:02.224 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:01.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:02.225 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:01.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:02.225 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:01.792+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:02.751 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:02.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:02.752 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:02.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:02.752 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:02.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:02.752 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:02.475+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:03.002 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:02.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:03.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:02.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:03.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:02.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:03.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:02.750+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:03.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:03.000+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:03.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:03.000+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:03.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:03.000+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:03.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:03.001+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:03.740 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:06:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:06:03.627+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51170 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:06:03.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:03.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:03.741 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:03.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:03.741 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:03.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:03.741 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:03.445+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:03.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:03.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:03.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:04.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:03.739+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:04.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:03.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:04.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:03.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:04.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:03.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:04.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:03.963+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:04.746 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:04.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:04.746 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:04.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:04.746 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:04.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:04.746 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:04.440+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:05.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:04.981+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:05.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:04.981+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:05.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:04.981+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:05.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:04.982+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:05.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:04.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:05.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:04.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:05.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:04.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:05.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:04.745+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:05.706 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:05.410+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:05.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:05.410+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:05.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:05.410+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:05.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:05.411+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:06.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:05.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:06.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:05.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:06.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:05.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:06.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:05.705+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:06.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:06.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:06.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:06.012+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:06.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:06.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:06.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:06.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:06.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:06.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:06.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:06.377+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:07.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:06.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:07.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:06.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:07.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:06.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:07.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:06.734+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:07.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:07.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:07.001+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:07.002+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:07.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:07.331+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:07.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:07.331+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:07.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:07.331+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:07.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:07.332+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:07.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:07.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:07.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:07.754+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:08.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:08.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:08.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:08.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:08.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:08.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:08.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:08.049+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:08.302 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:08.300+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:08.303 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:08.300+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:08.303 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:08.300+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:08.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:08.300+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:09.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:06:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:06:08.635+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51210 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:06:09.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:08.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:09.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:08.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:09.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:08.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:09.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:08.781+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:09.318 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:09.021+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:09.319 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:09.021+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:09.319 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:09.021+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:09.319 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:09.022+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:09.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:09.317+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:09.317+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:09.317+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:09.317+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:09.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:09.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:09.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:09.819+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:10.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:09.979+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:10.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:09.979+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:10.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:09.979+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:10.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:09.980+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:10.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:10.294+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:10.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:10.294+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:10.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:10.294+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:10.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:10.294+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:11.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:10.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:11.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:10.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:11.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:10.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:11.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:10.960+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:11.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:10.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:11.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:10.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:11.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:10.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:11.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:10.848+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:11.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:11.308+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:11.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:11.308+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:11.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:11.308+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:11.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:11.308+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:11.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:11.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:11.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:12.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:11.897+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:12.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:12.000+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:12.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:12.000+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:12.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:12.000+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:12.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:12.000+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:12.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:12.357+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:12.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:12.357+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:12.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:12.357+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:12.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:12.357+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:12.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:12.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:12.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:12.998+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:13.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:12.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:13.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:12.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:13.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:12.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:13.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:12.907+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:13.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:13.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:13.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:13.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:13.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:13.376+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:13.644 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:13.376+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:13.925 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:06:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:06:13.641+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51250 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:06:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:14.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:14.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:14.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:14.012+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:13.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:13.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:13.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:14.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:13.924+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:14.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:14.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:14.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:14.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:14.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:14.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:14.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:14.381+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:14.890+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:14.890+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:14.890+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:14.891+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:15.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:14.965+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:15.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:14.965+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:15.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:14.965+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:15.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:14.966+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:15.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:15.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:15.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:15.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:15.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:15.419+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:16.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:16.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:16.002+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:16.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:16.003+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:15.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:15.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:15.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:15.903+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:16.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:16.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:16.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:16.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:16.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:16.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:16.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:16.459+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:16.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:16.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:16.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:16.993+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:17.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:16.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:17.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:16.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:17.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:16.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:17.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:16.916+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:17.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:17.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:17.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:17.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:17.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:17.436+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:17.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:17.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:18.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:17.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:18.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:17.978+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:17.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:17.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:17.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:17.902+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:18.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:18.388+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:18.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:18.388+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:18.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:18.388+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:18.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:18.388+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:18.977 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:06:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:06:18.647+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51291 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:06:18.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:18.890+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:18.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:18.890+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:18.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:18.890+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:18.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:18.891+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:19.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:18.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:19.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:18.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:19.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:18.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:19.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:18.975+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:19.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:19.342+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:19.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:19.342+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:19.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:19.342+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:19.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:19.343+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:20.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:19.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:20.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:19.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:20.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:19.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:20.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:19.933+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:20.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:19.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:20.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:19.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:20.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:19.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:20.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:19.860+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:20.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:20.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:20.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:20.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:20.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:20.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:20.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:20.373+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:21.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:20.924+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:21.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:20.924+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:21.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:20.924+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:20.925+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:20.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:20.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:20.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:21.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:20.886+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:21.393+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:21.393+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:21.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:21.393+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:21.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:21.393+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:21.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:21.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:22.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:21.963+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:22.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:21.964+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:22.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:21.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:22.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:21.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:22.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:21.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:22.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:21.873+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:22.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:22.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:22.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:22.441+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:22.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:22.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:22.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:23.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:22.959+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:22.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:22.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:22.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:22.884+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:23.674 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:06:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:06:23.654+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51332 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:06:23.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:23.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:23.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:23.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:23.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:23.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:23.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:23.412+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:23.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:23.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:23.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:23.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:23.847+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:24.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:23.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:24.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:23.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:24.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:23.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:24.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:23.965+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:24.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:24.379+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:24.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:24.379+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:24.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:24.379+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:24.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:24.380+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:25.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:24.953+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:25.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:24.953+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:25.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:24.953+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:25.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:24.954+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:24.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:24.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:24.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:24.835+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:25.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:25.394+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:25.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:25.394+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:25.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:25.394+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:25.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:25.394+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:25.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:25.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:25.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:26.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:25.872+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:26.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:25.968+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:26.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:25.968+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:26.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:25.968+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:26.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:25.969+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:26.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:26.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:26.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:26.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:26.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:26.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:26.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:26.402+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:26.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:26.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:26.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:27.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:26.931+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:26.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:26.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:26.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:26.826+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:27.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:27.398+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:27.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:27.398+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:27.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:27.398+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:27.398+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:27.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:27.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:27.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:28.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:27.804+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:27.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:27.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:27.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:27.926+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:28.662 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:28.406+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:28.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:28.406+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:28.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:28.406+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:28.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:28.406+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:29.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:06:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:06:28.660+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51372 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:06:29.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:28.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:29.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:28.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:29.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:28.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:29.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:28.767+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:29.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:28.894+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:29.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:28.894+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:29.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:28.894+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:29.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:28.895+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:29.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:29.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:29.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:29.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:29.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:29.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:29.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:29.442+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:29.773+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:29.773+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:29.773+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:29.774+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:30.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:29.905+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:30.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:29.905+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:30.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:29.905+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:30.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:29.906+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:30.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:30.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:30.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:30.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:30.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:30.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:30.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:30.485+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:30.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:30.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:30.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:30.951+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:31.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:30.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:31.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:30.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:31.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:30.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:31.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:30.810+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:31.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:31.483+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:31.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:31.483+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:31.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:31.483+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:31.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:31.484+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:31.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:31.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:31.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:32.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:31.970+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:31.790+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:31.790+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:31.790+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:31.791+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:32.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:32.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:32.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:32.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:32.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:32.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:32.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:32.459+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:32.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:32.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:32.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:33.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:32.822+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:32.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:32.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:32.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:32.951+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:33.689 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:06:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:06:33.667+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51412 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:06:33.690 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:33.430+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:33.690 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:33.430+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:33.690 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:33.430+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:33.690 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:33.430+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:34.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:33.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:34.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:33.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:34.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:33.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:34.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:33.923+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:34.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:33.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:34.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:33.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:34.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:33.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:34.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:33.842+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:34.406+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:34.406+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:34.406+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:34.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:34.407+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:34.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:34.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:34.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:34.818+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:34.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:34.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:34.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:34.940+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:35.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:35.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:35.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:35.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:35.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:35.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:35.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:35.437+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:35.949+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:35.949+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:35.949+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:36.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:35.950+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:35.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:35.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:35.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:35.778+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:36.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:36.403+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:36.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:36.403+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:36.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:36.403+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:36.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:36.403+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:36.808+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:36.808+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:36.808+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:36.809+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:37.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:36.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:37.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:36.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:37.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:36.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:37.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:36.911+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:37.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:37.369+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:37.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:37.369+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:37.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:37.369+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:37.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:37.369+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:37.771+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:37.771+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:37.771+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:37.772+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:37.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:37.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:37.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:37.870+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:38.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:38.392+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:38.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:38.392+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:38.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:38.392+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:38.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:38.393+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:39.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:06:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:06:38.673+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51452 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:06:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:38.891+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:39.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:38.891+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:39.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:38.891+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:39.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:38.891+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:39.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:38.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:39.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:38.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:39.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:38.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:39.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:38.798+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:39.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:39.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:39.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:39.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:39.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:39.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:39.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:39.408+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:40.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:39.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:40.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:39.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:39.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:39.859+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:39.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:39.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:39.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:39.815+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:40.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:40.385+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:40.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:40.385+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:40.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:40.385+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:40.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:40.386+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:40.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:40.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:40.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:41.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:40.876+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:41.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:40.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:41.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:40.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:41.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:40.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:41.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:40.843+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:41.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:41.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:41.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:41.427+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:41.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:41.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:41.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:41.799+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:42.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:41.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:42.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:41.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:42.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:41.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:42.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:41.879+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:42.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:42.399+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:42.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:42.399+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:42.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:42.399+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:42.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:42.400+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:42.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:42.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:42.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:43.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:42.818+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:43.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:42.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:43.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:42.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:43.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:42.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:43.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:42.848+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:43.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:43.421+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:43.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:43.421+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:43.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:43.421+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:43.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:43.422+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:44.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:06:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:06:43.679+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51492 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:06:44.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:43.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:44.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:43.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:44.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:43.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:44.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:43.811+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:44.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:43.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:44.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:43.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:44.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:43.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:44.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:43.839+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:44.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:44.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:44.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:44.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:44.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:44.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:44.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:44.460+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:44.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:44.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:44.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:45.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:44.853+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:44.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:44.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:44.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:44.843+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:45.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:45.426+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:45.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:45.426+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:45.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:45.426+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:45.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:45.426+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:45.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:45.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:45.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:46.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:45.866+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:45.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:45.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:45.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:45.897+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:46.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:46.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:46.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:46.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:46.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:46.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:46.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:46.421+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:47.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:46.891+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:47.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:46.891+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:47.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:46.891+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:47.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:46.892+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:47.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:46.873+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:47.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:46.873+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:47.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:46.873+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:47.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:46.874+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:47.691 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:47.396+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:47.691 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:47.396+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:47.691 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:47.396+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:47.691 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:47.396+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:47.859+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:47.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:48.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:47.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:48.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:47.860+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:47.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:47.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:47.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:47.825+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:48.693 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:06:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:06:48.686+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51532 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:06:48.693 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:48.437+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:48.693 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:48.437+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:48.693 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:48.437+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:48.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:48.437+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:48.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:48.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:48.830+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:48.830+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:49.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:48.815+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:49.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:48.815+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:49.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:48.815+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:49.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:48.816+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:49.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:49.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:49.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:49.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:49.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:49.436+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:49.784+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:49.784+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:49.784+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:50.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:49.785+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:50.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:49.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:50.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:49.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:50.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:49.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:50.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:49.862+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:50.405+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:50.405+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:50.405+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:50.406+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:50.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:50.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:50.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:51.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:50.775+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:50.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:50.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:50.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:50.908+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:51.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:51.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:51.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:51.420+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:51.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:51.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:51.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:51.805+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:51.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:51.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:51.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:51.907+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:52.741 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:52.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:52.741 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:52.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:52.741 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:52.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:52.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:52.413+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:53.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:52.772+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:53.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:52.772+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:53.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:52.772+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:53.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:52.773+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:53.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:52.867+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:53.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:52.867+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:53.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:52.867+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:53.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:52.868+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:53.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:53.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:53.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:53.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:53.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:53.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:53.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:53.404+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:54.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:06:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:06:53.693+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51572 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:06:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:53.890+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:54.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:53.890+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:54.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:53.890+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:54.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:53.891+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:54.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:53.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:54.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:53.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:54.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:53.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:54.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:53.800+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:54.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:54.381+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:54.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:54.381+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:54.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:54.381+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:54.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:54.381+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:55.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:54.818+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:55.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:54.818+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:55.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:54.818+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:55.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:54.819+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:54.926+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:54.926+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:54.926+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:54.927+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:55.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:55.374+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:55.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:55.374+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:55.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:55.374+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:55.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:55.375+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:55.827+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:55.827+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:55.827+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:56.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:55.828+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:55.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:55.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:55.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:55.884+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:56.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:56.343+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:56.773 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:56.343+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:56.773 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:56.343+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:56.773 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:56.343+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:56.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:56.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:57.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:56.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:57.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:56.866+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:57.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:56.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:57.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:56.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:57.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:56.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:57.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:56.908+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:57.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:57.300+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:57.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:57.300+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:57.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:57.300+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:57.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:57.300+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:57.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:57.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:57.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:58.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:57.898+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:58.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:57.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:58.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:57.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:58.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:57.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:58.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:57.853+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:58.701 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:58.332+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:58.701 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:58.332+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:58.701 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:58.332+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:58.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:58.332+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:59.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:06:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:06:58.699+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51612 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:06:59.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:58.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:59.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:58.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:59.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:58.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:06:59.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:58.835+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:06:59.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:58.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:59.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:58.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:59.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:58.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:06:59.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:58.871+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:06:59.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:59.316+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:59.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:59.316+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:59.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:59.316+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:06:59.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:06:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:06:59.316+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:00.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:59.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:00.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:59.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:00.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:59.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:00.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:06:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:06:59.847+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:00.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:59.836+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:00.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:59.836+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:00.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:59.836+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:00.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:06:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:06:59.836+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:00.299+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:00.299+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:00.299+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:00.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:00.299+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:00.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:00.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:00.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:01.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:00.880+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:00.868+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:00.868+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:00.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:01.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:00.869+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:01.303+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:01.303+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:01.303+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:01.304+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:01.867+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:01.867+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:01.867+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:02.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:01.868+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:01.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:01.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:01.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:01.866+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:02.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:02.321+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:02.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:02.321+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:02.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:02.321+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:02.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:02.321+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:02.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:02.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:02.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:02.906+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:03.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:02.886+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:03.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:02.886+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:03.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:02.886+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:03.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:02.886+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:03.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:03.338+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:03.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:03.338+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:03.708 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:03.338+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:03.708 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:03.338+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:04.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:07:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:07:03.705+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51652 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:07:04.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:03.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:04.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:03.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:04.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:03.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:04.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:03.875+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:04.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:03.913+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:04.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:03.913+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:04.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:03.913+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:04.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:03.913+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:04.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:04.339+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:04.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:04.339+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:04.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:04.339+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:04.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:04.339+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:04.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:04.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:04.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:04.904+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:04.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:04.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:04.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:04.870+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:05.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:05.354+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:05.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:05.354+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:05.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:05.354+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:05.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:05.354+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:05.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:05.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:05.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:05.883+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:06.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:05.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:06.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:05.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:06.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:05.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:06.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:05.912+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:06.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:06.342+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:06.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:06.342+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:06.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:06.342+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:06.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:06.342+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:06.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:06.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:07.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:06.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:06.937+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:06.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:06.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:06.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:06.841+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:07.360+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:07.360+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:07.360+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:07.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:07.361+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:07.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:07.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:07.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:08.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:07.947+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:07.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:07.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:07.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:07.813+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:08.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:08.343+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:08.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:08.343+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:08.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:08.343+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:08.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:08.343+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:09.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:07:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:07:08.712+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51692 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:07:09.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:08.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:09.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:08.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:09.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:08.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:09.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:08.904+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:09.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:08.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:09.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:08.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:09.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:08.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:09.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:08.764+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:09.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:09.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:09.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:09.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:09.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:09.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:09.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:09.296+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:09.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:09.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:09.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:09.933+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:09.724+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:09.724+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:09.724+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:09.725+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:10.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:10.261+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:10.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:10.261+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:10.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:10.261+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:10.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:10.262+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:10.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:10.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:10.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:10.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:10.702+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:11.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:10.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:11.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:10.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:11.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:10.964+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:11.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:10.965+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:11.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:11.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:11.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:11.216+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:11.994 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:11.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:11.994 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:11.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:11.994 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:11.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:11.994 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:11.734+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:12.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:11.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:12.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:11.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:12.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:11.992+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:12.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:11.993+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:12.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:12.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:12.170+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:12.170+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:12.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:12.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:12.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:12.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:12.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:12.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:12.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:12.685+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:13.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:13.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:13.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:13.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:13.157+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:13.042+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:13.042+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:13.042+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:13.043+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:13.838 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:07:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:07:13.718+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51732 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:07:13.839 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:13.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:13.839 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:13.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:13.839 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:13.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:13.839 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:13.707+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:14.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:14.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:14.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:14.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:14.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:14.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:14.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:14.040+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:14.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:14.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:14.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:14.142+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:15.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:14.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:15.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:14.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:15.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:14.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:15.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:14.672+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:15.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:15.087+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:15.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:15.087+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:15.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:15.087+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:15.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:15.088+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:15.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:15.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:15.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:15.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:15.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:15.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:15.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:15.136+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:16.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:15.683+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:16.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:15.683+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:16.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:15.683+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:16.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:15.684+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:16.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:16.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:16.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:16.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:16.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:16.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:16.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:16.119+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:16.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:16.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:16.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:16.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:16.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:16.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:16.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:16.161+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:17.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:16.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:17.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:16.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:17.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:16.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:17.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:16.719+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:17.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:17.085+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:17.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:17.085+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:17.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:17.085+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:17.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:17.086+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:17.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:17.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:17.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:17.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:17.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:17.190+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:17.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:17.190+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:18.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:17.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:18.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:17.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:18.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:17.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:18.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:17.759+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:18.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:18.070+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:18.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:18.070+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:18.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:18.070+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:18.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:18.071+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:18.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:18.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:18.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:18.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:18.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:18.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:18.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:18.174+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:19.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:07:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:07:18.725+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51772 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:07:19.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:18.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:19.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:18.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:19.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:18.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:19.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:18.765+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:19.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:19.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:19.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:19.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:19.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:19.073+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:19.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:19.074+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:19.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:19.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:19.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:19.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:19.209+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:20.072 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:19.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:20.072 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:19.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:20.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:19.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:20.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:19.792+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:20.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:20.070+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:20.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:20.070+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:20.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:20.070+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:20.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:20.071+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:20.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:20.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:20.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:20.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:20.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:20.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:20.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:20.180+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:21.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:20.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:21.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:20.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:21.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:20.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:21.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:20.808+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:21.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:21.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:21.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:21.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:21.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:21.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:21.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:21.156+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:21.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:21.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:21.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:21.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:21.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:21.078+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:21.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:21.079+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:22.095 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:21.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:22.095 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:21.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:22.095 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:21.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:22.095 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:21.835+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:22.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:22.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:22.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:22.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:22.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:22.093+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:22.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:22.094+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:22.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:22.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:22.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:22.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:22.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:22.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:22.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:22.185+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:23.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:22.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:23.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:22.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:23.143 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:22.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:23.143 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:22.844+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:23.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:23.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:23.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:23.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:23.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:23.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:23.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:23.181+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:23.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:23.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:23.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:23.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:23.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:23.140+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:23.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:23.142+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:23.854 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:07:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:07:23.732+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51811 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:07:24.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:23.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:24.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:23.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:24.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:23.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:24.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:23.863+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:24.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:24.186+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:24.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:24.186+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:24.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:24.186+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:24.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:24.187+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:24.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:24.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:24.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:24.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:24.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:24.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:24.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:24.185+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:25.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:24.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:25.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:24.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:25.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:24.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:25.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:24.872+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:25.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:25.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:25.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:25.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:25.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:25.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:25.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:25.138+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:25.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:25.171+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:25.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:25.171+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:25.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:25.171+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:25.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:25.172+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:26.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:25.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:26.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:25.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:26.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:25.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:26.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:25.916+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:26.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:26.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:26.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:26.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:26.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:26.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:26.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:26.150+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:26.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:26.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:26.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:26.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:26.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:26.181+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:26.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:26.182+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:27.141+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:27.141+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:27.141+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:27.142+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:27.164+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:27.164+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:27.164+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:27.165+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:27.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:26.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:27.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:26.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:27.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:26.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:27.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:26.916+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:28.150 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:27.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:28.150 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:27.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:28.150 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:27.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:28.150 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:27.876+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:28.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:28.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:28.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:28.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:28.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:28.151+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:28.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:28.151+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:28.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:28.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:28.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:28.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:28.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:28.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:28.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:28.149+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:29.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:07:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:07:28.738+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51852 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:07:29.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:28.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:29.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:28.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:29.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:28.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:29.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:28.926+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:29.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:29.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:29.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:29.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:29.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:29.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:29.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:29.158+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:29.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:29.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:29.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:29.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:29.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:29.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:29.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:29.184+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:30.225 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:30.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:30.225 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:30.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:30.225 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:30.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:30.225 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:30.174+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:30.225 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:29.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:30.225 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:29.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:30.226 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:29.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:30.226 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:29.970+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:30.223+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:30.223+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:30.223+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:30.223+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:31.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:31.177+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:31.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:31.177+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:31.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:31.177+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:31.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:30.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:31.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:30.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:31.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:30.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:31.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:30.923+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:31.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:31.178+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:31.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:31.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:31.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:31.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:31.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:31.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:31.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:31.201+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:32.194 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:31.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:32.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:31.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:32.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:31.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:32.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:31.919+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:32.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:32.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:32.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:32.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:32.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:32.152+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:32.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:32.152+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:32.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:32.192+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:32.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:32.192+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:32.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:32.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:32.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:32.193+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:33.150 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:32.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:33.150 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:32.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:33.150 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:32.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:33.151 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:32.897+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:33.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:33.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:33.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:33.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:33.149+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:33.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:33.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:33.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:33.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:33.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:33.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:33.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:33.202+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:33.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:07:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:07:33.745+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51891 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:07:34.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:33.891+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:34.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:33.891+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:34.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:33.891+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:34.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:33.892+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:34.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:34.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:34.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:34.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:34.119+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:34.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:34.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:34.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:34.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:34.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:34.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:34.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:34.163+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:35.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:34.930+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:35.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:34.930+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:35.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:34.930+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:35.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:34.931+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:35.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:35.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:35.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:35.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:35.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:35.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:35.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:35.074+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:35.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:35.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:35.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:35.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:35.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:35.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:35.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:35.200+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:36.170 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:35.946+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:36.170 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:35.946+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:36.170 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:35.946+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:36.170 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:35.947+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:36.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:36.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:36.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:36.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:36.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:36.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:36.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:36.035+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:36.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:36.168+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:36.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:36.168+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:36.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:36.168+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:36.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:36.169+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:37.212+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:37.212+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:37.212+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:37.214+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:36.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:36.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:36.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:36.983+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:37.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:37.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:37.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:37.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:37.059+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:38.187+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:38.187+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:38.187+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:38.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:38.188+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:37.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:37.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:37.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:38.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:37.978+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:38.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:38.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:38.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:38.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:38.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:38.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:38.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:38.027+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:39.012 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:07:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:07:38.751+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51932 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:07:39.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:38.929+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:39.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:38.929+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:39.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:38.929+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:39.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:38.930+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:39.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:39.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:39.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:39.208+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:39.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:39.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:39.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:39.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:39.011+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:40.244 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:39.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:40.244 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:39.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:40.244 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:39.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:40.245 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:39.909+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:40.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:40.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:40.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:40.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:40.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:40.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:40.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:40.014+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:40.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:40.242+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:40.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:40.242+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:40.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:40.242+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:40.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:40.243+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:41.228 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:40.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:41.228 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:40.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:41.228 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:40.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:41.228 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:40.882+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:41.228 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:41.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:41.229 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:41.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:41.229 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:41.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:41.229 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:41.026+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:41.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:41.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:41.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:41.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:41.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:41.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:41.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:41.227+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:41.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:41.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:41.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:41.887+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:42.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:42.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:42.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:42.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:42.055+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:42.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:42.269+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:42.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:42.269+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:42.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:42.269+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:42.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:42.270+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:43.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:42.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:43.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:42.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:43.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:42.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:43.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:42.866+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:43.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:43.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:43.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:43.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:43.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:43.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:43.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:43.028+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:43.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:43.307+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:43.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:43.307+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:43.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:43.307+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:43.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:43.308+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:43.877 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:07:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:07:43.758+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 51971 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:07:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:43.913+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:43.913+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:43.913+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:43.914+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:44.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:44.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:44.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:44.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:44.020+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:44.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:44.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:44.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:44.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:44.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:44.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:44.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:44.341+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:44.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:44.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:44.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:44.873+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:45.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:45.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:45.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:45.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:45.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:45.010+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:45.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:45.010+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:45.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:45.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:45.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:45.313+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:45.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:45.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:45.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:46.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:45.847+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:46.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:46.006+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:46.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:46.006+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:46.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:46.006+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:46.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:46.006+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:46.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:46.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:46.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:46.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:46.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:46.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:46.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:46.283+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:47.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:46.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:47.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:46.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:47.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:46.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:47.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:46.813+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:47.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:47.039+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:47.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:47.039+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:47.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:47.039+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:47.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:47.040+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:47.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:47.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:47.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:47.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:47.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:47.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:47.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:47.281+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:48.051 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:47.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:48.052 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:47.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:48.052 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:47.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:48.052 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:47.798+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:48.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:48.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:48.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:48.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:48.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:48.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:48.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:48.258+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:48.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:48.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:48.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:48.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:48.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:48.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:48.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:48.050+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:49.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:07:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:07:48.764+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52012 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:07:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:48.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:48.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:49.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:48.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:49.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:48.806+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:49.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:49.248+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:49.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:49.248+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:49.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:49.248+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:49.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:49.249+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:49.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:49.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:49.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:49.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:49.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:49.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:49.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:49.046+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:50.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:49.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:50.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:49.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:50.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:49.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:50.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:49.824+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:50.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:50.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:50.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:50.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:50.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:50.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:50.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:50.020+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:50.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:50.266+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:50.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:50.266+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:50.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:50.266+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:50.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:50.267+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:51.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:50.815+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:51.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:50.815+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:51.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:50.815+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:51.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:50.816+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:51.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:51.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:51.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:51.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:51.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:51.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:51.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:51.004+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:51.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:51.315+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:51.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:51.315+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:51.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:51.315+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:51.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:51.316+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:52.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:52.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:52.018+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:52.018+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:51.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:51.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:51.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:51.850+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:52.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:52.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:52.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:52.350+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:52.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:52.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:52.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:52.878+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:53.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:53.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:53.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:53.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:53.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:53.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:53.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:53.060+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:53.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:53.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:53.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:53.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:53.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:53.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:53.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:53.310+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:53.867 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:07:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:07:53.771+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52051 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:07:54.228 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:53.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:54.228 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:53.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:54.228 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:53.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:54.228 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:53.866+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:54.228 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:54.098+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:54.229 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:54.098+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:54.229 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:54.098+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:54.229 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:54.098+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:54.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:54.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:54.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:54.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:54.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:54.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:54.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:54.357+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:54.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:54.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:54.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:54.833+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:55.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:55.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:55.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:55.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:55.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:55.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:55.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:55.052+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:55.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:55.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:55.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:55.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:55.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:55.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:55.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:55.365+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:56.057 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:55.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:56.057 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:55.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:56.057 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:55.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:56.057 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:55.797+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:56.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:56.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:56.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:56.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:56.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:56.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:56.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:56.055+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:56.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:56.413+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:56.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:56.413+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:56.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:56.413+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:56.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:56.413+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:57.059 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:56.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:57.060 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:56.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:57.060 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:56.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:57.060 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:56.798+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:57.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:57.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:57.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:57.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:57.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:57.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:57.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:57.058+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:57.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:57.457+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:57.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:57.457+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:57.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:57.457+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:57.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:57.458+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:57.836+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:57.836+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:57.836+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:58.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:57.837+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:58.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:58.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:58.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:58.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:58.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:58.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:58.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:58.028+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:58.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:58.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:58.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:58.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:58.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:58.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:58.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:58.504+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:59.049 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:07:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:07:58.777+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52092 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:07:59.049 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:58.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:59.049 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:58.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:59.049 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:58.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:07:59.050 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:58.826+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:07:59.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:59.047+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:59.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:59.047+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:59.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:59.047+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:07:59.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:07:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:07:59.048+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:07:59.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:59.456+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:59.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:59.456+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:59.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:59.456+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:07:59.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:07:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:07:59.457+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:59.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:59.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:59.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:07:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:07:59.875+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:00.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:00.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:00.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:00.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:00.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:00.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:00.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:00.017+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:00.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:00.476+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:00.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:00.476+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:00.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:00.476+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:00.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:00.477+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:00.899+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:00.899+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:00.899+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:00.900+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:01.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:01.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:01.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:01.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:01.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:01.014+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:01.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:01.015+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:01.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:01.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:01.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:01.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:01.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:01.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:01.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:01.505+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:02.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:02.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:02.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:02.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:02.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:02.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:02.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:02.044+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:01.913+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:01.913+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:01.913+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:02.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:01.914+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:02.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:02.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:02.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:02.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:02.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:02.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:02.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:02.463+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:02.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:02.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:02.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:02.867+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:03.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:03.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:03.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:03.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:03.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:03.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:03.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:03.060+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:03.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:03.459+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:03.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:03.459+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:03.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:03.459+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:03.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:03.461+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:04.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:08:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:08:03.784+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52131 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:08:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:03.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:03.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:03.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:04.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:03.830+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:04.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:04.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:04.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:04.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:04.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:04.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:04.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:04.046+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:04.449+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:04.449+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:04.449+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:04.450+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:04.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:04.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:04.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:04.847+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:05.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:05.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:05.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:05.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:05.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:05.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:05.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:05.018+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:05.469+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:05.469+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:05.469+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:05.469+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:05.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:05.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:05.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:06.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:05.800+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:06.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:06.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:06.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:06.003+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:06.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:06.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:06.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:06.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:06.493+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:07.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:07.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:07.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:07.042+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:06.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:06.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:06.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:06.793+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:07.826 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:07.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:07.826 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:07.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:07.827 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:07.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:07.827 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:07.543+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:07.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:07.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:07.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:07.825+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:08.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:08.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:08.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:08.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:08.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:08.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:08.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:08.061+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:08.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:08.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:08.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:08.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:08.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:08.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:08.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:08.579+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:09.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:08:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:08:08.790+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52172 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:08:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:08.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:08.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:09.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:08.824+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:09.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:08.825+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:09.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:09.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:09.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:09.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:09.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:09.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:09.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:09.032+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:09.837 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:09.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:09.837 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:09.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:09.837 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:09.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:09.838 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:09.579+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:10.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:10.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:10.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:10.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:10.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:10.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:10.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:10.055+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:09.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:09.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:09.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:09.836+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:10.601+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:10.601+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:11.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:10.601+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:11.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:10.601+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:10.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:10.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:10.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:10.847+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:11.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:11.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:11.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:11.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:11.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:11.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:11.079+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:11.887 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:11.574+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:11.887 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:11.574+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:11.888 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:11.574+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:11.888 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:11.575+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:11.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:11.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:11.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:11.886+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:12.093+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:12.093+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:12.093+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:12.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:12.094+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:12.890 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:12.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:12.890 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:12.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:12.890 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:12.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:12.891 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:12.577+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:12.888+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:12.888+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:12.888+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:12.889+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:13.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:13.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:13.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:13.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:13.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:13.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:13.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:13.056+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:13.818 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:08:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:08:13.797+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52211 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:08:13.818 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:13.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:13.818 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:13.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:13.819 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:13.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:13.819 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:13.619+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:14.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:13.932+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:14.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:13.932+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:14.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:13.932+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:14.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:13.933+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:14.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:14.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:14.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:14.078+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:14.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:14.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:14.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:14.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:14.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:14.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:14.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:14.632+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:14.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:14.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:14.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:14.894+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:15.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:15.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:15.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:15.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:15.104+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:15.890 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:15.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:15.891 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:15.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:15.891 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:15.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:15.891 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:15.630+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:15.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:15.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:15.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:15.889+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:16.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:16.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:16.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:16.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:16.079+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:16.855 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:16.595+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:16.856 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:16.595+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:16.856 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:16.595+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:16.856 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:16.597+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:17.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:17.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:17.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:17.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:17.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:17.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:17.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:17.046+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:17.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:16.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:17.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:16.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:17.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:16.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:17.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:16.854+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:17.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:17.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:17.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:17.625+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:18.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:17.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:18.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:17.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:18.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:17.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:18.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:17.864+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:18.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:18.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:18.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:18.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:18.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:18.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:18.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:18.074+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:18.925 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:08:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:08:18.804+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52252 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:08:18.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:18.637+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:18.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:18.637+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:18.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:18.637+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:18.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:18.638+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:18.926 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:18.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:18.926 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:18.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:18.926 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:18.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:18.926 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:18.836+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:19.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:19.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:19.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:19.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:19.110+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:20.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:19.673+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:20.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:19.673+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:20.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:19.673+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:20.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:19.674+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:20.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:19.828+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:20.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:19.828+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:20.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:19.828+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:20.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:19.829+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:20.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:20.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:20.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:20.080+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:21.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:20.630+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:21.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:20.630+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:21.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:20.630+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:21.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:20.631+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:21.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:20.790+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:21.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:20.790+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:21.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:20.790+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:21.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:20.791+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:21.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:21.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:21.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:21.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:21.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:21.050+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:21.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:21.050+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:21.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:21.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:21.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:21.583+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:22.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:21.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:22.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:21.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:22.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:21.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:22.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:21.759+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:22.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:22.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:22.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:22.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:22.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:22.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:22.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:22.055+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:23.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:22.583+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:23.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:22.583+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:23.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:22.583+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:23.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:22.584+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:23.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:22.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:23.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:22.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:23.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:22.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:23.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:22.725+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:23.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:23.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:23.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:23.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:23.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:23.058+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:23.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:23.059+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:23.812 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:23.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:23.813 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:23.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:23.813 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:23.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:23.813 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:23.543+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:23.813 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:23.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:23.813 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:23.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:23.813 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:23.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:23.813 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:23.735+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:24.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:08:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:08:23.810+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52291 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:08:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:24.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:24.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:24.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:24.056+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:24.948 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:24.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:24.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:24.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:24.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:24.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:24.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:24.718+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:24.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:24.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:24.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:24.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:24.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:24.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:24.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:24.586+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:25.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:25.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:25.100+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:25.101+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:25.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:25.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:25.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:25.627+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:26.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:25.737+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:26.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:25.737+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:26.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:25.737+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:26.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:25.738+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:26.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:26.117+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:26.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:26.117+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:26.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:26.117+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:26.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:26.117+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:27.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:26.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:27.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:26.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:27.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:26.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:27.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:26.786+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:27.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:26.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:27.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:26.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:27.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:26.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:27.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:26.617+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:27.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:27.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:27.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:27.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:27.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:27.102+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:27.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:27.103+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:28.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:27.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:28.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:27.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:28.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:27.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:28.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:27.815+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:28.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:27.596+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:28.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:27.596+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:28.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:27.596+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:28.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:27.597+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:28.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:28.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:28.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:28.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:28.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:28.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:28.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:28.084+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:28.936 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:08:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:08:28.817+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52332 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:08:28.937 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:28.614+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:28.937 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:28.614+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:28.937 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:28.614+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:28.937 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:28.615+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:28.937 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:28.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:28.937 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:28.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:28.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:28.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:28.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:28.796+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:29.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:29.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:29.087+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:29.088+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:29.614+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:29.614+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:29.614+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:29.615+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:30.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:29.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:30.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:29.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:30.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:29.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:30.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:29.747+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:30.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:30.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:30.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:30.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:30.096+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:31.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:30.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:31.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:30.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:31.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:30.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:31.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:30.770+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:30.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:30.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:30.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:30.610+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:31.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:31.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:31.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:31.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:31.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:31.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:31.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:31.105+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:32.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:31.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:32.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:31.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:32.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:31.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:32.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:31.743+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:32.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:31.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:32.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:31.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:32.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:31.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:32.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:31.608+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:32.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:32.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:32.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:32.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:32.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:32.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:32.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:32.154+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:32.608+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:32.608+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:32.608+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:32.609+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:32.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:32.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:32.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:32.768+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:33.201+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:33.201+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:33.201+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:33.201+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:33.843 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:08:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:08:33.824+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52371 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:08:33.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:33.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:33.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:33.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:33.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:33.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:33.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:33.798+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:33.844 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:33.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:33.844 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:33.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:33.844 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:33.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:33.844 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:33.615+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:34.250+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:34.250+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:34.250+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:34.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:34.250+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:34.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:34.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:34.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:35.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:34.844+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:35.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:34.628+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:35.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:34.628+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:35.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:34.628+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:35.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:34.629+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:35.628 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:35.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:35.628 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:35.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:35.628 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:35.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:35.628 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:35.285+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:35.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:35.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:36.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:35.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:36.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:35.627+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:35.830+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:35.830+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:35.830+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:35.831+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:36.592 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:36.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:36.592 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:36.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:36.592 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:36.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:36.592 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:36.285+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:36.792+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:37.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:36.792+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:37.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:36.792+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:37.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:36.793+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:37.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:36.590+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:37.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:36.590+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:37.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:36.590+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:37.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:36.591+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:37.562 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:37.274+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:37.562 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:37.274+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:37.562 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:37.274+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:37.562 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:37.274+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:37.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:37.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:38.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:37.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:38.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:37.759+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:38.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:37.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:38.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:37.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:38.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:37.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:38.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:37.561+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:38.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:38.285+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:38.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:38.285+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:38.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:38.285+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:38.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:38.286+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:38.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:38.533+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:38.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:38.533+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:38.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:38.533+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:38.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:38.534+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:39.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:08:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:08:38.831+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52412 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:08:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:38.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:38.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:38.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:38.770+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:39.726 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:39.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:39.726 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:39.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:39.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:39.510+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:39.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:39.511+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:39.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:39.276+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:39.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:39.276+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:39.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:39.276+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:39.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:39.276+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:40.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:39.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:40.023 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:39.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:40.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:39.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:40.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:39.726+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:40.560 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:40.306+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:40.560 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:40.306+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:40.561 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:40.306+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:40.561 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:40.306+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:40.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:40.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:40.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:40.560+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:40.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:40.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:40.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:40.751+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:41.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:41.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:41.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:41.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:41.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:41.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:41.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:41.540+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:41.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:41.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:41.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:41.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:41.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:41.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:41.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:41.297+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:42.248 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:41.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:42.249 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:41.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:42.249 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:41.770+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:42.249 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:41.771+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:42.247+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:42.247+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:42.247+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:42.247+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:42.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:42.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:42.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:42.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:42.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:42.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:42.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:42.543+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:43.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:42.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:43.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:42.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:43.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:42.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:43.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:42.797+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:43.263+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:43.263+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:43.263+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:43.264+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:43.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:43.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:43.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:43.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:43.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:43.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:43.778 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:43.519+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:44.222 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:08:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:08:43.837+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52451 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:08:44.222 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:43.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:44.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:43.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:44.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:43.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:44.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:43.778+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:44.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:44.221+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:44.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:44.221+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:44.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:44.221+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:44.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:44.221+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:44.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:44.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:44.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:45.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:44.570+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:45.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:44.788+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:45.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:44.788+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:45.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:44.788+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:45.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:44.789+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:45.619 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:45.271+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:45.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:45.271+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:45.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:45.271+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:45.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:45.271+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:46.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:45.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:46.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:45.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:46.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:45.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:46.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:45.813+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:45.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:45.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:45.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:45.618+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:46.633 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:46.289+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:46.634 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:46.289+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:46.634 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:46.289+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:46.634 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:46.289+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:46.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:46.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:46.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:46.633+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:47.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:46.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:47.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:46.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:47.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:46.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:47.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:46.836+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:47.604 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:47.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:47.604 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:47.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:47.604 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:47.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:47.605 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:47.297+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:47.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:47.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:47.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:47.603+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:47.845+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:47.845+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:47.845+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:47.846+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:48.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:48.270+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:48.644 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:48.270+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:48.644 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:48.270+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:48.644 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:48.270+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:48.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:08:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:08:48.844+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52492 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:08:48.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:48.851+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:48.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:48.851+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:48.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:48.851+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:48.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:48.852+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:48.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:48.642+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:48.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:48.642+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:48.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:48.642+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:48.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:48.643+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:49.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:49.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:49.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:49.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:49.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:49.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:49.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:49.239+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:49.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:49.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:49.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:49.610+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:50.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:49.830+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:50.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:49.830+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:50.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:49.830+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:50.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:49.831+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:50.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:50.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:50.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:50.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:50.211+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:50.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:50.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:50.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:51.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:50.588+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:51.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:50.808+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:51.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:50.808+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:51.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:50.808+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:51.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:50.809+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:51.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:51.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:51.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:51.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:51.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:51.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:51.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:51.210+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:51.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:51.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:51.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:51.634+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:52.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:51.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:52.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:51.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:52.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:51.841+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:52.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:51.842+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:52.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:52.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:52.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:52.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:52.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:52.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:52.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:52.160+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:53.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:52.622+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:53.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:52.622+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:53.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:52.622+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:53.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:52.623+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:53.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:52.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:53.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:52.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:53.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:52.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:53.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:52.842+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:53.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:53.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:53.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:53.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:53.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:53.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:53.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:53.211+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:53.852 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:53.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:53.852 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:53.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:53.853 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:53.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:53.853 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:53.807+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:53.853 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:53.633+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:53.853 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:53.634+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:53.853 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:53.634+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:53.853 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:53.635+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:54.213 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:08:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:08:53.850+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52531 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:08:54.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:54.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:54.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:54.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:54.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:54.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:54.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:54.212+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:54.645+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:54.645+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:54.645+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:54.646+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:55.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:54.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:55.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:54.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:55.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:54.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:55.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:54.831+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:55.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:55.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:55.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:55.211+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:55.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:55.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:55.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:55.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:55.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:55.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:55.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:55.618+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:56.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:55.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:56.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:55.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:56.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:55.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:56.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:55.873+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:56.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:56.254+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:56.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:56.254+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:56.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:56.254+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:56.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:56.254+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:56.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:56.628+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:56.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:56.628+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:56.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:56.628+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:56.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:56.629+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:57.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:56.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:57.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:56.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:57.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:56.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:57.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:56.883+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:57.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:57.259+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:57.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:57.259+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:57.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:57.259+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:57.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:57.259+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:57.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:57.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:57.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:57.880+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:57.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:57.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:57.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:57.652+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:58.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:58.258+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:58.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:58.258+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:58.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:58.258+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:58.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:58.258+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:58.977 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:08:58.857+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52572 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:08:58.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:58.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:58.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:58.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:58.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:58.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:08:58.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:58.897+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:08:58.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:58.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:58.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:58.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:58.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:58.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:08:58.979 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:58.700+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:08:59.745 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:59.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:59.746 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:59.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:59.746 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:59.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:08:59.746 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:08:59.285+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:59.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:59.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:59.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:08:59.853+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:00.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:59.744+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:00.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:59.744+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:00.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:59.744+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:00.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:08:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:08:59.745+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:00.248+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:00.248+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:00.248+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:00.248+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:01.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:00.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:01.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:00.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:01.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:00.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:01.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:00.765+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:01.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:00.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:01.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:00.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:01.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:00.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:01.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:00.887+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:01.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:01.251+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:01.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:01.252+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:01.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:01.252+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:01.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:01.252+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:01.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:01.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:01.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:02.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:01.743+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:02.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:01.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:02.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:01.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:02.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:01.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:02.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:01.839+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:02.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:02.217+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:02.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:02.217+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:02.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:02.217+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:02.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:02.218+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:03.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:02.888+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:03.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:02.888+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:03.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:02.888+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:03.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:02.889+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:02.733+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:02.733+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:02.733+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:02.734+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:03.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:03.234+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:03.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:03.234+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:03.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:03.234+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:03.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:03.234+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:03.865 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:03.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:03.865 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:03.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:03.865 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:03.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:03.866 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:03.761+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:04.128 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:09:03.863+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52611 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:09:04.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:03.928+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:04.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:03.928+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:04.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:03.928+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:04.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:03.929+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:04.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:04.270+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:04.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:04.270+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:04.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:04.270+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:04.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:04.271+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:04.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:04.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:04.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:04.913+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:04.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:04.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:04.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:04.786+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:05.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:05.293+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:05.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:05.293+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:05.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:05.293+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:05.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:05.294+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:05.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:05.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:05.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:05.907+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:06.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:05.752+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:06.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:05.752+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:06.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:05.752+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:06.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:05.753+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:06.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:06.318+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:06.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:06.318+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:06.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:06.318+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:06.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:06.319+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:06.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:06.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:06.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:06.913+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:06.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:06.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:06.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:07.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:06.794+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:07.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:07.348+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:07.348+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:07.348+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:07.349+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:07.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:07.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:07.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:07.948+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:08.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:07.768+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:08.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:07.768+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:08.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:07.768+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:08.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:07.769+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:08.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:08.330+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:08.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:08.330+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:08.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:08.330+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:08.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:08.331+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:09.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:09:08.870+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52648 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:09:09.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:08.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:09.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:08.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:09.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:08.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:09.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:08.905+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:09.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:08.737+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:09.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:08.737+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:09.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:08.737+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:09.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:08.738+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:09.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:09.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:09.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:09.372+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:09.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:09.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:09.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:10.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:09.728+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:09.876+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:09.876+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:09.876+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:09.876+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:10.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:10.409+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:10.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:10.409+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:10.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:10.409+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:10.700 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:10.410+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:11.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:10.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:11.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:10.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:11.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:10.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:10.909+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:11.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:10.698+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:11.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:10.698+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:11.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:10.698+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:11.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:10.699+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:11.724 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:11.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:11.724 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:11.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:11.724 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:11.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:11.724 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:11.373+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:12.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:11.722+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:12.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:11.722+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:12.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:11.722+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:12.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:11.723+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:11.926+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:11.926+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:11.926+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:11.927+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:12.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:12.369+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:12.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:12.369+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:12.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:12.369+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:12.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:12.369+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:12.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:12.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:12.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:12.878+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:12.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:12.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:12.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:12.771+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:13.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:13.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:13.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:13.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:13.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:13.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:13.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:13.405+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:14.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:09:13.877+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52687 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:09:14.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:13.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:14.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:13.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:14.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:13.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:14.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:13.844+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:14.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:13.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:14.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:13.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:14.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:13.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:14.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:13.781+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:14.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:14.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:14.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:14.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:14.405+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:14.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:14.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:14.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:15.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:14.826+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:14.826+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:14.826+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:14.826+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:15.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:14.827+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:15.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:15.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:15.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:15.412+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:15.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:15.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:15.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:16.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:15.801+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:15.865+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:15.865+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:15.865+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:16.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:15.866+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:16.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:16.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:16.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:16.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:16.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:16.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:16.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:16.415+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:16.840+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:16.840+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:16.840+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:16.841+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:17.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:16.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:17.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:16.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:17.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:16.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:17.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:16.860+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:17.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:17.451+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:17.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:17.451+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:17.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:17.451+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:17.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:17.452+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:17.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:17.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:17.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:17.811+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:17.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:17.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:17.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:18.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:17.819+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:18.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:18.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:18.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:18.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:18.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:18.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:18.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:18.458+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:19.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:09:18.883+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52728 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:09:19.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:18.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:19.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:18.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:19.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:18.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:19.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:18.849+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:19.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:18.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:19.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:18.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:19.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:18.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:19.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:18.763+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:19.750 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:19.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:19.750 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:19.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:19.750 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:19.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:19.750 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:19.423+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:20.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:19.748+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:20.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:19.748+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:20.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:19.748+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:20.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:19.749+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:20.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:19.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:20.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:19.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:20.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:19.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:20.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:19.800+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:20.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:20.430+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:20.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:20.430+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:20.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:20.430+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:20.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:20.430+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:20.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:20.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:20.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:20.804+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:21.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:20.716+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:21.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:20.716+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:21.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:20.716+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:21.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:20.716+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:21.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:21.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:21.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:21.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:21.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:21.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:21.678 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:21.401+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:21.675+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:21.675+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:21.675+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:21.676+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:22.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:21.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:22.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:21.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:22.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:21.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:22.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:21.853+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:22.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:22.410+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:22.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:22.410+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:22.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:22.410+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:22.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:22.410+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:22.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:22.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:22.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:22.833+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:23.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:22.678+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:23.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:22.678+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:23.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:22.678+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:23.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:22.679+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:23.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:23.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:23.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:23.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:23.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:23.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:23.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:23.419+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:24.010 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:09:23.890+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52767 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:09:24.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:23.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:24.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:23.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:24.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:23.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:24.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:23.673+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:24.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:23.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:24.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:23.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:24.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:23.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:24.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:23.881+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:24.665 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:24.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:24.665 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:24.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:24.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:24.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:24.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:24.372+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:25.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:24.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:25.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:24.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:25.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:24.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:25.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:24.664+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:24.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:24.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:24.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:24.905+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:25.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:25.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:25.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:25.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:25.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:25.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:25.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:25.625+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:25.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:25.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:25.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:25.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:25.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:25.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:25.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:25.409+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:25.903+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:25.903+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:25.903+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:25.904+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:26.628 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:26.437+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:26.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:26.437+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:26.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:26.437+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:26.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:26.437+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:26.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:26.627+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:26.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:26.627+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:26.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:26.627+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:26.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:26.628+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:26.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:26.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:26.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:26.924+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:27.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:27.653+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:27.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:27.653+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:27.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:27.653+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:27.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:27.654+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:27.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:27.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:27.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:27.441+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:27.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:27.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:27.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:27.921+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:28.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:28.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:28.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:28.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:28.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:28.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:28.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:28.636+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:28.393+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:28.393+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:28.393+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:28.393+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:29.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:09:28.896+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52808 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:09:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:28.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:28.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:28.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:29.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:28.942+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:29.660 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:29.356+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:29.660 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:29.356+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:29.661 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:29.356+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:29.661 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:29.356+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:29.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:29.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:29.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:29.659+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:30.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:29.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:30.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:29.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:30.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:29.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:30.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:29.897+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:30.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:30.610+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:30.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:30.610+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:30.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:30.610+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:30.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:30.610+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:30.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:30.390+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:30.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:30.390+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:30.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:30.390+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:30.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:30.390+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:30.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:30.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:30.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:30.917+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:31.424+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:31.424+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:31.424+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:31.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:31.425+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:31.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:31.610+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:31.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:31.610+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:31.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:31.610+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:31.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:31.610+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:31.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:31.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:31.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:31.938+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:32.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:32.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:32.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:32.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:32.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:32.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:32.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:32.457+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:32.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:32.583+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:32.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:32.583+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:32.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:32.583+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:32.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:32.583+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:32.891+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:32.891+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:32.891+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:32.892+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:33.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:33.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:33.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:33.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:33.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:33.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:33.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:33.624+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:33.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:33.444+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:33.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:33.444+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:33.765 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:33.444+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:33.765 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:33.444+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:34.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:09:33.903+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52847 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:09:34.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:33.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:34.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:33.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:34.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:33.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:34.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:33.878+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:34.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:34.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:34.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:34.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:34.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:34.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:34.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:34.673+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:34.466+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:34.466+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:34.466+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:34.467+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:34.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:34.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:34.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:34.833+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:35.673+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:35.673+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:35.673+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:35.674+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:35.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:35.504+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:35.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:35.504+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:35.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:35.504+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:35.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:35.504+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:35.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:35.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:35.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:35.860+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:36.905 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:36.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:36.905 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:36.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:36.905 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:36.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:36.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:36.652+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:36.906 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:36.521+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:36.906 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:36.521+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:36.906 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:36.521+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:36.906 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:36.522+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:36.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:36.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:36.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:36.904+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:37.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:37.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:37.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:37.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:37.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:37.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:37.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:37.494+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:37.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:37.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:37.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:37.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:37.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:37.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:37.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:37.633+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:37.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:37.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:37.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:37.886+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:38.595+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:38.595+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:38.595+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:38.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:38.596+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:38.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:38.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:38.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:38.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:38.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:38.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:38.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:38.474+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:39.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:09:38.910+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52888 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:09:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:38.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:38.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:38.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:38.870+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:39.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:39.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:39.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:39.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:39.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:39.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:39.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:39.627+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:39.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:39.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:39.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:39.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:39.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:39.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:39.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:39.509+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:39.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:39.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:39.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:39.835+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:40.842 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:40.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:40.842 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:40.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:40.842 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:40.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:40.842 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:40.616+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:40.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:40.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:40.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:40.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:40.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:40.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:40.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:40.529+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:40.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:40.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:40.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:40.841+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:41.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:41.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:41.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:41.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:41.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:41.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:41.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:41.616+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:41.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:41.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:41.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:41.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:41.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:41.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:41.506+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:41.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:41.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:41.874+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:41.875+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:42.862 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:42.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:42.862 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:42.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:42.862 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:42.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:42.862 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:42.620+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:42.863 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:42.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:42.863 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:42.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:42.863 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:42.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:42.863 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:42.549+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:42.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:42.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:42.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:42.861+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:43.881 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:43.590+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:43.881 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:43.590+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:43.881 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:43.590+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:43.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:43.591+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:43.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:43.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:43.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:43.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:43.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:43.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:43.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:43.580+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:44.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:09:43.916+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52927 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:09:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:43.879+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:43.879+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:43.879+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:44.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:43.880+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:44.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:44.584+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:44.884 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:44.584+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:44.884 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:44.584+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:44.884 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:44.584+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:44.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:44.585+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:44.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:44.585+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:44.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:44.585+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:44.885 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:44.586+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:44.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:44.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:44.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:44.886+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:45.849 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:45.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:45.849 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:45.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:45.849 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:45.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:45.849 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:45.558+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:45.850 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:45.619+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:45.850 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:45.619+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:45.850 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:45.619+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:45.850 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:45.619+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:45.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:45.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:45.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:45.848+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:46.880 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:46.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:46.880 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:46.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:46.880 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:46.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:46.880 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:46.589+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:46.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:46.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:46.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:46.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:46.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:46.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:46.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:46.577+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:46.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:46.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:46.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:46.879+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:47.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:47.612+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:47.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:47.612+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:47.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:47.612+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:47.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:47.613+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:47.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:47.581+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:47.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:47.581+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:47.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:47.581+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:47.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:47.581+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:47.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:47.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:47.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:47.871+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:48.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:48.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:48.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:48.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:48.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:48.847+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:48.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:48.848+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:48.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:48.634+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:48.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:48.634+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:48.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:48.634+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:48.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:48.635+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:48.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:48.613+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:48.927 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:48.613+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:48.927 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:48.613+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:48.927 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:48.614+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:49.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:09:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:09:48.923+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 52968 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:09:50.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:49.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:50.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:49.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:50.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:49.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:50.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:49.819+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:50.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:49.649+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:50.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:49.649+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:50.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:49.649+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:50.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:49.650+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:50.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:49.657+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:50.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:49.657+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:50.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:49.657+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:50.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:49.657+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:50.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:50.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:50.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:50.626+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:51.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:50.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:51.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:50.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:51.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:50.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:51.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:50.794+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:51.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:50.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:51.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:50.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:51.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:50.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:51.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:50.646+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:52.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:51.618+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:52.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:51.618+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:52.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:51.618+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:52.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:51.618+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:52.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:51.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:52.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:51.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:52.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:51.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:52.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:51.606+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:52.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:51.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:52.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:51.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:52.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:51.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:52.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:51.762+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:53.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:52.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:53.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:52.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:53.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:52.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:53.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:52.652+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:53.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:52.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:53.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:52.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:53.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:52.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:53.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:52.784+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:53.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:52.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:53.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:52.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:53.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:52.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:53.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:52.573+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:53.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:53.789+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:53.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:53.789+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:53.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:53.789+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:53.933 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:53.790+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:53.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:53.596+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:53.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:53.596+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:53.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:53.596+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:53.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:53.596+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:53.933 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:53.690+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:53.934 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:53.690+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:53.934 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:53.690+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:53.934 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:53.691+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:54.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:09:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:09:53.930+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53007 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:09:55.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:54.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:55.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:54.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:55.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:54.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:55.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:54.550+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:54.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:54.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:54.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:54.722+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:55.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:54.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:55.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:54.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:55.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:54.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:55.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:54.741+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:56.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:55.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:56.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:55.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:56.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:55.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:56.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:55.787+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:56.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:55.579+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:56.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:55.579+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:56.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:55.579+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:56.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:55.579+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:56.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:55.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:56.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:55.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:56.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:55.699+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:56.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:55.700+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:56.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:56.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:56.921 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:56.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:56.921 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:56.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:56.921 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:56.781+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:56.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:56.745+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:56.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:56.745+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:56.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:56.745+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:56.922 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:56.746+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:56.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:56.611+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:56.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:56.611+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:56.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:56.611+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:56.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:56.611+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:57.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:57.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:57.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:57.752+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:57.777+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:57.777+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:57.777+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:58.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:57.778+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:58.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:57.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:58.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:57.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:58.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:57.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:58.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:57.572+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:58.938 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:58.768+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:58.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:58.768+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:58.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:58.768+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:09:58.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:58.769+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:58.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:58.775+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:58.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:58.775+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:58.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:58.775+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:09:58.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:58.776+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:09:58.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:58.557+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:58.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:58.557+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:58.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:58.557+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:09:58.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:58.557+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:09:59.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:09:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:09:58.937+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53048 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:10:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:59.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:59.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:59.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:09:59.762+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:00.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:59.533+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:00.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:59.533+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:00.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:59.533+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:00.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:09:59.533+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:00.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:59.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:00.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:59.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:00.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:59.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:00.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:09:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:09:59.768+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:00.795 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:00.746+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:00.795 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:00.746+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:00.795 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:00.746+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:00.795 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:00.747+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:00.796 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:00.521+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:00.796 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:00.521+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:00.796 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:00.521+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:00.796 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:00.521+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:00.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:00.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:00.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:00.794+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:01.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:01.500+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:01.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:01.500+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:01.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:01.500+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:01.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:01.501+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:01.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:01.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:01.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:02.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:01.754+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:02.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:01.837+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:02.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:01.837+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:02.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:01.837+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:02.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:01.838+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:02.713 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:02.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:02.713 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:02.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:02.713 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:02.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:02.713 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:02.459+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:03.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:02.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:03.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:02.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:03.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:02.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:03.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:02.813+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:02.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:02.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:02.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:02.712+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:03.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:03.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:03.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:03.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:03.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:03.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:03.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:03.696+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:03.767 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:03.491+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:03.767 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:03.491+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:03.767 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:03.491+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:03.767 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:03.491+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:04.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:10:03.943+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53087 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:10:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:03.800+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:03.800+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:03.800+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:04.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:03.801+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:04.765 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:04.477+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:04.765 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:04.477+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:04.765 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:04.477+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:04.765 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:04.478+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:04.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:04.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:04.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:04.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:04.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:04.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:04.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:04.669+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:05.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:04.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:05.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:04.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:05.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:04.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:05.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:04.813+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:05.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:05.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:05.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:05.713+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:05.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:05.514+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:05.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:05.514+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:05.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:05.514+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:05.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:05.514+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:05.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:05.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:05.862+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:05.863+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:06.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:06.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:06.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:06.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:06.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:06.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:06.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:06.499+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:07.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:06.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:07.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:06.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:07.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:06.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:07.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:06.755+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:07.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:06.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:07.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:06.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:07.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:06.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:07.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:06.864+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:07.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:07.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:07.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:07.513+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:07.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:07.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:07.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:07.775+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:07.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:07.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:07.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:07.890+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:08.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:08.731+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:08.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:08.731+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:08.937 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:08.731+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:08.937 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:08.732+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:08.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:08.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:08.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:08.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:08.938 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:08.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:08.938 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:08.557+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:09.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:10:08.950+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53128 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:10:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:08.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:08.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:09.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:08.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:09.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:08.936+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:09.816 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:09.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:09.817 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:09.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:09.817 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:09.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:09.817 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:09.701+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:09.817 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:09.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:09.817 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:09.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:09.817 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:09.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:09.818 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:09.519+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:09.895+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:09.895+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:09.895+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:09.895+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:10.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:10.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:10.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:10.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:10.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:10.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:10.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:10.474+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:11.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:10.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:11.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:10.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:11.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:10.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:11.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:10.847+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:10.733+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:11.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:10.733+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:11.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:10.733+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:11.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:10.734+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:11.766 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:11.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:11.766 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:11.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:11.767 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:11.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:11.767 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:11.510+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:12.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:11.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:12.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:11.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:12.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:11.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:12.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:11.810+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:12.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:11.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:12.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:11.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:12.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:11.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:12.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:11.766+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:12.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:12.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:12.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:12.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:12.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:12.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:12.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:12.508+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:12.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:12.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:12.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:12.778+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:12.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:12.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:12.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:13.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:12.792+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:13.800 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:13.771+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:13.800 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:13.771+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:13.800 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:13.771+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:13.801 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:13.772+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:13.801 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:13.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:13.801 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:13.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:13.801 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:13.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:13.801 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:13.531+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:14.078 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:10:13.957+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53167 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:10:14.078 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:13.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:14.078 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:13.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:14.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:13.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:14.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:13.799+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:14.862 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:14.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:14.862 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:14.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:14.862 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:14.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:14.862 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:14.767+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:14.863 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:14.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:14.863 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:14.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:14.863 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:14.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:14.863 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:14.766+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:14.863 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:14.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:14.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:14.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:14.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:14.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:14.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:14.572+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:16.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:15.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:16.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:15.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:16.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:15.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:16.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:15.743+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:16.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:15.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:16.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:15.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:16.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:15.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:16.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:15.755+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:16.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:15.603+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:16.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:15.603+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:16.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:15.603+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:16.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:15.604+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:16.560+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:16.560+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:16.560+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:16.560+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:17.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:16.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:17.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:16.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:17.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:16.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:17.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:16.771+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:17.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:16.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:17.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:16.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:17.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:16.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:17.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:16.749+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:18.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:17.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:18.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:17.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:18.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:17.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:18.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:17.777+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:17.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:17.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:18.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:17.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:18.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:17.787+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:18.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:17.583+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:18.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:17.583+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:18.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:17.583+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:18.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:17.584+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:18.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:18.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:18.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:18.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:18.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:18.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:18.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:18.754+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:18.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:18.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:18.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:18.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:18.590+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:18.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:18.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:18.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:18.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:18.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:18.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:18.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:18.787+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:19.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:10:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:10:18.963+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53208 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:10:19.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:19.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:19.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:19.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:19.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:19.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:19.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:19.763+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:19.914 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:19.563+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:19.915 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:19.563+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:19.915 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:19.563+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:19.915 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:19.563+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:19.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:19.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:19.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:19.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:19.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:19.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:19.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:19.719+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:20.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:20.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:20.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:20.712+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:21.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:20.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:21.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:20.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:21.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:20.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:21.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:20.728+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:21.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:20.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:21.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:20.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:21.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:20.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:21.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:20.553+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:21.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:21.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:21.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:22.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:21.743+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:22.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:21.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:22.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:21.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:22.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:21.747+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:22.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:21.748+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:22.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:21.600+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:22.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:21.600+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:22.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:21.600+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:22.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:21.600+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:22.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:22.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:22.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:22.792+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:23.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:22.717+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:23.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:22.717+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:23.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:22.717+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:23.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:22.718+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:23.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:22.596+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:23.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:22.596+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:23.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:22.596+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:23.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:22.597+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:23.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:23.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:23.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:23.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:23.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:23.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:23.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:23.752+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:23.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:23.698+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:23.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:23.698+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:23.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:23.698+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:23.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:23.698+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:23.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:23.612+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:23.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:23.612+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:23.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:23.612+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:23.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:23.612+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:24.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:10:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:10:23.969+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53247 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:10:24.963 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:24.664+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:24.964 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:24.664+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:24.964 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:24.664+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:24.964 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:24.664+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:24.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:24.736+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:24.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:24.736+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:24.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:24.736+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:24.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:24.737+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:24.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:24.647+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:24.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:24.647+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:24.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:24.647+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:24.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:24.647+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:25.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:25.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:25.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:26.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:25.708+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:26.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:25.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:26.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:25.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:26.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:25.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:26.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:25.748+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:26.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:25.655+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:26.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:25.656+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:26.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:25.656+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:26.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:25.656+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:26.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:26.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:26.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:26.695+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:27.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:26.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:27.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:26.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:27.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:26.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:27.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:26.781+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:27.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:26.636+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:27.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:26.636+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:27.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:26.636+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:27.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:26.637+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:27.671+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:27.671+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:27.671+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:27.672+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:28.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:27.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:28.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:27.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:28.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:27.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:28.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:27.766+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:28.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:27.669+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:28.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:27.669+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:28.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:27.669+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:28.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:27.669+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:28.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:28.781+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:28.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:28.781+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:28.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:28.781+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:28.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:28.781+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:28.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:28.693+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:28.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:28.693+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:28.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:28.693+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:28.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:28.693+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:28.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:28.662+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:28.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:28.662+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:28.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:28.662+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:28.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:28.663+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:29.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:10:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:10:28.976+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53288 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:10:30.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:29.696+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:30.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:29.696+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:30.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:29.696+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:30.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:29.697+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:30.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:29.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:30.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:29.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:30.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:29.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:30.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:29.623+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:30.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:29.783+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:30.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:29.783+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:30.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:29.783+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:30.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:29.784+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:31.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:30.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:31.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:30.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:31.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:30.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:31.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:30.695+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:30.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:30.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:30.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:30.671+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:31.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:30.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:31.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:30.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:31.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:30.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:31.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:30.813+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:31.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:31.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:31.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:31.658+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:31.827+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:31.827+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:31.827+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:31.828+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:32.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:31.698+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:32.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:31.698+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:32.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:31.698+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:32.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:31.698+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:33.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:32.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:33.044 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:32.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:33.044 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:32.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:33.045 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:32.682+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:33.045 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:32.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:33.045 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:32.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:33.045 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:32.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:33.045 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:32.736+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:33.046 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:32.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:33.046 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:32.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:33.046 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:32.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:33.046 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:32.854+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:33.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:33.683+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:33.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:33.683+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:33.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:33.683+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:33.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:33.683+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:33.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:33.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:33.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:33.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:33.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:33.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:33.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:33.713+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:33.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:33.826+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:33.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:33.826+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:33.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:33.826+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:33.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:33.826+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:34.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:10:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:10:33.983+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53327 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:10:35.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:34.700+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:35.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:34.700+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:35.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:34.700+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:35.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:34.701+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:35.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:34.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:35.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:34.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:35.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:34.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:35.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:34.835+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:35.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:34.727+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:35.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:34.727+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:35.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:34.727+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:35.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:34.727+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:35.683+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:35.683+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:35.683+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:35.684+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:35.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:35.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:35.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:35.847+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:36.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:35.711+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:36.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:35.711+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:36.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:35.711+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:36.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:35.712+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:36.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:36.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:36.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:36.881+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:37.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:36.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:37.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:36.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:37.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:36.711+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:37.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:36.712+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:37.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:36.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:37.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:36.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:37.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:36.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:37.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:36.683+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:38.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:37.690+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:38.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:37.690+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:38.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:37.690+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:38.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:37.691+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:38.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:37.926+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:38.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:37.926+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:38.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:37.926+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:38.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:37.926+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:38.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:37.728+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:38.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:37.728+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:38.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:37.728+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:38.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:37.729+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:38.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:38.709+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:38.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:38.709+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:38.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:38.709+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:38.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:38.710+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:38.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:38.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:38.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:38.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:38.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:38.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:38.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:38.877+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:38.992 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:38.686+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:38.993 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:38.686+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:38.993 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:38.686+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:38.993 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:38.686+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:39.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:10:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:10:38.989+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53368 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:10:40.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:39.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:40.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:39.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:40.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:39.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:40.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:39.689+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:39.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:39.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:39.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:39.851+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:40.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:39.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:40.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:39.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:40.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:39.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:40.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:39.714+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:40.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:40.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:40.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:40.880+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:40.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:40.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:40.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:40.665+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:41.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:40.684+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:41.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:40.684+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:41.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:40.684+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:41.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:40.685+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:42.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:41.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:42.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:41.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:42.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:41.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:42.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:41.658+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:42.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:41.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:42.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:41.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:42.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:41.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:42.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:41.849+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:42.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:41.643+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:42.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:41.643+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:42.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:41.643+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:42.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:41.643+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:42.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:42.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:42.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:42.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:42.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:42.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:42.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:42.671+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:42.900 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:42.600+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:42.900 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:42.600+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:42.900 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:42.600+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:42.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:42.601+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:42.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:42.898+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:42.898+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:42.898+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:43.860 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:43.644+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:43.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:43.644+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:43.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:43.644+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:43.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:43.645+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:43.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:43.579+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:43.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:43.579+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:43.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:43.579+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:43.862 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:43.579+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:44.116 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:10:43.995+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53407 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:10:44.116 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:43.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:44.117 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:43.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:44.117 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:43.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:44.117 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:43.860+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:44.880 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:44.679+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:44.880 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:44.679+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:44.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:44.679+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:44.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:44.680+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:44.881 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:44.541+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:44.881 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:44.541+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:44.881 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:44.541+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:44.881 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:44.542+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:45.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:44.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:45.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:44.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:45.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:44.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:45.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:44.880+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:45.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:45.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:45.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:45.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:45.662+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:45.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:45.499+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:45.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:45.499+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:45.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:45.499+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:45.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:45.499+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:45.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:45.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:45.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:46.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:45.873+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:46.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:46.639+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:46.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:46.639+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:46.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:46.639+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:46.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:46.641+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:46.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:46.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:46.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:46.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:46.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:46.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:46.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:46.487+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:46.873+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:46.873+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:46.873+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:46.873+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:47.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:47.465+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:47.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:47.465+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:47.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:47.465+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:47.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:47.466+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:47.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:47.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:47.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:47.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:47.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:47.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:47.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:47.632+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:48.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:47.837+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:48.316 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:47.837+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:48.316 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:47.837+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:48.316 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:47.838+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:48.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:48.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:48.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:48.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:48.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:48.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:48.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:48.511+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:48.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:48.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:48.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:48.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:48.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:48.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:48.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:48.589+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:49.122 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:10:49.002+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53448 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:10:49.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:48.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:49.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:48.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:49.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:48.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:49.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:48.815+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:49.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:49.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:49.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:49.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:49.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:49.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:49.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:49.574+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:49.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:49.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:49.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:49.503+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:50.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:49.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:50.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:49.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:50.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:49.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:50.222 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:49.788+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:51.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:50.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:51.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:50.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:51.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:50.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:51.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:50.750+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:51.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:50.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:51.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:50.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:51.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:50.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:51.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:50.624+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:51.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:50.535+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:51.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:50.535+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:51.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:50.535+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:51.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:50.536+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:52.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:51.771+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:52.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:51.771+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:52.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:51.771+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:52.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:51.771+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:52.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:51.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:52.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:51.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:52.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:51.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:52.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:51.549+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:52.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:51.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:52.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:51.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:52.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:51.621+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:52.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:51.622+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:52.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:52.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:52.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:52.659+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:52.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:52.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:52.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:52.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:52.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:52.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:52.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:52.508+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:52.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:52.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:52.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:52.797+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:53.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:53.639+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:53.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:53.639+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:53.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:53.639+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:53.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:53.640+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:53.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:53.480+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:53.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:53.480+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:53.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:53.480+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:53.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:53.481+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:54.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:10:54.009+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53487 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:10:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:53.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:53.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:53.844+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:53.845+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:54.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:54.639+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:54.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:54.639+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:54.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:54.639+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:54.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:54.639+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:54.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:54.450+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:54.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:54.450+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:54.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:54.450+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:54.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:54.450+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:54.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:54.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:54.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:54.850+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:55.640 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:55.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:55.640 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:55.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:55.640 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:55.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:55.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:55.412+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:55.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:55.638+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:55.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:55.638+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:55.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:55.638+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:55.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:55.639+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:55.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:55.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:55.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:55.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:55.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:55.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:55.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:55.873+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:56.682 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:56.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:56.682 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:56.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:56.683 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:56.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:56.683 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:56.431+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:56.683 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:56.680+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:56.683 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:56.680+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:56.683 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:56.680+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:56.683 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:56.681+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:57.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:56.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:57.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:56.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:57.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:56.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:57.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:56.890+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:57.722 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:57.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:57.722 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:57.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:57.722 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:57.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:57.723 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:57.442+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:58.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:57.720+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:58.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:57.720+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:58.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:57.720+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:58.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:57.721+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:58.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:57.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:58.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:57.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:58.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:57.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:58.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:57.885+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:58.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:58.434+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:58.703 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:58.434+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:58.703 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:58.434+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:58.703 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:58.434+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:59.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:58.868+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:59.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:58.868+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:59.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:58.868+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:10:59.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:58.869+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:10:59.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:58.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:59.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:58.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:59.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:58.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:59.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:58.702+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:59.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:10:59.015+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53527 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:10:59.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:59.430+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:59.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:59.430+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:59.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:59.430+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:10:59.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:10:59.431+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:10:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:59.664+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:59.664+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:59.664+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:10:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:10:59.665+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:59.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:59.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:59.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:10:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:10:59.851+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:00.618 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:00.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:00.618 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:00.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:00.619 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:00.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:00.619 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:00.439+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:01.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:00.810+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:01.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:00.810+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:01.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:00.810+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:01.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:00.811+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:01.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:00.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:01.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:00.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:01.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:00.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:01.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:00.618+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:01.398+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:01.398+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:01.398+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:01.398+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:01.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:01.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:01.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:01.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:01.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:01.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:01.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:01.594+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:01.828+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:01.828+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:01.828+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:01.829+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:02.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:02.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:02.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:02.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:02.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:02.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:02.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:02.547+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:02.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:02.434+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:02.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:02.434+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:02.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:02.434+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:02.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:02.434+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:02.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:02.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:02.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:02.849+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:03.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:03.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:03.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:03.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:03.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:03.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:03.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:03.540+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:03.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:03.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:03.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:03.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:03.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:03.441+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:03.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:03.441+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:04.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:03.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:04.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:03.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:04.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:03.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:04.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:03.840+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:04.425 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:11:04.022+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53567 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:11:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:04.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:04.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:04.582+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:04.583+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:04.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:04.424+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:04.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:04.424+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:04.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:04.424+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:04.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:04.425+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:05.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:04.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:05.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:04.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:05.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:04.882+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:05.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:04.883+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:05.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:05.399+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:05.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:05.399+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:05.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:05.399+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:05.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:05.399+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:05.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:05.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:05.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:05.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:05.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:05.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:05.875 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:05.538+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:05.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:05.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:05.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:05.873+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:06.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:06.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:06.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:06.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:06.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:06.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:06.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:06.405+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:06.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:06.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:06.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:06.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:06.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:06.496+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:06.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:06.497+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:06.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:06.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:06.887+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:07.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:06.887+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:07.429+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:07.429+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:07.429+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:07.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:07.429+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:07.490+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:07.490+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:07.490+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:07.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:07.491+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:07.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:07.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:07.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:07.843+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:08.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:08.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:08.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:08.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:08.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:08.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:08.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:08.517+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:08.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:08.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:08.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:08.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:08.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:08.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:08.772 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:08.446+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:09.148 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:11:09.029+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53607 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:11:09.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:08.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:09.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:08.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:09.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:08.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:09.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:08.841+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:09.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:09.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:09.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:09.568+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:09.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:09.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:09.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:09.404+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:09.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:09.404+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:09.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:09.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:09.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:09.842+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:10.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:10.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:10.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:10.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:10.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:10.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:10.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:10.528+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:10.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:10.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:10.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:10.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:10.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:10.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:10.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:10.427+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:10.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:10.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:10.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:10.832+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:11.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:11.443+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:11.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:11.443+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:11.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:11.443+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:11.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:11.443+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:11.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:11.513+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:11.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:11.513+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:11.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:11.513+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:11.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:11.514+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:11.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:11.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:11.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:11.812+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:12.494+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:12.494+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:12.494+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:12.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:12.495+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:12.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:12.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:12.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:12.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:12.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:12.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:12.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:12.445+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:12.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:12.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:12.835+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:12.836+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:13.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:13.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:13.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:13.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:13.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:13.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:13.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:13.505+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:13.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:13.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:13.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:13.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:13.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:13.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:13.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:13.476+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:14.038 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:13.856+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:14.038 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:13.856+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:14.038 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:13.856+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:14.038 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:13.857+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:14.419 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:11:14.036+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53647 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:11:14.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:14.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:14.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:14.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:14.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:14.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:14.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:14.524+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:14.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:14.434+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:14.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:14.434+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:14.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:14.434+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:14.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:14.435+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:14.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:14.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:14.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:14.861+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:15.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:15.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:15.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:15.474+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:15.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:15.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:15.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:15.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:15.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:15.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:15.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:15.536+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:15.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:15.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:15.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:15.859+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:16.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:16.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:16.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:16.554+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:16.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:16.434+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:16.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:16.434+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:16.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:16.434+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:16.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:16.435+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:16.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:16.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:16.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:16.841+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:17.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:17.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:17.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:17.546+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:17.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:17.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:17.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:17.440+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:17.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:17.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:17.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:17.890+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:18.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:18.390+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:18.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:18.390+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:18.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:18.390+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:18.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:18.390+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:18.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:18.519+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:18.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:18.519+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:18.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:18.519+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:18.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:18.520+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:19.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:11:19.043+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53687 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:11:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:18.927+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:18.927+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:18.927+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:18.927+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:19.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:19.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:19.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:19.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:19.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:19.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:19.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:19.544+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:19.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:19.357+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:19.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:19.357+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:19.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:19.357+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:19.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:19.357+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:19.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:19.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:19.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:19.953+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:20.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:20.574+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:20.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:20.574+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:20.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:20.574+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:20.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:20.574+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:20.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:20.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:20.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:20.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:20.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:20.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:20.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:20.373+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:20.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:20.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:20.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:20.953+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:21.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:21.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:21.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:21.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:21.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:21.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:21.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:21.556+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:21.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:21.394+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:21.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:21.394+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:21.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:21.394+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:21.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:21.395+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:21.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:21.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:21.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:21.968+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:22.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:22.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:22.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:22.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:22.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:22.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:22.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:22.518+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:22.387+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:22.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:22.387+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:22.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:22.387+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:22.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:22.387+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:22.959+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:22.959+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:22.959+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:22.959+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:23.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:23.358+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:23.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:23.358+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:23.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:23.358+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:23.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:23.358+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:23.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:23.534+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:23.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:23.534+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:23.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:23.534+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:23.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:23.535+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:24.051 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:23.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:24.051 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:23.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:24.051 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:23.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:24.051 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:23.986+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:24.317 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:11:24.049+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53727 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:11:24.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:24.562+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:24.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:24.562+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:24.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:24.562+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:24.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:24.562+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:24.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:24.316+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:24.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:24.316+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:24.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:24.316+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:24.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:24.317+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:24.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:24.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:24.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:24.978+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:25.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:25.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:25.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:25.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:25.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:25.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:25.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:25.551+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:25.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:25.355+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:25.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:25.355+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:25.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:25.355+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:25.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:25.355+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:25.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:25.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:25.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:25.961+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:26.542 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:26.370+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:26.542 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:26.370+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:26.542 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:26.370+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:26.543 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:26.370+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:26.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:26.541+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:26.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:26.541+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:26.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:26.541+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:26.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:26.542+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:26.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:26.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:26.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:26.954+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:27.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:27.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:27.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:27.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:27.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:27.560+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:27.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:27.561+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:27.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:27.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:27.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:27.380+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:28.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:27.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:27.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:27.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:27.921+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:28.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:28.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:28.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:28.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:28.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:28.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:28.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:28.612+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:28.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:28.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:28.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:28.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:28.380+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:29.175 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:11:29.057+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53767 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:11:29.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:28.903+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:29.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:28.903+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:29.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:28.903+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:29.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:28.904+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:29.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:29.370+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:29.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:29.370+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:29.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:29.370+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:29.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:29.370+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:29.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:29.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:29.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:29.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:29.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:29.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:29.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:29.620+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:29.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:29.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:29.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:29.906+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:30.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:30.340+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:30.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:30.340+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:30.616 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:30.340+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:30.616 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:30.340+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:30.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:30.614+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:30.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:30.614+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:30.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:30.614+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:30.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:30.615+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:30.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:30.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:30.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:30.881+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:31.657 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:31.293+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:31.658 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:31.293+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:31.658 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:31.293+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:31.658 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:31.294+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:32.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:31.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:32.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:31.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:32.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:31.886+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:32.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:31.887+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:32.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:31.656+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:32.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:31.656+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:32.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:31.656+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:32.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:31.657+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:32.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:32.338+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:32.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:32.338+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:32.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:32.338+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:32.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:32.338+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:33.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:32.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:33.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:32.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:33.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:32.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:33.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:32.913+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:32.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:33.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:32.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:33.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:32.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:33.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:32.703+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:33.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:33.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:33.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:33.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:33.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:33.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:33.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:33.324+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:33.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:33.734+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:33.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:33.734+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:33.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:33.734+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:33.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:33.734+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:34.183 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:11:34.064+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53807 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:11:34.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:33.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:34.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:33.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:34.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:33.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:34.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:33.873+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:34.286+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:34.286+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:34.286+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:34.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:34.286+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:35.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:34.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:35.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:34.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:35.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:34.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:35.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:34.702+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:35.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:34.888+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:35.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:34.888+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:35.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:34.888+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:35.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:34.888+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:35.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:35.323+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:35.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:35.323+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:35.687 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:35.323+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:35.687 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:35.323+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:35.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:35.740+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:35.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:35.740+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:35.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:35.740+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:35.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:35.740+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:35.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:35.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:35.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:35.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:35.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:35.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:35.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:35.855+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:36.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:36.295+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:36.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:36.295+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:36.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:36.295+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:36.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:36.295+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:36.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:36.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:36.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:36.760+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:37.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:36.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:37.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:36.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:37.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:36.818+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:37.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:36.819+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:37.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:37.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:37.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:37.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:37.728 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:37.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:37.728 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:37.324+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:38.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:37.726+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:38.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:37.726+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:38.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:37.726+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:38.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:37.726+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:38.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:37.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:38.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:37.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:38.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:37.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:38.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:37.808+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:38.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:38.195+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:38.196070+0000) 2024-02-19T20:11:38.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:38.391+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:38.392231+0000) 2024-02-19T20:11:38.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:38.338+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:38.705 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:38.338+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:38.705 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:38.338+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:38.705 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:38.339+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:38.705 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:38.472+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:38.472996+0000) 2024-02-19T20:11:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:38.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:38.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:38.850+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:38.851+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:39.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:38.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:39.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:38.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:39.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:38.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:39.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:38.703+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:39.297 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:11:39.072+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53847 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:11:39.297 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:39.068+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:39.069926+0000) 2024-02-19T20:11:39.298 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:39.068+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:39.069492+0000) 2024-02-19T20:11:39.298 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:39.068+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:39.069492+0000) 2024-02-19T20:11:39.298 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:39.195+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:39.196243+0000) 2024-02-19T20:11:39.701 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:39.391+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:39.392387+0000) 2024-02-19T20:11:39.701 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:39.378+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:39.701 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:39.378+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:39.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:39.378+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:39.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:39.379+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:39.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:39.472+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:39.473142+0000) 2024-02-19T20:11:40.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:39.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:40.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:39.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:40.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:39.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:40.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:39.843+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:40.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:39.700+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:40.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:39.700+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:40.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:39.700+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:40.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:39.701+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:40.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:40.197+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:40.198807+0000) 2024-02-19T20:11:40.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:40.656+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:40.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:40.656+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:40.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:40.656+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:40.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:40.394+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:40.395668+0000) 2024-02-19T20:11:40.659 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:40.384+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:40.659 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:40.384+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:40.659 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:40.384+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:40.659 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:40.384+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:40.659 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:40.474+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:40.475176+0000) 2024-02-19T20:11:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:40.657+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:40.800+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:40.800+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:40.800+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:40.801+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:41.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:41.199+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:41.200972+0000) 2024-02-19T20:11:41.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:41.396+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:41.397774+0000) 2024-02-19T20:11:41.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:41.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:41.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:41.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:41.708 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:41.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:41.708 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:41.423+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:41.708 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:41.476+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:41.477334+0000) 2024-02-19T20:11:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:41.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:41.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:41.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:41.795+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:42.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:41.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:42.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:41.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:42.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:41.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:42.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:41.706+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:42.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:42.201+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:42.202410+0000) 2024-02-19T20:11:42.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:42.398+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:42.399217+0000) 2024-02-19T20:11:42.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:42.424+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:42.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:42.424+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:42.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:42.424+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:42.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:42.425+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:42.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:42.477+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:42.478932+0000) 2024-02-19T20:11:43.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:42.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:43.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:42.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:43.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:42.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:43.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:42.797+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:43.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:42.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:43.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:42.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:43.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:42.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:43.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:42.695+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:43.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:43.201+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:43.202624+0000) 2024-02-19T20:11:43.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:43.398+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:43.399370+0000) 2024-02-19T20:11:43.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:43.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:43.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:43.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:43.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:43.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:43.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:43.469+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:43.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:43.478+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:43.479129+0000) 2024-02-19T20:11:44.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:43.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:44.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:43.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:44.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:43.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:44.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:43.743+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:44.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:43.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:44.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:43.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:44.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:43.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:44.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:43.805+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:44.297 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:11:44.079+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53887 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:11:44.298 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:44.077+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:44.078830+0000) 2024-02-19T20:11:44.298 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:44.077+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:44.078012+0000) 2024-02-19T20:11:44.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:44.077+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:44.078117+0000) 2024-02-19T20:11:44.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:44.201+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:44.202796+0000) 2024-02-19T20:11:44.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:44.398+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:44.399570+0000) 2024-02-19T20:11:44.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:44.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:44.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:44.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:44.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:44.463+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:44.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:44.478+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:44.479296+0000) 2024-02-19T20:11:45.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:44.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:45.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:44.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:45.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:44.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:45.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:44.795+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:45.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:44.788+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:45.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:44.788+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:45.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:44.788+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:45.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:44.789+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:45.472 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:45.400+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:45.401517+0000) 2024-02-19T20:11:45.473 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:45.204+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:45.205204+0000) 2024-02-19T20:11:45.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:45.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:45.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:45.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:45.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:45.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:45.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:45.471+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:45.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:45.480+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:45.481272+0000) 2024-02-19T20:11:46.207 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:45.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:46.208 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:45.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:46.208 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:45.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:46.208 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:45.827+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:46.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:45.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:46.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:45.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:46.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:45.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:46.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:45.830+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:46.479 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:46.402+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:46.403568+0000) 2024-02-19T20:11:46.480 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:46.206+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:46.207192+0000) 2024-02-19T20:11:46.480 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:46.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:46.480 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:46.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:46.480 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:46.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:46.481 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:46.453+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:46.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:46.482+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:46.483363+0000) 2024-02-19T20:11:47.039 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:46.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:47.039 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:46.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:47.039 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:46.794+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:47.039 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:46.794+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:47.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:46.779+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:47.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:46.779+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:47.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:46.779+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:47.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:46.779+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:47.298 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:47.208+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:47.209008+0000) 2024-02-19T20:11:47.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:47.404+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:47.405017+0000) 2024-02-19T20:11:47.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:47.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:47.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:47.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:47.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:47.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:47.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:47.474+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:47.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:47.483+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:47.484642+0000) 2024-02-19T20:11:48.209 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:47.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:48.209 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:47.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:48.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:47.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:48.210 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:47.812+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:48.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:47.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:48.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:47.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:48.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:47.811+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:48.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:47.812+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:48.461 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:48.208+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:48.209168+0000) 2024-02-19T20:11:48.461 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:48.404+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:48.405161+0000) 2024-02-19T20:11:48.462 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:48.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:48.462 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:48.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:48.462 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:48.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:48.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:48.460+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:48.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:48.483+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:48.484811+0000) 2024-02-19T20:11:49.039 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:48.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:49.039 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:48.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:49.040 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:48.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:49.040 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:48.779+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:49.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:48.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:49.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:48.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:49.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:48.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:49.040 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:48.781+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:49.298 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:11:49.086+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53927 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:11:49.298 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:49.082+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:49.083957+0000) 2024-02-19T20:11:49.299 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:49.083+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:49.084257+0000) 2024-02-19T20:11:49.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:49.083+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:49.084699+0000) 2024-02-19T20:11:49.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:49.208+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:49.209398+0000) 2024-02-19T20:11:49.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:49.404+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:49.405325+0000) 2024-02-19T20:11:49.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:49.448+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:49.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:49.448+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:49.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:49.448+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:49.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:49.448+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:49.484+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:49.484974+0000) 2024-02-19T20:11:50.028 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:49.769+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:50.028 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:49.769+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:50.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:49.769+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:50.029 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:49.770+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:50.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:49.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:50.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:49.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:50.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:49.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:50.029 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:49.792+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:50.298 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:50.210+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:50.211558+0000) 2024-02-19T20:11:50.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:50.406+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:50.407451+0000) 2024-02-19T20:11:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:50.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:50.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:50.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:50.471+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:50.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:50.486+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:50.487360+0000) 2024-02-19T20:11:51.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:50.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:51.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:50.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:51.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:50.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:51.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:50.839+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:51.214 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:50.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:51.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:50.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:51.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:50.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:51.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:50.815+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:51.480 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:51.212+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:51.213548+0000) 2024-02-19T20:11:51.480 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:51.408+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:51.409417+0000) 2024-02-19T20:11:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:51.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:51.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:51.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:51.489+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:51.490520+0000) 2024-02-19T20:11:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:51.490+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:52.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:51.794+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:51.794+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:51.794+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:52.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:51.795+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:51.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:51.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:52.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:51.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:52.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:51.808+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:52.480 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:52.214+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:52.215425+0000) 2024-02-19T20:11:52.480 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:52.410+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:52.411119+0000) 2024-02-19T20:11:52.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:52.490+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:52.491932+0000) 2024-02-19T20:11:52.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:52.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:52.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:52.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:52.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:52.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:52.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:52.503+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:53.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:52.773+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:53.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:52.773+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:53.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:52.773+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:53.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:52.773+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:53.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:52.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:53.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:52.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:53.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:52.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:53.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:52.759+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:53.298 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:53.214+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:53.215595+0000) 2024-02-19T20:11:53.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:53.410+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:53.411333+0000) 2024-02-19T20:11:53.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:53.491+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:53.492100+0000) 2024-02-19T20:11:53.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:53.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:53.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:53.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:53.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:53.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:53.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:53.540+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:53.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:53.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:53.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:54.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:53.794+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:54.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:53.757+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:54.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:53.757+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:54.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:53.757+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:54.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:53.758+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:54.299 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:11:54.092+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 53967 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:11:54.299 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:54.090+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:54.091692+0000) 2024-02-19T20:11:54.299 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:54.090+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:54.091366+0000) 2024-02-19T20:11:54.300 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:54.090+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:54.091324+0000) 2024-02-19T20:11:54.300 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:54.214+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:54.215765+0000) 2024-02-19T20:11:54.748 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:54.410+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:54.411511+0000) 2024-02-19T20:11:54.748 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:54.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:54.748 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:54.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:54.749 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:54.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:54.749 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:54.491+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:54.492282+0000) 2024-02-19T20:11:54.749 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:54.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:54.749 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:54.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:54.749 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:54.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:54.749 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:54.528+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:55.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:54.747+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:54.772+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:54.772+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:54.772+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:54.772+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:55.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:55.216+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:55.217917+0000) 2024-02-19T20:11:55.551 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:55.412+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:55.413568+0000) 2024-02-19T20:11:55.551 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:55.493+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:55.494713+0000) 2024-02-19T20:11:55.551 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:55.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:55.552 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:55.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:55.552 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:55.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:20.397072+0000 (oldest deadline 2024-02-19T20:01:40.397072+0000) 2024-02-19T20:11:55.906 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:55.550+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:55.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:55.736+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:55.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:55.736+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:55.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:55.736+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:55.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:55.737+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:55.908 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:55.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:55.908 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:55.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:55.908 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:55.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:55.908 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:55.765+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:56.480 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:56.415+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:56.416081+0000) 2024-02-19T20:11:56.481 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:56.219+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:56.220312+0000) 2024-02-19T20:11:56.766 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:56.496+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:56.497088+0000) 2024-02-19T20:11:56.766 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:56.590+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:56.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:56.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:56.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:56.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:56.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:56.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:56.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:56.709+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:57.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:56.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:57.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:56.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:57.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:56.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:57.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:56.766+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:57.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:57.220+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:57.221887+0000) 2024-02-19T20:11:57.602 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:57.416+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:57.417419+0000) 2024-02-19T20:11:57.603 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:57.498+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:57.499321+0000) 2024-02-19T20:11:57.603 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:57.601+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:58.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:57.734+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:58.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:57.734+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:58.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:57.734+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:58.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:57.735+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:57.721+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:58.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:57.721+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:58.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:57.722+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:58.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:57.722+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:58.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:58.221+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:58.222061+0000) 2024-02-19T20:11:58.647 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:58.416+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:58.417575+0000) 2024-02-19T20:11:58.648 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:58.498+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:58.499488+0000) 2024-02-19T20:11:58.648 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:58.646+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:58.692+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:58.692+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:58.692+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:11:59.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:58.693+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:11:59.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:58.735+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:59.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:58.735+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:59.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:58.735+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:11:59.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:58.736+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:11:59.299 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:11:59.100+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54007 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:11:59.299 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:59.098+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:59.099243+0000) 2024-02-19T20:11:59.300 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:59.096+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:59.097741+0000) 2024-02-19T20:11:59.300 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:59.097+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:59.098482+0000) 2024-02-19T20:11:59.300 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:59.221+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:59.222235+0000) 2024-02-19T20:11:59.599 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:59.416+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:59.417744+0000) 2024-02-19T20:11:59.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:59.498+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:59.499657+0000) 2024-02-19T20:12:00.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:11:59.598+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:00.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:59.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:00.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:59.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:00.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:59.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:00.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:11:59.695+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:00.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:59.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:12:00.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:59.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:12:00.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:59.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:24.922734+0000 (oldest deadline 2024-02-19T20:01:44.922734+0000) 2024-02-19T20:12:00.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:11:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:11:59.726+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:00.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:00.222+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:00.224654+0000) 2024-02-19T20:12:00.604 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:00.417+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:00.419864+0000) 2024-02-19T20:12:00.605 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:00.500+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:00.502172+0000) 2024-02-19T20:12:00.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:00.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:00.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:00.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:00.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:00.703+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:00.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:00.704+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:00.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:00.690+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:00.955 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:00.602+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:01.481 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:01.225+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:01.226789+0000) 2024-02-19T20:12:01.481 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:01.422+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:01.423477+0000) 2024-02-19T20:12:01.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:01.693+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:01.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:01.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:01.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:01.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:01.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:01.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:01.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:01.692+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:01.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:01.503+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:01.504623+0000) 2024-02-19T20:12:01.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:01.640+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:02.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:02.227+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:02.228276+0000) 2024-02-19T20:12:02.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:02.424+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:02.425192+0000) 2024-02-19T20:12:02.657 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:02.505+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:02.506074+0000) 2024-02-19T20:12:03.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:02.655+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:03.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:02.714+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:02.678+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:02.678+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:02.678+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:03.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:02.680+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:03.300 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:03.227+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:03.228452+0000) 2024-02-19T20:12:03.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:03.424+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:03.425422+0000) 2024-02-19T20:12:03.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:03.505+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:03.506237+0000) 2024-02-19T20:12:04.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:03.669+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:04.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:03.727+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:04.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:03.710+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:04.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:03.710+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:04.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:03.710+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:04.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:03.711+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:04.300 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:12:04.106+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54047 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:12:04.300 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:04.104+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:04.105905+0000) 2024-02-19T20:12:04.300 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:04.105+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:04.106024+0000) 2024-02-19T20:12:04.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:04.104+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:04.104995+0000) 2024-02-19T20:12:04.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:04.227+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:04.228622+0000) 2024-02-19T20:12:04.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:04.424+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:04.425628+0000) 2024-02-19T20:12:04.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:04.505+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:04.506402+0000) 2024-02-19T20:12:04.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:04.672+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:05.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:04.764+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:05.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:04.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:05.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:04.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:05.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:04.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:05.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:04.692+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:05.300 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:05.229+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:05.230647+0000) 2024-02-19T20:12:05.694 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:05.427+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:05.428218+0000) 2024-02-19T20:12:05.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:05.508+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:05.509851+0000) 2024-02-19T20:12:06.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:05.726+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:06.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:05.693+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:06.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:05.738+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:06.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:05.738+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:06.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:05.738+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:06.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:05.739+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:06.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:06.231+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:06.232761+0000) 2024-02-19T20:12:06.683 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:06.511+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:06.512484+0000) 2024-02-19T20:12:06.684 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:06.429+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:06.430833+0000) 2024-02-19T20:12:06.684 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:06.682+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:07.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:06.707+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:07.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:06.693+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:07.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:06.693+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:07.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:06.693+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:01:33.843189+0000 (oldest deadline 2024-02-19T20:01:53.843189+0000) 2024-02-19T20:12:07.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:06.694+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:07.300 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:07.233+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:07.234208+0000) 2024-02-19T20:12:07.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:07.431+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:07.432281+0000) 2024-02-19T20:12:07.660 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:07.513+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:07.514030+0000) 2024-02-19T20:12:07.660 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:07.658+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:08.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:07.676+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:07.706+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:08.300 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:08.233+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:08.234383+0000) 2024-02-19T20:12:08.680 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:08.431+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:08.432479+0000) 2024-02-19T20:12:08.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:08.513+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:08.514223+0000) 2024-02-19T20:12:09.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:08.679+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:09.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:08.725+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:09.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:08.706+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:09.301 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:12:09.113+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54087 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:12:09.301 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:09.110+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:09.111794+0000) 2024-02-19T20:12:09.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:09.112+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:09.113566+0000) 2024-02-19T20:12:09.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:09.233+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:09.234585+0000) 2024-02-19T20:12:09.302 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:09.111+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:09.112425+0000) 2024-02-19T20:12:09.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:09.513+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:09.514390+0000) 2024-02-19T20:12:09.698 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:09.431+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:09.432677+0000) 2024-02-19T20:12:09.698 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:09.696+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:10.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:09.726+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:10.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:09.736+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:10.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:10.235+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:10.236743+0000) 2024-02-19T20:12:10.670 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:10.434+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:10.435121+0000) 2024-02-19T20:12:10.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:10.516+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:10.517020+0000) 2024-02-19T20:12:11.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:10.669+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:11.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:10.745+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:10.684+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:11.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:11.238+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:11.239338+0000) 2024-02-19T20:12:11.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:11.518+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:11.518976+0000) 2024-02-19T20:12:11.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:11.437+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:11.438605+0000) 2024-02-19T20:12:11.638 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:11.636+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:12.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:11.777+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:12.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:11.636+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:12.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:12.240+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:12.241577+0000) 2024-02-19T20:12:12.607 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:12.520+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:12.521117+0000) 2024-02-19T20:12:12.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:12.439+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:12.440939+0000) 2024-02-19T20:12:13.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:12.666+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:13.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:12.606+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:13.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:12.801+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:13.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:13.240+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:13.241797+0000) 2024-02-19T20:12:13.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:13.440+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:13.441120+0000) 2024-02-19T20:12:13.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:13.520+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:13.521288+0000) 2024-02-19T20:12:14.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:13.628+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:14.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:13.624+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:14.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:13.803+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:14.301 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:12:14.120+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54127 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:12:14.301 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:14.117+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:14.118823+0000) 2024-02-19T20:12:14.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:14.118+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:14.119311+0000) 2024-02-19T20:12:14.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:14.241+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:14.241989+0000) 2024-02-19T20:12:14.302 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:14.118+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:14.119591+0000) 2024-02-19T20:12:14.664 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:14.520+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:14.521447+0000) 2024-02-19T20:12:14.664 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:14.440+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:14.441290+0000) 2024-02-19T20:12:14.665 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:14.662+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:15.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:14.759+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:14.673+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:15.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:15.242+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:15.243923+0000) 2024-02-19T20:12:15.655 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:15.442+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:15.443807+0000) 2024-02-19T20:12:15.656 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:15.522+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:15.523624+0000) 2024-02-19T20:12:15.923 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:15.654+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:15.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:15.699+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:15.924 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:15.803+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:16.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:16.245+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:16.246340+0000) 2024-02-19T20:12:16.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:16.445+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:16.446297+0000) 2024-02-19T20:12:16.821 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:16.677+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:16.822 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:16.699+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:16.822 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:16.524+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:16.525676+0000) 2024-02-19T20:12:16.822 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:16.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:16.823 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:16.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:16.823 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:16.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:17.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:16.820+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:17.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:17.247+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:17.248506+0000) 2024-02-19T20:12:17.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:17.447+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:17.448394+0000) 2024-02-19T20:12:17.824 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:17.679+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:17.825 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:17.725+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:17.825 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:17.526+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:17.527020+0000) 2024-02-19T20:12:17.825 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:17.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:17.825 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:17.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:17.825 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:17.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:18.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:17.824+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:18.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:18.247+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:18.248669+0000) 2024-02-19T20:12:18.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:18.447+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:18.448581+0000) 2024-02-19T20:12:18.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:18.713+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:18.857 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:18.706+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:18.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:18.526+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:18.527191+0000) 2024-02-19T20:12:18.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:18.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:18.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:18.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:18.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:18.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:19.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:18.856+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:19.449 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:12:19.127+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54167 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:12:19.449 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:19.124+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:19.125703+0000) 2024-02-19T20:12:19.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:19.125+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:19.126884+0000) 2024-02-19T20:12:19.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:19.247+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:19.248825+0000) 2024-02-19T20:12:19.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:19.125+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:19.126448+0000) 2024-02-19T20:12:19.752 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:19.670+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:19.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:19.526+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:19.527352+0000) 2024-02-19T20:12:19.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:19.447+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:19.448763+0000) 2024-02-19T20:12:19.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:19.751+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:20.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:19.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:20.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:19.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:20.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:19.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:20.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:19.833+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:20.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:20.250+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:20.251407+0000) 2024-02-19T20:12:20.710 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:20.449+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:20.450949+0000) 2024-02-19T20:12:20.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:20.529+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:20.530940+0000) 2024-02-19T20:12:21.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:20.708+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:20.745+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:21.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:20.786+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:21.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:20.786+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:21.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:20.786+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:21.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:20.787+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:21.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:21.252+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:21.253566+0000) 2024-02-19T20:12:21.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:21.451+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:21.452927+0000) 2024-02-19T20:12:21.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:21.532+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:21.533111+0000) 2024-02-19T20:12:22.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:21.695+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:22.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:21.775+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:22.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:21.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:22.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:21.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:22.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:21.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:22.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:21.813+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:22.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:22.254+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:22.255161+0000) 2024-02-19T20:12:22.739 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:22.453+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:22.454446+0000) 2024-02-19T20:12:22.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:22.533+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:22.534567+0000) 2024-02-19T20:12:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:22.817+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:23.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:22.737+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:23.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:22.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:23.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:22.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:23.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:22.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:23.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:22.832+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:23.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:23.254+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:23.255310+0000) 2024-02-19T20:12:23.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:23.453+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:23.454629+0000) 2024-02-19T20:12:23.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:23.533+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:23.534755+0000) 2024-02-19T20:12:24.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:23.708+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:24.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:23.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:24.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:23.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:24.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:23.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:24.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:23.807+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:24.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:23.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:24.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:23.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:24.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:23.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:24.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:23.792+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:24.302 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:12:24.134+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54207 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:12:24.303 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:24.131+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:24.132125+0000) 2024-02-19T20:12:24.303 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:24.131+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:24.132201+0000) 2024-02-19T20:12:24.303 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:24.254+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:24.255500+0000) 2024-02-19T20:12:24.304 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:24.131+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:24.132957+0000) 2024-02-19T20:12:24.721 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:24.453+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:24.454802+0000) 2024-02-19T20:12:24.721 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:24.533+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:24.534949+0000) 2024-02-19T20:12:25.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:24.719+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:25.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:24.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:25.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:24.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:25.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:24.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:25.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:24.807+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:24.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:24.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:24.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:24.758+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:25.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:25.256+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:25.257819+0000) 2024-02-19T20:12:25.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:25.455+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:25.456879+0000) 2024-02-19T20:12:25.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:25.536+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:25.537322+0000) 2024-02-19T20:12:26.032 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:25.769+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:26.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:25.781+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:26.033 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:25.781+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:26.034 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:25.781+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:26.034 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:25.782+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:26.034 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:25.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:26.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:25.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:26.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:25.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:26.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:25.840+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:26.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:26.260+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:26.261080+0000) 2024-02-19T20:12:26.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:26.459+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:26.460382+0000) 2024-02-19T20:12:26.731 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:26.539+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:26.540579+0000) 2024-02-19T20:12:27.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:26.730+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:27.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:26.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:27.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:26.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:27.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:26.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:27.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:26.770+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:27.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:26.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:27.024 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:26.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:27.024 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:26.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:27.024 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:26.857+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:27.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:27.261+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:27.262515+0000) 2024-02-19T20:12:27.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:27.541+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:27.542457+0000) 2024-02-19T20:12:27.739 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:27.460+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:27.461812+0000) 2024-02-19T20:12:27.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:27.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:27.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:27.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:28.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:27.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:28.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:27.738+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:28.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:27.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:28.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:27.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:28.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:27.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:28.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:27.841+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:28.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:27.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:28.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:27.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:28.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:27.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:28.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:27.767+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:28.303 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:28.261+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:28.262671+0000) 2024-02-19T20:12:28.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:28.461+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:28.462033+0000) 2024-02-19T20:12:28.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:28.541+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:28.542620+0000) 2024-02-19T20:12:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:28.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:28.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:28.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:28.771+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:29.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:28.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:29.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:28.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:29.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:28.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:29.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:28.716+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:29.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:28.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:29.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:28.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:29.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:28.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:29.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:28.819+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:29.303 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:12:29.140+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54247 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:12:29.303 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:29.138+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:29.139608+0000) 2024-02-19T20:12:29.304 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:29.137+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:29.138899+0000) 2024-02-19T20:12:29.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:29.138+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:29.139432+0000) 2024-02-19T20:12:29.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:29.261+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:29.262845+0000) 2024-02-19T20:12:29.737 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:29.461+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:29.462204+0000) 2024-02-19T20:12:29.738 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:29.541+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:29.542833+0000) 2024-02-19T20:12:30.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:29.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:30.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:29.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:30.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:29.804+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:30.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:29.804+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:29.736+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:29.736+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:29.736+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:29.737+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:30.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:29.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:30.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:29.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:30.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:29.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:30.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:29.745+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:30.303 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:30.264+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:30.265274+0000) 2024-02-19T20:12:30.709 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:30.463+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:30.464292+0000) 2024-02-19T20:12:30.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:30.543+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:30.544933+0000) 2024-02-19T20:12:31.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:30.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:31.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:30.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:31.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:30.752+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:31.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:30.753+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:31.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:30.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:30.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:30.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:30.708+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:31.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:30.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:31.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:30.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:31.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:30.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:31.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:30.800+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:31.303 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:31.267+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:31.268782+0000) 2024-02-19T20:12:31.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:31.465+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:31.466315+0000) 2024-02-19T20:12:31.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:31.546+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:31.547144+0000) 2024-02-19T20:12:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:31.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:31.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:31.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:31.775+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:32.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:31.738+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:32.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:31.738+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:32.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:31.738+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:32.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:31.740+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:31.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:32.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:31.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:32.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:31.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:32.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:31.785+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:32.303 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:32.269+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:32.270289+0000) 2024-02-19T20:12:32.725 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:32.467+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:32.468116+0000) 2024-02-19T20:12:32.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:32.548+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:32.549544+0000) 2024-02-19T20:12:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:32.724+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:32.724+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:32.724+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:33.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:32.725+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:32.773+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:32.773+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:32.773+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:32.774+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:33.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:32.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:33.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:32.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:33.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:32.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:33.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:32.756+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:33.303 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:33.269+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:33.270460+0000) 2024-02-19T20:12:33.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:33.467+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:33.468299+0000) 2024-02-19T20:12:33.581 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:33.548+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:33.549725+0000) 2024-02-19T20:12:34.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:33.713+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:34.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:33.713+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:34.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:33.713+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:34.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:33.713+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:34.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:33.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:34.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:33.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:34.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:33.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:34.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:33.698+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:34.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:33.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:34.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:33.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:34.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:33.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:34.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:33.792+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:34.303 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:12:34.147+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54287 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:12:34.304 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:34.144+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:34.145268+0000) 2024-02-19T20:12:34.304 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:34.144+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:34.145633+0000) 2024-02-19T20:12:34.305 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:34.145+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:34.146079+0000) 2024-02-19T20:12:34.305 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:34.269+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:34.270618+0000) 2024-02-19T20:12:34.667 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:34.467+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:34.468479+0000) 2024-02-19T20:12:34.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:34.548+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:34.549905+0000) 2024-02-19T20:12:34.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:34.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:34.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:34.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:34.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:34.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:35.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:34.666+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:35.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:34.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:35.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:34.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:35.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:34.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:35.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:34.669+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:35.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:34.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:35.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:34.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:35.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:34.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:35.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:34.778+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:35.303 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:35.272+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:35.273023+0000) 2024-02-19T20:12:35.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:35.469+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:35.470575+0000) 2024-02-19T20:12:35.661 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:35.551+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:35.552276+0000) 2024-02-19T20:12:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:35.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:35.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:35.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:35.660+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:35.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:35.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:35.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:35.739+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:36.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:35.706+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:36.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:35.706+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:36.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:35.706+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:36.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:35.707+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:36.303 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:36.275+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:36.276205+0000) 2024-02-19T20:12:36.556 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:36.471+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:36.472885+0000) 2024-02-19T20:12:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:36.643+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:37.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:36.643+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:37.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:36.643+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:37.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:36.644+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:37.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:36.772+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:37.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:36.772+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:37.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:36.772+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:37.024 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:36.773+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:37.024 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:36.554+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:36.555871+0000) 2024-02-19T20:12:37.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:36.670+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:37.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:36.670+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:37.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:36.670+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:37.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:36.671+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:37.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:37.276+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:37.277603+0000) 2024-02-19T20:12:37.557 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:37.473+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:37.474743+0000) 2024-02-19T20:12:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:37.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:37.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:37.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:37.751+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:38.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:37.653+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:38.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:37.654+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:38.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:37.654+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:38.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:37.655+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:38.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:37.556+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:37.557421+0000) 2024-02-19T20:12:38.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:37.700+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:38.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:37.700+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:38.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:37.700+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:38.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:37.700+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:38.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:38.276+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:38.277780+0000) 2024-02-19T20:12:38.557 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:38.473+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:38.474918+0000) 2024-02-19T20:12:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:38.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:38.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:38.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:39.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:38.703+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:39.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:38.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:39.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:38.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:39.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:38.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:39.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:38.761+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:39.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:38.556+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:38.557612+0000) 2024-02-19T20:12:39.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:38.691+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:39.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:38.691+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:39.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:38.691+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:39.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:38.691+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:39.304 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:12:39.154+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54327 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:12:39.305 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:39.150+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:39.151317+0000) 2024-02-19T20:12:39.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:39.151+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:39.152257+0000) 2024-02-19T20:12:39.305 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:39.152+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:39.153046+0000) 2024-02-19T20:12:39.305 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:39.276+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:39.277963+0000) 2024-02-19T20:12:39.558 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:39.474+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:39.475101+0000) 2024-02-19T20:12:40.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:39.739+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:40.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:39.739+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:40.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:39.739+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:40.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:39.739+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:39.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:39.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:39.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:39.748+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:40.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:39.556+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:39.557772+0000) 2024-02-19T20:12:40.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:39.662+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:40.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:39.662+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:40.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:39.662+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:40.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:39.663+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:40.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:40.279+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:40.280581+0000) 2024-02-19T20:12:40.560 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:40.477+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:40.478634+0000) 2024-02-19T20:12:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:40.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:40.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:40.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:40.757+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:40.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:40.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:40.787+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:40.787+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:40.558+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:40.559956+0000) 2024-02-19T20:12:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:40.633+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:41.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:40.633+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:41.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:40.633+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:41.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:40.633+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:41.307 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:41.282+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:41.283131+0000) 2024-02-19T20:12:41.563 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:41.479+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:41.480906+0000) 2024-02-19T20:12:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:41.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:41.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:41.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:41.759+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:42.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:41.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:42.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:41.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:42.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:41.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:42.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:41.758+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:42.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:41.562+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:41.563411+0000) 2024-02-19T20:12:42.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:41.660+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:42.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:41.660+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:42.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:41.660+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:42.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:41.661+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:42.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:42.284+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:42.285513+0000) 2024-02-19T20:12:42.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:42.481+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:42.482656+0000) 2024-02-19T20:12:43.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:42.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:43.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:42.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:43.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:42.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:43.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:42.765+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:43.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:42.805+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:43.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:42.805+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:43.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:42.805+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:43.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:42.805+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:43.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:42.564+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:42.565882+0000) 2024-02-19T20:12:43.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:42.711+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:43.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:42.711+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:43.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:42.711+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:43.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:42.711+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:43.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:43.284+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:43.285707+0000) 2024-02-19T20:12:43.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:43.481+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:43.482858+0000) 2024-02-19T20:12:44.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:43.773+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:44.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:43.773+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:44.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:43.773+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:44.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:43.774+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:44.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:43.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:44.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:43.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:44.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:43.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:44.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:43.768+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:44.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:43.565+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:43.566062+0000) 2024-02-19T20:12:44.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:43.686+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:44.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:43.686+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:44.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:43.686+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:44.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:43.686+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:44.286 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:12:44.161+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54367 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:12:44.286 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:44.159+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:44.159994+0000) 2024-02-19T20:12:44.287 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:44.158+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:44.159604+0000) 2024-02-19T20:12:44.287 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:44.159+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:44.160208+0000) 2024-02-19T20:12:44.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:44.284+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:44.285905+0000) 2024-02-19T20:12:44.567 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:44.482+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:44.482998+0000) 2024-02-19T20:12:45.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:44.565+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:44.566240+0000) 2024-02-19T20:12:45.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:44.730+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:45.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:44.730+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:45.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:44.730+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:45.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:44.730+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:45.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:44.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:45.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:44.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:45.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:44.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:45.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:44.784+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:45.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:44.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:45.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:44.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:45.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:44.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:45.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:44.745+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:45.569 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:45.287+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:45.288430+0000) 2024-02-19T20:12:45.569 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:45.485+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:45.486156+0000) 2024-02-19T20:12:46.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:45.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:46.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:45.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:46.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:45.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:46.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:45.757+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:45.788+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:45.788+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:45.788+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:45.788+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:46.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:45.567+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:45.568390+0000) 2024-02-19T20:12:46.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:45.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:46.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:45.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:46.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:45.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:46.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:45.752+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:46.571 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:46.488+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:46.489206+0000) 2024-02-19T20:12:46.572 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:46.290+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:46.291233+0000) 2024-02-19T20:12:47.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:46.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:47.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:46.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:47.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:46.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:47.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:46.796+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:47.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:46.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:47.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:46.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:47.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:46.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:47.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:46.771+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:47.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:46.570+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:46.571140+0000) 2024-02-19T20:12:47.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:46.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:47.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:46.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:47.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:46.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:47.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:46.756+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:47.573 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:47.291+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:47.292914+0000) 2024-02-19T20:12:47.573 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:47.491+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:47.491039+0000) 2024-02-19T20:12:47.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:47.746+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:47.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:47.746+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:47.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:47.746+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:47.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:47.747+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:47.835 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:47.571+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:47.572697+0000) 2024-02-19T20:12:47.835 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:47.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:47.835 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:47.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:47.835 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:47.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:47.836 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:47.771+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:48.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:47.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:48.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:47.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:48.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:47.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:48.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:47.833+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:48.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:48.292+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:48.293090+0000) 2024-02-19T20:12:48.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:48.490+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:48.491184+0000) 2024-02-19T20:12:48.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:48.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:48.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:48.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:48.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:48.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:48.833 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:48.775+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:48.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:48.571+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:48.572946+0000) 2024-02-19T20:12:48.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:48.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:48.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:48.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:48.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:48.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:48.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:48.757+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:49.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:48.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:49.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:48.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:49.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:48.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:49.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:48.832+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:49.486 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:12:49.166+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54407 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:12:49.487 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:49.164+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:49.166323+0000) 2024-02-19T20:12:49.487 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:49.164+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:49.166425+0000) 2024-02-19T20:12:49.487 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:49.164+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:49.166799+0000) 2024-02-19T20:12:49.487 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:49.291+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:49.293287+0000) 2024-02-19T20:12:49.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:49.489+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:49.491352+0000) 2024-02-19T20:12:49.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:49.572+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:49.573144+0000) 2024-02-19T20:12:50.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:49.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:50.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:49.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:50.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:49.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:50.036 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:49.792+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:50.036 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:49.827+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:50.037 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:49.827+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:50.037 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:49.827+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:50.037 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:49.827+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:50.037 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:49.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:50.038 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:49.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:50.038 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:49.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:50.038 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:49.775+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:50.575 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:50.493+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:50.493986+0000) 2024-02-19T20:12:50.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:50.294+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:50.295552+0000) 2024-02-19T20:12:50.849 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:50.761+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:50.849 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:50.761+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:50.850 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:50.761+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:50.850 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:50.762+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:50.850 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:50.574+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:50.575128+0000) 2024-02-19T20:12:50.850 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:50.755+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:50.850 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:50.755+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:50.850 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:50.755+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:50.851 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:50.756+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:51.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:50.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:51.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:50.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:51.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:50.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:51.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:50.849+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:51.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:51.495+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:51.496329+0000) 2024-02-19T20:12:51.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:51.296+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:51.297972+0000) 2024-02-19T20:12:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:51.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:51.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:51.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:51.783+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:52.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:51.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:52.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:51.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:52.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:51.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:52.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:51.805+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:52.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:51.576+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:51.577527+0000) 2024-02-19T20:12:52.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:51.786+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:52.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:51.786+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:52.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:51.786+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:52.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:51.787+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:52.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:52.298+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:52.299423+0000) 2024-02-19T20:12:52.579 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:52.497+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:52.498020+0000) 2024-02-19T20:12:52.835 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:52.578+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:52.579174+0000) 2024-02-19T20:12:52.835 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:52.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:52.835 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:52.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:52.835 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:52.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:52.836 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:52.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:52.836 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:52.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:52.836 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:52.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:52.836 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:52.787+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:52.836 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:52.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:52.836 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:52.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:52.837 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:52.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:52.837 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:52.770+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:53.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:52.834+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:53.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:53.497+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:53.498210+0000) 2024-02-19T20:12:53.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:53.298+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:53.299662+0000) 2024-02-19T20:12:54.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:53.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:54.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:53.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:54.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:53.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:54.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:53.757+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:54.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:53.721+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:54.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:53.721+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:54.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:53.721+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:54.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:53.722+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:54.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:53.578+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:53.579353+0000) 2024-02-19T20:12:54.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:53.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:54.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:53.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:54.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:53.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:54.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:53.814+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:54.293 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:12:54.174+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54447 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:12:54.294 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:54.172+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:54.173618+0000) 2024-02-19T20:12:54.294 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:54.171+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:54.172182+0000) 2024-02-19T20:12:54.294 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:54.171+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:54.172894+0000) 2024-02-19T20:12:54.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:54.298+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:54.299827+0000) 2024-02-19T20:12:54.580 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:54.497+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:54.498389+0000) 2024-02-19T20:12:55.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:54.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:55.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:54.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:55.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:54.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:55.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:54.733+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:54.710+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:54.710+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:54.710+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:55.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:54.711+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:55.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:54.578+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:54.579532+0000) 2024-02-19T20:12:55.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:54.826+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:55.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:54.826+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:55.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:54.826+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:55.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:54.826+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:55.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:55.302+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:55.303265+0000) 2024-02-19T20:12:55.583 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:55.499+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:55.500487+0000) 2024-02-19T20:12:55.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:55.581+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:55.582740+0000) 2024-02-19T20:12:55.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:55.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:55.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:55.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:55.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:55.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:55.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:55.718+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:55.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:55.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:55.862 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:55.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:55.862 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:55.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:55.862 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:55.669+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:56.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:55.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:56.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:55.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:56.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:55.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:56.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:55.859+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:56.540 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:56.305+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:56.306432+0000) 2024-02-19T20:12:56.540 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:56.501+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:56.502595+0000) 2024-02-19T20:12:56.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:56.583+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:56.584920+0000) 2024-02-19T20:12:56.860 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:56.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:56.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:56.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:56.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:56.707+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:56.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:56.708+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:56.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:56.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:56.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:56.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:56.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:56.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:56.862 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:56.749+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:57.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:56.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:57.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:56.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:57.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:56.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:57.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:56.859+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:57.587 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:57.307+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:57.308849+0000) 2024-02-19T20:12:57.587 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:57.503+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:57.504022+0000) 2024-02-19T20:12:57.908 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:57.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:57.908 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:57.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:57.909 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:57.760+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:57.909 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:57.761+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:57.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:57.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:57.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:57.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:57.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:57.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:57.909 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:57.664+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:57.910 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:57.585+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:57.586452+0000) 2024-02-19T20:12:57.910 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:57.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:57.910 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:57.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:57.910 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:57.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:58.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:57.907+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:58.587 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:58.308+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:58.309024+0000) 2024-02-19T20:12:58.587 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:58.503+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:58.504186+0000) 2024-02-19T20:12:58.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:58.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:58.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:58.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:58.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:58.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:58.875 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:58.696+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:58.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:58.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:58.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:58.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:58.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:58.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:58.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:58.745+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:12:58.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:58.585+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:58.586636+0000) 2024-02-19T20:12:58.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:58.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:58.879 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:58.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:58.879 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:58.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:12:59.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:58.873+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:59.487 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:12:59.181+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54487 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:12:59.487 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:59.179+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:59.180354+0000) 2024-02-19T20:12:59.488 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:59.178+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:59.179716+0000) 2024-02-19T20:12:59.488 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:59.179+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:59.180165+0000) 2024-02-19T20:12:59.488 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:59.308+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:59.309197+0000) 2024-02-19T20:12:59.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:59.585+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:59.586799+0000) 2024-02-19T20:12:59.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:59.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:59.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:59.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:59.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:59.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:12:59.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:12:59.658+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:12:59.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:59.503+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:59.504382+0000) 2024-02-19T20:12:59.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:59.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:59.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:59.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:59.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:59.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:12:59.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:12:59.733+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:00.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:59.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:00.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:59.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:00.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:59.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:00.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:12:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:12:59.841+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:00.590 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:00.505+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:00.506486+0000) 2024-02-19T20:13:00.590 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:00.310+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:00.311251+0000) 2024-02-19T20:13:00.879 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:00.589+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:00.590030+0000) 2024-02-19T20:13:00.879 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:00.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:00.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:00.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:00.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:00.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:00.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:00.742+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:00.880 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:00.657+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:00.880 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:00.657+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:00.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:00.657+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:00.881 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:00.657+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:01.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:00.878+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:01.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:00.878+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:01.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:00.878+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:01.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:00.878+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:01.591 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:01.312+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:01.313954+0000) 2024-02-19T20:13:01.592 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:01.507+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:01.508479+0000) 2024-02-19T20:13:01.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:01.729+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:01.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:01.729+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:01.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:01.729+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:01.866 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:01.730+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:01.866 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:01.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:01.866 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:01.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:01.866 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:01.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:01.866 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:01.617+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:01.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:01.591+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:01.592452+0000) 2024-02-19T20:13:01.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:01.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:01.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:01.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:01.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:01.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:02.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:01.864+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:02.595 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:02.314+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:02.315574+0000) 2024-02-19T20:13:02.595 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:02.509+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:02.510769+0000) 2024-02-19T20:13:02.862 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:02.593+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:02.594699+0000) 2024-02-19T20:13:02.863 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:02.647+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:02.863 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:02.647+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:02.863 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:02.647+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:02.863 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:02.648+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:02.863 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:02.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:02.863 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:02.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:02.864 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:02.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:02.864 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:02.749+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:03.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:02.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:03.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:02.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:03.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:02.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:03.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:02.861+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:03.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:03.314+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:03.315776+0000) 2024-02-19T20:13:03.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:03.510+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:03.510994+0000) 2024-02-19T20:13:03.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:03.671+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:03.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:03.671+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:03.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:03.671+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:03.858 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:03.671+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:03.859 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:03.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:03.859 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:03.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:03.859 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:03.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:03.859 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:03.779+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:03.859 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:03.593+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:03.594918+0000) 2024-02-19T20:13:03.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:03.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:03.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:03.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:03.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:03.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:04.184 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:03.857+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:04.487 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:13:04.188+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54527 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:13:04.488 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:04.185+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:04.186774+0000) 2024-02-19T20:13:04.488 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:04.184+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:04.185898+0000) 2024-02-19T20:13:04.488 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:04.314+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:04.315953+0000) 2024-02-19T20:13:04.489 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:04.185+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:04.186229+0000) 2024-02-19T20:13:04.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:04.510+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:04.511151+0000) 2024-02-19T20:13:04.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:04.594+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:04.595101+0000) 2024-02-19T20:13:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:04.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:04.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:04.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:04.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:04.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:04.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:04.719+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:05.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:04.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:05.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:04.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:05.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:04.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:05.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:04.817+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:05.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:04.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:05.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:04.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:05.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:04.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:05.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:04.895+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:05.487 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:05.317+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:05.318602+0000) 2024-02-19T20:13:05.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:05.513+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:05.514482+0000) 2024-02-19T20:13:05.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:05.596+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:05.597757+0000) 2024-02-19T20:13:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:05.722+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:05.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:05.722+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:05.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:05.722+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:05.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:05.723+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:05.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:05.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:05.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:05.820+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:06.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:05.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:06.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:05.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:05.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:05.919+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:06.601 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:06.321+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:06.321996+0000) 2024-02-19T20:13:06.602 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:06.516+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:06.517867+0000) 2024-02-19T20:13:06.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:06.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:06.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:06.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:06.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:06.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:06.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:06.757+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:06.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:06.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:06.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:06.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:06.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:06.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:06.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:06.832+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:06.945 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:06.600+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:06.601043+0000) 2024-02-19T20:13:06.945 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:06.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:06.945 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:06.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:06.945 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:06.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:06.942+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:07.603 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:07.322+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:07.323840+0000) 2024-02-19T20:13:07.603 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:07.518+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:07.519502+0000) 2024-02-19T20:13:07.877 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:07.601+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:07.602969+0000) 2024-02-19T20:13:07.877 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:07.733+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:07.877 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:07.733+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:07.877 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:07.733+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:07.878 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:07.734+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:07.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:07.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:07.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:08.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:07.876+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:08.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:07.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:08.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:07.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:08.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:07.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:08.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:07.960+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:08.603 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:08.323+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:08.323995+0000) 2024-02-19T20:13:08.604 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:08.518+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:08.519706+0000) 2024-02-19T20:13:08.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:08.602+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:08.603149+0000) 2024-02-19T20:13:08.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:08.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:08.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:08.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:08.937 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:08.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:08.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:08.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:08.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:08.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:08.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:08.848+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:08.938 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:08.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:08.938 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:08.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:08.938 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:08.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:08.938 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:08.742+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:09.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:08.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:09.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:08.936+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:09.488 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:13:09.195+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54567 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:13:09.488 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:09.192+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:09.193936+0000) 2024-02-19T20:13:09.489 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:09.193+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:09.194509+0000) 2024-02-19T20:13:09.489 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:09.193+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:09.194780+0000) 2024-02-19T20:13:09.489 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:09.323+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:09.324157+0000) 2024-02-19T20:13:09.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:09.602+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:09.603305+0000) 2024-02-19T20:13:09.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:09.518+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:09.519873+0000) 2024-02-19T20:13:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:09.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:09.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:09.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:09.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:09.719+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:10.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:09.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:10.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:09.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:10.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:09.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:10.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:09.963+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:09.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:09.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:09.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:09.809+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:10.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:10.325+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:10.326737+0000) 2024-02-19T20:13:10.605 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:10.522+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:10.523074+0000) 2024-02-19T20:13:10.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:10.737+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:10.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:10.737+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:10.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:10.737+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:10.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:10.738+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:10.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:10.792+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:10.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:10.792+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:10.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:10.792+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:10.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:10.793+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:10.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:10.604+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:10.605287+0000) 2024-02-19T20:13:10.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:10.923+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:10.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:10.923+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:10.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:10.923+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:11.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:10.923+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:11.607 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:11.327+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:11.328781+0000) 2024-02-19T20:13:11.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:11.525+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:11.526388+0000) 2024-02-19T20:13:11.952 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:11.756+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:11.952 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:11.756+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:11.952 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:11.756+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:11.953 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:11.757+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:11.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:11.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:11.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:11.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:11.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:11.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:11.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:11.760+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:11.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:11.606+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:11.607489+0000) 2024-02-19T20:13:11.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:11.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:11.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:11.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:11.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:11.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:12.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:11.951+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:12.609 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:12.330+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:12.331272+0000) 2024-02-19T20:13:12.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:12.526+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:12.527844+0000) 2024-02-19T20:13:12.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:12.608+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:12.609511+0000) 2024-02-19T20:13:12.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:12.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:12.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:12.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:12.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:12.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:12.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:12.766+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:12.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:12.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:12.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:12.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:12.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:12.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:12.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:12.792+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:13.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:12.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:12.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:12.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:13.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:12.985+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:13.580 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:13.527+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:13.528014+0000) 2024-02-19T20:13:13.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:13.330+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:13.331426+0000) 2024-02-19T20:13:13.995 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:13.608+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:13.609704+0000) 2024-02-19T20:13:13.995 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:13.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:13.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:13.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:13.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:13.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:13.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:13.755+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:13.996 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:13.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:13.996 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:13.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:13.996 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:13.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:13.997 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:13.781+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:14.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:13:14.202+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54607 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:13:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:14.200+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:14.201502+0000) 2024-02-19T20:13:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:14.200+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:14.201130+0000) 2024-02-19T20:13:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:13.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:13.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:13.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:13.993+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:14.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:14.200+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:14.201972+0000) 2024-02-19T20:13:14.610 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:14.330+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:14.331586+0000) 2024-02-19T20:13:14.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:14.527+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:14.528191+0000) 2024-02-19T20:13:15.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:14.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:15.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:14.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:15.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:14.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:15.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:14.803+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:15.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:14.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:15.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:14.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:15.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:14.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:15.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:14.762+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:15.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:14.608+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:14.609901+0000) 2024-02-19T20:13:15.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:15.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:15.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:15.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:15.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:15.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:15.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:15.010+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:15.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:15.332+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:15.333468+0000) 2024-02-19T20:13:15.612 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:15.529+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:15.530360+0000) 2024-02-19T20:13:16.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:15.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:16.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:15.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:16.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:15.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:16.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:15.814+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:16.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:15.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:16.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:15.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:16.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:15.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:16.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:15.715+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:16.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:15.610+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:15.611957+0000) 2024-02-19T20:13:16.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:16.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:16.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:16.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:16.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:16.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:16.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:16.016+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:16.532 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:16.336+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:16.337028+0000) 2024-02-19T20:13:16.859 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:16.613+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:16.614391+0000) 2024-02-19T20:13:16.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:16.531+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:16.532523+0000) 2024-02-19T20:13:16.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:16.681+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:16.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:16.681+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:16.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:16.681+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:16.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:16.682+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:17.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:16.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:16.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:16.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:17.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:16.859+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:17.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:16.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:17.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:16.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:16.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:17.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:16.997+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:17.534 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:17.337+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:17.338835+0000) 2024-02-19T20:13:17.844 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:17.615+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:17.616424+0000) 2024-02-19T20:13:17.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:17.533+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:17.534039+0000) 2024-02-19T20:13:17.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:17.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:17.845 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:17.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:17.845 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:17.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:17.845 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:17.658+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:18.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:17.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:17.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:17.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:18.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:17.843+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:18.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:17.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:18.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:17.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:18.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:17.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:18.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:17.963+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:18.534 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:18.338+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:18.339004+0000) 2024-02-19T20:13:18.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:18.615+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:18.616627+0000) 2024-02-19T20:13:18.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:18.533+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:18.534185+0000) 2024-02-19T20:13:18.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:18.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:18.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:18.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:18.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:18.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:18.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:18.692+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:19.204 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:18.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:19.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:18.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:19.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:18.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:19.205 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:18.869+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:19.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:18.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:19.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:18.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:19.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:18.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:19.206 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:18.993+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:19.489 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:13:19.209+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54648 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:13:19.489 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:19.206+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:19.207710+0000) 2024-02-19T20:13:19.490 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:19.206+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:19.207728+0000) 2024-02-19T20:13:19.490 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:19.206+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:19.207746+0000) 2024-02-19T20:13:19.490 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:19.338+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:19.339175+0000) 2024-02-19T20:13:19.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:19.615+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:19.616821+0000) 2024-02-19T20:13:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:19.533+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:19.534406+0000) 2024-02-19T20:13:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:19.721+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:19.721+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:19.721+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:19.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:19.722+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:20.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:19.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:20.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:19.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:20.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:19.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:20.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:19.849+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:20.011+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:20.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:20.011+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:20.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:20.011+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:20.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:20.012+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:20.536 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:20.341+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:20.342690+0000) 2024-02-19T20:13:20.889 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:20.618+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:20.618986+0000) 2024-02-19T20:13:20.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:20.535+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:20.536411+0000) 2024-02-19T20:13:20.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:20.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:20.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:20.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:20.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:20.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:20.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:20.753+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:21.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:21.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:21.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:21.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:21.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:21.025+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:21.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:20.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:21.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:20.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:20.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:21.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:20.888+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:21.539 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:21.344+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:21.345873+0000) 2024-02-19T20:13:21.796 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:21.620+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:21.621533+0000) 2024-02-19T20:13:21.797 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:21.537+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:21.538916+0000) 2024-02-19T20:13:21.797 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:21.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:21.797 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:21.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:21.797 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:21.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:21.797 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:21.764+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:22.048 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:21.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:22.048 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:21.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:22.048 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:21.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:22.048 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:21.897+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:22.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:22.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:22.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:22.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:22.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:22.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:22.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:22.047+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:22.624 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:22.346+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:22.347324+0000) 2024-02-19T20:13:22.624 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:22.539+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:22.540771+0000) 2024-02-19T20:13:22.902 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:22.623+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:22.624049+0000) 2024-02-19T20:13:22.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:22.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:22.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:22.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:22.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:22.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:22.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:22.814+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:22.901+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:22.901+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:22.901+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:22.902+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:23.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:23.040+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:23.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:23.040+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:23.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:23.040+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:23.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:23.040+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:23.541 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:23.346+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:23.347522+0000) 2024-02-19T20:13:23.795 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:23.623+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:23.624206+0000) 2024-02-19T20:13:23.795 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:23.540+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:23.540965+0000) 2024-02-19T20:13:23.795 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:23.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:23.795 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:23.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:23.795 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:23.793+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:24.048 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:23.794+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:24.049 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:23.880+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:24.049 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:23.880+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:24.049 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:23.880+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:24.049 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:23.881+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:24.309 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:13:24.215+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54688 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:13:24.310 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:24.214+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:24.215616+0000) 2024-02-19T20:13:24.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:24.213+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:24.214583+0000) 2024-02-19T20:13:24.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:24.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:24.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:24.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:24.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:24.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:24.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:24.071+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:24.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:24.213+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:24.214267+0000) 2024-02-19T20:13:24.624 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:24.346+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:24.347675+0000) 2024-02-19T20:13:24.625 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:24.540+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:24.541157+0000) 2024-02-19T20:13:24.887 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:24.623+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:24.624406+0000) 2024-02-19T20:13:24.888 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:24.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:24.888 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:24.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:24.888 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:24.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:24.888 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:24.814+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:25.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:24.886+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:25.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:24.886+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:25.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:24.886+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:25.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:24.887+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:25.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:25.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:25.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:25.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:25.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:25.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:25.103+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:25.544 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:25.350+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:25.351020+0000) 2024-02-19T20:13:25.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:25.625+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:25.626941+0000) 2024-02-19T20:13:25.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:25.543+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:25.544658+0000) 2024-02-19T20:13:25.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:25.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:25.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:25.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:25.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:25.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:25.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:25.777+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:26.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:25.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:26.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:25.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:26.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:25.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:26.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:25.917+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:26.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:26.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:26.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:26.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:26.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:26.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:26.273 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:26.076+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:26.547 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:26.352+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:26.353046+0000) 2024-02-19T20:13:26.842 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:26.628+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:26.629483+0000) 2024-02-19T20:13:26.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:26.545+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:26.546865+0000) 2024-02-19T20:13:26.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:26.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:26.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:26.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:26.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:26.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:26.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:26.734+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:26.955+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:26.955+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:26.955+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:26.955+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:27.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:27.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:27.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:27.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:27.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:27.029+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:27.549 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:27.353+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:27.354463+0000) 2024-02-19T20:13:27.934 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:27.631+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:27.632030+0000) 2024-02-19T20:13:27.934 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:27.547+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:27.548667+0000) 2024-02-19T20:13:27.934 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:27.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:27.934 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:27.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:27.934 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:27.706+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:27.934 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:27.707+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:27.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:27.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:27.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:28.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:27.988+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:28.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:27.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:27.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:27.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:28.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:27.933+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:28.548 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:28.353+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:28.354639+0000) 2024-02-19T20:13:28.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:28.631+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:28.632220+0000) 2024-02-19T20:13:28.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:28.547+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:28.548856+0000) 2024-02-19T20:13:28.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:28.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:28.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:28.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:28.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:28.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:28.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:28.720+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:29.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:28.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:29.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:28.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:29.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:28.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:29.219 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:28.904+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:29.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:28.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:29.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:28.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:29.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:28.976+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:29.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:28.977+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:29.490 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:13:29.222+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54728 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:13:29.490 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:29.220+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:29.221021+0000) 2024-02-19T20:13:29.491 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:29.221+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:29.222460+0000) 2024-02-19T20:13:29.491 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:29.353+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:29.354837+0000) 2024-02-19T20:13:29.491 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:29.219+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:29.220703+0000) 2024-02-19T20:13:29.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:29.548+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:29.549022+0000) 2024-02-19T20:13:29.741 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:29.631+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:29.632385+0000) 2024-02-19T20:13:30.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:29.929+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:30.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:29.929+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:30.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:29.929+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:30.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:29.929+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:29.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:29.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:29.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:29.916+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:30.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:29.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:30.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:29.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:30.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:29.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:30.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:29.740+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:30.551 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:30.356+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:30.357418+0000) 2024-02-19T20:13:30.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:30.634+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:30.635027+0000) 2024-02-19T20:13:30.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:30.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:30.923 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:30.550+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:30.551057+0000) 2024-02-19T20:13:30.923 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:30.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:30.923 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:30.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:30.923 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:30.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:30.923 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:30.710+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:30.940+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:30.940+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:30.940+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:30.941+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:31.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:30.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:30.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:31.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:30.921+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:31.553 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:31.358+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:31.359511+0000) 2024-02-19T20:13:31.888 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:31.636+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:31.637308+0000) 2024-02-19T20:13:31.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:31.552+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:31.553768+0000) 2024-02-19T20:13:31.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:31.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:31.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:31.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:31.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:31.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:31.889 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:31.715+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:31.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:31.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:31.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:31.945+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:32.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:31.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:32.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:31.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:32.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:31.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:32.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:31.901+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:32.555 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:32.361+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:32.362024+0000) 2024-02-19T20:13:32.902 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:32.638+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:32.639925+0000) 2024-02-19T20:13:32.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:32.554+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:32.555720+0000) 2024-02-19T20:13:32.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:32.703+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:32.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:32.703+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:32.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:32.703+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:32.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:32.704+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:33.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:32.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:33.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:32.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:33.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:32.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:33.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:32.901+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:32.994+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:32.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:32.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:32.995+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:33.556 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:33.361+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:33.362170+0000) 2024-02-19T20:13:33.859 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:33.554+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:33.555917+0000) 2024-02-19T20:13:33.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:33.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:33.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:33.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:33.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:33.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:33.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:33.712+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:33.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:33.639+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:33.640107+0000) 2024-02-19T20:13:33.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:33.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:33.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:33.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:34.225 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:33.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:34.225 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:33.858+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:34.225 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:33.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:34.226 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:33.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:34.226 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:33.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:34.226 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:33.971+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:34.491 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:13:34.229+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54768 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:13:34.491 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:34.227+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:34.228658+0000) 2024-02-19T20:13:34.492 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:34.227+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:34.228169+0000) 2024-02-19T20:13:34.492 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:34.226+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:34.227882+0000) 2024-02-19T20:13:34.492 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:34.361+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:34.362313+0000) 2024-02-19T20:13:34.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:34.555+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:34.556113+0000) 2024-02-19T20:13:34.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:34.639+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:34.640277+0000) 2024-02-19T20:13:35.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:34.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:34.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:34.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:34.759+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:35.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:34.988+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:35.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:34.988+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:35.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:34.988+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:35.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:34.989+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:35.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:34.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:35.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:34.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:35.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:34.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:35.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:34.814+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:35.558 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:35.363+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:35.364371+0000) 2024-02-19T20:13:35.859 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:35.557+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:35.558064+0000) 2024-02-19T20:13:35.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:35.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:35.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:35.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:35.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:35.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:35.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:35.715+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:35.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:35.641+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:35.642382+0000) 2024-02-19T20:13:35.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:35.786+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:35.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:35.786+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:35.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:35.786+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:35.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:35.786+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:36.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:36.014+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:36.014+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:36.014+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:36.015+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:36.560 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:36.366+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:36.367893+0000) 2024-02-19T20:13:36.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:36.643+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:36.644471+0000) 2024-02-19T20:13:36.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:36.769+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:36.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:36.769+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:36.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:36.769+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:36.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:36.769+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:36.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:36.559+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:36.560205+0000) 2024-02-19T20:13:36.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:36.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:36.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:36.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:36.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:36.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:36.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:36.690+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:37.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:37.065+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:37.065+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:37.065+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:37.066+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:37.562 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:37.368+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:37.369362+0000) 2024-02-19T20:13:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:37.561+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:37.561968+0000) 2024-02-19T20:13:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:37.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:37.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:37.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:37.691+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:38.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:37.645+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:37.646026+0000) 2024-02-19T20:13:38.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:37.768+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:38.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:37.769+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:38.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:37.769+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:38.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:37.769+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:38.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:38.099+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:38.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:38.099+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:38.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:38.099+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:38.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:38.099+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:38.562 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:38.367+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:38.369526+0000) 2024-02-19T20:13:38.814 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:38.645+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:38.646208+0000) 2024-02-19T20:13:38.814 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:38.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:38.814 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:38.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:38.814 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:38.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:38.815 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:38.560+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:38.562702+0000) 2024-02-19T20:13:38.815 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:38.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:38.815 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:38.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:38.815 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:38.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:38.815 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:38.685+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:39.068 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:38.813+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:39.321 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:13:39.235+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54809 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:13:39.322 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:39.233+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:39.234779+0000) 2024-02-19T20:13:39.322 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:39.233+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:39.234705+0000) 2024-02-19T20:13:39.322 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:39.067+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:39.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:39.067+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:39.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:39.067+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:39.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:39.068+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:39.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:39.233+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:39.234839+0000) 2024-02-19T20:13:39.639 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:39.561+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:39.562958+0000) 2024-02-19T20:13:39.639 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:39.368+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:39.369727+0000) 2024-02-19T20:13:40.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:39.638+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:40.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:39.638+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:40.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:39.638+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:40.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:39.639+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:40.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:39.645+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:39.646438+0000) 2024-02-19T20:13:40.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:39.790+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:40.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:39.790+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:40.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:39.790+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:40.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:39.790+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:40.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:40.032+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:40.300 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:40.032+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:40.300 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:40.032+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:40.300 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:40.033+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:40.565 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:40.371+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:40.372200+0000) 2024-02-19T20:13:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:40.564+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:40.565351+0000) 2024-02-19T20:13:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:40.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:40.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:40.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:40.674+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:40.649+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:40.650037+0000) 2024-02-19T20:13:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:40.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:40.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:40.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:41.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:40.758+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:41.300 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:41.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:41.300 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:41.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:41.300 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:41.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:41.300 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:41.057+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:41.569 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:41.373+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:41.374615+0000) 2024-02-19T20:13:41.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:41.652+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:41.653624+0000) 2024-02-19T20:13:41.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:41.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:41.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:41.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:41.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:41.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:41.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:41.752+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:41.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:41.567+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:41.568864+0000) 2024-02-19T20:13:41.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:41.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:41.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:41.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:41.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:41.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:41.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:41.697+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:42.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:42.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:42.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:42.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:42.009+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:42.571 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:42.375+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:42.376314+0000) 2024-02-19T20:13:43.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:42.654+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:42.655296+0000) 2024-02-19T20:13:43.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:42.790+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:43.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:42.790+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:43.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:42.790+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:43.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:42.791+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:43.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:42.569+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:42.570900+0000) 2024-02-19T20:13:43.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:42.703+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:43.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:42.704+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:43.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:42.704+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:43.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:42.705+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:43.300 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:43.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:43.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:43.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:43.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:43.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:43.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:43.039+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:43.571 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:43.375+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:43.376532+0000) 2024-02-19T20:13:44.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:43.570+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:43.571183+0000) 2024-02-19T20:13:44.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:43.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:44.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:43.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:44.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:43.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:44.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:43.706+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:44.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:43.654+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:43.655593+0000) 2024-02-19T20:13:44.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:43.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:44.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:43.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:44.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:43.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:44.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:43.764+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:44.300 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:13:44.242+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54849 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:13:44.301 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:44.243+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:44.244793+0000) 2024-02-19T20:13:44.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:44.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:44.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:44.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:44.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:44.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:44.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:44.055+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:44.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:44.240+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:44.241879+0000) 2024-02-19T20:13:44.302 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:44.240+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:44.241030+0000) 2024-02-19T20:13:44.571 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:44.375+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:44.376698+0000) 2024-02-19T20:13:45.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:44.570+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:44.571359+0000) 2024-02-19T20:13:45.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:44.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:45.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:44.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:45.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:44.725+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:45.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:44.725+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:45.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:44.654+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:44.655776+0000) 2024-02-19T20:13:45.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:44.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:45.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:44.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:45.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:44.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:45.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:44.787+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:45.303 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:45.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:45.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:45.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:45.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:45.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:45.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:45.082+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:45.574 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:45.378+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:45.379235+0000) 2024-02-19T20:13:45.830 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:45.572+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:45.573773+0000) 2024-02-19T20:13:45.830 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:45.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:45.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:45.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:45.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:45.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:45.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:45.762+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:45.831 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:45.656+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:45.657916+0000) 2024-02-19T20:13:45.831 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:45.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:45.831 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:45.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:45.831 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:45.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:46.087 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:45.829+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:46.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:46.086+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:46.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:46.086+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:46.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:46.086+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:46.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:46.087+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:46.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:46.575+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:46.576179+0000) 2024-02-19T20:13:46.660 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:46.380+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:46.381632+0000) 2024-02-19T20:13:47.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:46.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:47.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:46.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:47.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:46.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:47.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:46.765+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:47.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:46.659+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:46.660122+0000) 2024-02-19T20:13:47.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:46.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:47.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:46.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:47.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:46.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:47.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:46.809+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:47.303 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:47.094+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:47.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:47.094+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:47.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:47.094+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:47.304 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:47.094+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:47.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:47.383+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:47.384239+0000) 2024-02-19T20:13:47.844 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:47.661+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:47.662781+0000) 2024-02-19T20:13:47.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:47.577+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:47.577999+0000) 2024-02-19T20:13:47.845 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:47.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:47.845 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:47.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:47.845 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:47.758+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:47.845 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:47.759+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:48.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:48.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:48.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:48.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:48.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:48.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:48.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:48.072+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:48.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:47.843+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:48.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:47.843+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:48.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:47.843+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:48.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:47.843+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:48.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:48.383+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:48.384393+0000) 2024-02-19T20:13:48.830 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:48.661+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:48.662952+0000) 2024-02-19T20:13:48.831 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:48.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:48.831 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:48.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:48.831 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:48.829+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:48.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:48.577+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:48.578176+0000) 2024-02-19T20:13:48.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:48.788+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:48.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:48.788+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:48.831 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:48.788+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:48.832 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:48.789+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:49.244 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:48.829+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:49.245 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:49.062+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:49.245 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:49.062+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:49.245 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:49.062+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:49.245 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:49.063+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:49.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:13:49.249+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54889 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:13:49.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:49.247+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:49.248741+0000) 2024-02-19T20:13:49.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:49.246+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:49.247385+0000) 2024-02-19T20:13:49.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:49.247+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:49.248473+0000) 2024-02-19T20:13:49.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:49.383+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:49.384564+0000) 2024-02-19T20:13:49.837 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:49.662+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:49.663122+0000) 2024-02-19T20:13:49.837 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:49.577+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:49.578379+0000) 2024-02-19T20:13:49.838 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:49.836+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:49.838 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:49.836+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:49.838 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:49.836+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:50.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:49.837+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:50.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:50.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:50.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:50.023+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:50.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:49.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:50.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:49.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:50.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:49.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:50.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:49.845+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:50.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:50.385+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:50.386609+0000) 2024-02-19T20:13:50.878 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:50.579+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:50.580335+0000) 2024-02-19T20:13:50.879 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:50.664+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:50.665201+0000) 2024-02-19T20:13:51.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:50.887+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:50.887+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:50.887+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:51.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:50.888+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:50.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:51.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:50.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:51.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:50.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:51.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:50.981+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:50.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:50.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:50.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:50.878+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:51.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:51.387+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:51.388783+0000) 2024-02-19T20:13:51.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:51.581+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:51.582855+0000) 2024-02-19T20:13:51.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:51.666+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:51.667304+0000) 2024-02-19T20:13:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:51.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:51.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:51.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:51.935+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:51.986+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:52.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:51.986+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:52.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:51.986+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:52.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:51.986+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:51.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:51.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:51.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:51.872+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:52.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:52.389+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:52.390390+0000) 2024-02-19T20:13:52.856 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:52.668+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:52.669275+0000) 2024-02-19T20:13:52.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:52.583+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:52.584604+0000) 2024-02-19T20:13:52.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:52.855+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:52.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:52.855+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:52.855+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:52.856+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:53.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:52.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:53.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:52.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:53.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:52.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:53.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:52.957+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:53.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:52.890+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:53.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:52.890+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:53.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:52.890+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:53.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:52.891+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:53.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:53.389+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:53.390557+0000) 2024-02-19T20:13:53.889 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:53.668+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:53.669436+0000) 2024-02-19T20:13:53.889 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:53.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:53.889 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:53.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:53.890 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:53.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:53.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:53.583+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:53.584805+0000) 2024-02-19T20:13:53.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:53.823+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:53.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:53.823+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:53.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:53.823+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:53.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:53.824+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:53.950 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:53.888+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:53.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:53.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:53.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:53.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:53.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:53.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:53.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:53.948+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:54.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:13:54.257+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54929 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:13:54.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:54.254+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:54.255313+0000) 2024-02-19T20:13:54.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:54.254+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:54.255347+0000) 2024-02-19T20:13:54.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:54.253+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:54.254933+0000) 2024-02-19T20:13:54.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:54.389+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:54.390727+0000) 2024-02-19T20:13:54.854 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:54.584+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:54.585007+0000) 2024-02-19T20:13:54.854 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:54.668+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:54.669613+0000) 2024-02-19T20:13:55.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:54.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:55.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:54.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:55.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:54.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:55.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:54.900+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:55.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:54.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:55.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:54.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:55.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:54.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:55.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:54.871+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:54.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:55.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:54.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:55.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:54.853+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:55.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:54.853+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:55.587 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:55.393+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:55.394254+0000) 2024-02-19T20:13:55.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:55.670+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:55.671772+0000) 2024-02-19T20:13:55.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:55.586+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:55.586999+0000) 2024-02-19T20:13:55.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:55.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:56.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:55.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:55.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:55.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:56.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:55.915+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:56.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:55.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:55.872+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:56.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:55.873+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:55.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:55.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:55.913+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:56.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:55.914+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:56.589 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:56.396+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:56.397504+0000) 2024-02-19T20:13:56.855 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:56.587+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:56.588953+0000) 2024-02-19T20:13:56.856 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:56.673+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:56.674018+0000) 2024-02-19T20:13:57.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:56.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:57.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:56.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:57.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:56.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:57.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:56.855+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:57.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:56.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:57.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:56.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:57.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:56.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:57.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:56.899+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:57.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:56.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:57.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:56.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:57.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:56.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:57.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:56.920+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:57.493 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:57.398+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:57.399209+0000) 2024-02-19T20:13:57.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:57.589+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:57.590834+0000) 2024-02-19T20:13:57.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:57.674+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:57.675665+0000) 2024-02-19T20:13:58.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:57.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:58.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:57.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:58.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:57.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:58.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:57.958+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:58.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:57.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:58.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:57.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:58.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:57.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:58.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:57.880+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:58.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:57.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:58.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:57.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:58.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:57.807+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:58.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:57.808+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:58.591 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:58.398+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:58.399397+0000) 2024-02-19T20:13:58.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:58.674+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:58.675841+0000) 2024-02-19T20:13:58.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:58.590+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:58.590999+0000) 2024-02-19T20:13:58.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:58.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:58.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:58.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:58.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:58.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:58.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:58.786+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:13:59.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:58.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:59.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:58.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:59.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:58.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:59.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:58.847+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:59.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:58.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:59.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:58.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:59.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:58.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:59.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:58.920+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:59.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:13:59.264+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 54969 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:13:59.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:59.261+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:59.262233+0000) 2024-02-19T20:13:59.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:59.398+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:59.399575+0000) 2024-02-19T20:13:59.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:59.261+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:59.262428+0000) 2024-02-19T20:13:59.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:59.263+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:59.264174+0000) 2024-02-19T20:13:59.896 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:59.675+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:59.676028+0000) 2024-02-19T20:13:59.896 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:59.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:59.897 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:59.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:59.897 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:59.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:13:59.897 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:59.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:59.897 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:59.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:59.897 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:59.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:13:59.897 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:13:59.836+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:13:59.898 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:59.590+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:59.591155+0000) 2024-02-19T20:13:59.898 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:59.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:59.898 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:59.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:59.898 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:59.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:13:59.898 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:13:59.805+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:00.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:13:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:13:59.895+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:00.594 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:00.400+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:00.401658+0000) 2024-02-19T20:14:00.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:00.677+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:00.678693+0000) 2024-02-19T20:14:00.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:00.592+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:00.593703+0000) 2024-02-19T20:14:00.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:00.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:00.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:00.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:00.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:00.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:00.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:00.840+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:01.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:00.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:01.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:00.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:01.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:00.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:01.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:00.919+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:00.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:01.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:00.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:01.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:00.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:01.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:00.870+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:01.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:01.402+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:01.403695+0000) 2024-02-19T20:14:01.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:01.594+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:01.595914+0000) 2024-02-19T20:14:01.865 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:01.680+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:01.681217+0000) 2024-02-19T20:14:02.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:01.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:02.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:01.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:02.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:01.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:02.212 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:01.875+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:02.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:01.864+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:02.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:01.864+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:02.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:01.864+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:02.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:01.864+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:02.213 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:01.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:02.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:01.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:02.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:01.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:02.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:01.888+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:02.493 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:02.404+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:02.405408+0000) 2024-02-19T20:14:02.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:02.596+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:02.597813+0000) 2024-02-19T20:14:02.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:02.681+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:02.682873+0000) 2024-02-19T20:14:03.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:02.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:03.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:02.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:03.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:02.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:03.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:02.896+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:02.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:02.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:02.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:02.905+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:03.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:02.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:03.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:02.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:03.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:02.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:03.272 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:02.898+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:03.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:03.404+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:03.405619+0000) 2024-02-19T20:14:03.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:03.597+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:03.598000+0000) 2024-02-19T20:14:03.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:03.682+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:03.683017+0000) 2024-02-19T20:14:04.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:03.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:04.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:03.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:04.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:03.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:04.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:03.876+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:04.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:03.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:04.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:03.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:04.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:03.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:04.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:03.922+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:04.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:03.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:04.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:03.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:04.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:03.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:04.268 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:03.871+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:04.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:14:04.271+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55008 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:14:04.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:04.268+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:04.269471+0000) 2024-02-19T20:14:04.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:04.268+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:04.269180+0000) 2024-02-19T20:14:04.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:04.268+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:04.269207+0000) 2024-02-19T20:14:04.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:04.404+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:04.405773+0000) 2024-02-19T20:14:04.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:04.682+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:04.683182+0000) 2024-02-19T20:14:04.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:04.597+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:04.598161+0000) 2024-02-19T20:14:04.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:04.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:04.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:04.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:04.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:04.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:04.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:04.835+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:05.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:04.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:04.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:04.916+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:05.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:04.917+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:04.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:04.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:04.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:05.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:04.960+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:05.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:05.407+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:05.408877+0000) 2024-02-19T20:14:05.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:05.685+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:05.686726+0000) 2024-02-19T20:14:05.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:05.599+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:05.600131+0000) 2024-02-19T20:14:05.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:05.790+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:05.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:05.790+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:05.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:05.790+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:05.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:05.791+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:06.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:05.872+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:05.872+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:05.872+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:06.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:05.873+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:06.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:05.911+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:06.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:05.911+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:05.911+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:05.911+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:06.603 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:06.410+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:06.411382+0000) 2024-02-19T20:14:06.863 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:06.689+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:06.690283+0000) 2024-02-19T20:14:06.863 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:06.602+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:06.603025+0000) 2024-02-19T20:14:06.863 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:06.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:06.864 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:06.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:06.864 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:06.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:06.864 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:06.807+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:07.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:06.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:07.262 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:06.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:07.262 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:06.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:07.262 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:06.863+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:07.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:06.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:07.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:06.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:07.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:06.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:07.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:06.883+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:07.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:07.412+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:07.412996+0000) 2024-02-19T20:14:07.855 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:07.690+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:07.691817+0000) 2024-02-19T20:14:07.855 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:07.603+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:07.604944+0000) 2024-02-19T20:14:07.855 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:07.781+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:07.856 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:07.781+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:07.856 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:07.781+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:07.856 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:07.782+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:08.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:07.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:08.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:07.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:08.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:07.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:08.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:07.856+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:08.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:07.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:08.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:07.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:08.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:07.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:08.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:07.855+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:08.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:08.412+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:08.413160+0000) 2024-02-19T20:14:09.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:08.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:09.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:08.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:09.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:08.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:09.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:08.811+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:09.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:08.691+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:08.692007+0000) 2024-02-19T20:14:09.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:08.806+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:09.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:08.806+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:09.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:08.806+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:09.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:08.807+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:09.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:08.604+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:08.605154+0000) 2024-02-19T20:14:09.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:08.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:09.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:08.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:09.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:08.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:09.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:08.768+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:09.296 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:14:09.278+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55048 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:14:09.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:09.275+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:09.276966+0000) 2024-02-19T20:14:09.297 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:09.275+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:09.276692+0000) 2024-02-19T20:14:09.297 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:09.275+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:09.276627+0000) 2024-02-19T20:14:09.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:09.412+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:09.413357+0000) 2024-02-19T20:14:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:09.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:09.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:09.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:10.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:09.807+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:10.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:09.691+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:09.692180+0000) 2024-02-19T20:14:10.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:09.769+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:10.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:09.769+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:10.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:09.769+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:10.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:09.770+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:10.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:09.604+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:09.605365+0000) 2024-02-19T20:14:10.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:09.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:10.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:09.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:10.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:09.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:10.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:09.726+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:10.607 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:10.414+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:10.415447+0000) 2024-02-19T20:14:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:10.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:10.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:10.837+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:11.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:10.837+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:10.606+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:10.607750+0000) 2024-02-19T20:14:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:10.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:10.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:10.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:10.719+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:11.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:10.693+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:10.694399+0000) 2024-02-19T20:14:11.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:10.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:11.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:10.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:11.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:10.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:11.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:10.732+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:11.610 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:11.416+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:11.417455+0000) 2024-02-19T20:14:12.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:11.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:12.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:11.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:12.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:11.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:12.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:11.809+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:12.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:11.608+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:11.609946+0000) 2024-02-19T20:14:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:11.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:11.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:11.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:11.720+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:12.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:11.695+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:11.696482+0000) 2024-02-19T20:14:12.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:11.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:12.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:11.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:12.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:11.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:12.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:11.751+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:12.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:12.418+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:12.419243+0000) 2024-02-19T20:14:13.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:12.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:13.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:12.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:13.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:12.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:13.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:12.772+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:13.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:12.610+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:12.611611+0000) 2024-02-19T20:14:13.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:12.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:13.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:12.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:13.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:12.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:13.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:12.734+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:13.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:12.698+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:12.699078+0000) 2024-02-19T20:14:13.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:12.744+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:13.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:12.744+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:13.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:12.744+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:13.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:12.744+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:13.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:13.418+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:13.419434+0000) 2024-02-19T20:14:14.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:13.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:14.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:13.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:14.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:13.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:14.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:13.750+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:14.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:13.610+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:13.611791+0000) 2024-02-19T20:14:14.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:13.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:14.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:13.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:14.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:13.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:14.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:13.739+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:14.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:13.698+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:13.699254+0000) 2024-02-19T20:14:14.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:13.706+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:14.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:13.706+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:14.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:13.706+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:14.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:13.706+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:14.304 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:14:14.285+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55088 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:14:14.305 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:14.281+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:14.282678+0000) 2024-02-19T20:14:14.305 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:14.282+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:14.283033+0000) 2024-02-19T20:14:14.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:14.282+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:14.283616+0000) 2024-02-19T20:14:14.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:14.418+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:14.419602+0000) 2024-02-19T20:14:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:14.789+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:14.790+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:14.790+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:15.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:14.790+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:15.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:14.611+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:14.612009+0000) 2024-02-19T20:14:15.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:14.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:15.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:14.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:15.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:14.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:15.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:14.768+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:15.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:14.679+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:15.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:14.679+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:15.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:14.679+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:15.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:14.680+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:15.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:14.698+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:14.699431+0000) 2024-02-19T20:14:15.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:15.421+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:15.422059+0000) 2024-02-19T20:14:16.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:15.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:16.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:15.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:16.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:15.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:16.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:15.768+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:16.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:15.613+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:15.614320+0000) 2024-02-19T20:14:16.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:15.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:16.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:15.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:16.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:15.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:16.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:15.792+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:16.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:15.678+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:16.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:15.679+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:16.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:15.679+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:16.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:15.679+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:16.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:15.700+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:15.701571+0000) 2024-02-19T20:14:16.618 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:16.423+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:16.424087+0000) 2024-02-19T20:14:17.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:16.781+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:17.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:16.781+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:17.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:16.781+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:17.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:16.782+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:17.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:16.616+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:16.617816+0000) 2024-02-19T20:14:17.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:16.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:17.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:16.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:17.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:16.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:17.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:16.819+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:17.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:16.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:17.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:16.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:17.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:16.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:17.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:16.695+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:17.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:16.702+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:16.703861+0000) 2024-02-19T20:14:17.619 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:17.424+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:17.425594+0000) 2024-02-19T20:14:18.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:17.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:17.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:17.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:18.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:17.753+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:18.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:17.618+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:17.619332+0000) 2024-02-19T20:14:18.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:17.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:18.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:17.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:18.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:17.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:18.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:17.840+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:18.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:17.705+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:17.706625+0000) 2024-02-19T20:14:18.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:17.719+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:18.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:17.719+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:18.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:17.719+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:18.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:17.720+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:18.619 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:18.424+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:18.425770+0000) 2024-02-19T20:14:18.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:18.618+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:18.619489+0000) 2024-02-19T20:14:18.872 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:18.801+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:18.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:18.801+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:18.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:18.801+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:18.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:18.802+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:18.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:18.705+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:18.706773+0000) 2024-02-19T20:14:18.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:18.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:18.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:18.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:18.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:18.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:18.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:18.718+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:19.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:18.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:18.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:18.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:18.871+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:19.620 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:14:19.291+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55128 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:14:19.620 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:19.288+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:19.289900+0000) 2024-02-19T20:14:19.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:19.288+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:19.289754+0000) 2024-02-19T20:14:19.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:19.289+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:19.290590+0000) 2024-02-19T20:14:19.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:19.425+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:19.425975+0000) 2024-02-19T20:14:19.922 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:19.823+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:19.923 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:19.823+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:19.923 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:19.823+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:19.923 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:19.824+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:19.923 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:19.618+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:19.619693+0000) 2024-02-19T20:14:19.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:19.921+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:19.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:19.921+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:19.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:19.921+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:19.924 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:19.677+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:19.924 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:19.677+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:19.924 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:19.677+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:19.925 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:19.677+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:19.925 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:19.705+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:19.706948+0000) 2024-02-19T20:14:20.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:19.922+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:20.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:20.427+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:20.428148+0000) 2024-02-19T20:14:20.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:20.621+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:20.622228+0000) 2024-02-19T20:14:20.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:20.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:20.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:20.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:20.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:20.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:20.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:20.816+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:20.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:20.676+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:20.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:20.676+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:20.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:20.676+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:20.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:20.676+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:20.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:20.708+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:20.709356+0000) 2024-02-19T20:14:21.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:20.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:20.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:20.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:20.955+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:21.624 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:21.430+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:21.431242+0000) 2024-02-19T20:14:21.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:21.623+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:21.624132+0000) 2024-02-19T20:14:21.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:21.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:21.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:21.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:21.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:21.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:21.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:21.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:21.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:21.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:21.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:21.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:21.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:21.786+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:21.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:21.678+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:21.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:21.678+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:21.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:21.678+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:21.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:21.679+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:21.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:21.710+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:21.711491+0000) 2024-02-19T20:14:22.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:21.934+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:22.626 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:22.431+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:22.432801+0000) 2024-02-19T20:14:22.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:22.625+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:22.626145+0000) 2024-02-19T20:14:22.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:22.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:22.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:22.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:22.947 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:22.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:22.947 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:22.829+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:22.947 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:22.696+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:22.947 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:22.696+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:22.947 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:22.696+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:22.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:22.697+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:22.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:22.713+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:22.714252+0000) 2024-02-19T20:14:23.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:22.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:22.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:22.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:22.946+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:23.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:23.431+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:23.432956+0000) 2024-02-19T20:14:23.856 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:23.625+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:23.626323+0000) 2024-02-19T20:14:23.857 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:23.713+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:23.714416+0000) 2024-02-19T20:14:23.857 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:23.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:23.857 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:23.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:23.857 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:23.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:23.857 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:23.737+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:23.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:23.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:23.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:23.946+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:24.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:23.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:24.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:23.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:24.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:23.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:24.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:23.856+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:24.626 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:14:24.299+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55168 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:14:24.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:24.296+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:24.297107+0000) 2024-02-19T20:14:24.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:24.295+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:24.296765+0000) 2024-02-19T20:14:24.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:24.296+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:24.297353+0000) 2024-02-19T20:14:24.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:24.432+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:24.433152+0000) 2024-02-19T20:14:24.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:24.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:24.971 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:24.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:24.971 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:24.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:24.971 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:24.817+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:24.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:24.625+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:24.626533+0000) 2024-02-19T20:14:24.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:24.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:24.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:24.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:24.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:24.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:24.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:24.713+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:24.714534+0000) 2024-02-19T20:14:24.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:24.719+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:24.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:24.719+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:24.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:24.719+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:24.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:24.720+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:25.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:24.970+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:25.632 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:25.434+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:25.435213+0000) 2024-02-19T20:14:25.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:25.628+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:25.629022+0000) 2024-02-19T20:14:25.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:25.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:25.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:25.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:25.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:25.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:25.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:25.825+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:25.969 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:25.707+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:25.969 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:25.707+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:25.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:25.707+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:25.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:25.707+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:25.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:25.716+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:25.717150+0000) 2024-02-19T20:14:26.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:25.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:25.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:25.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:25.967+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:26.632 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:26.437+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:26.437792+0000) 2024-02-19T20:14:26.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:26.631+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:26.631721+0000) 2024-02-19T20:14:26.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:26.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:26.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:26.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:26.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:26.798+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:26.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:26.799+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:26.989 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:26.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:26.989 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:26.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:26.989 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:26.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:26.989 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:26.717+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:26.989 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:26.718+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:26.719249+0000) 2024-02-19T20:14:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:26.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:26.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:26.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:26.986+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:27.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:27.437+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:27.439501+0000) 2024-02-19T20:14:28.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:27.632+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:27.634467+0000) 2024-02-19T20:14:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:27.776+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:27.776+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:27.776+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:28.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:27.777+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:28.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:27.718+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:27.720945+0000) 2024-02-19T20:14:28.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:27.742+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:28.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:27.742+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:28.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:27.742+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:28.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:27.743+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:28.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:28.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:28.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:28.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:28.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:28.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:28.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:28.033+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:28.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:28.438+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:28.439667+0000) 2024-02-19T20:14:29.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:28.633+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:28.634666+0000) 2024-02-19T20:14:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:28.720+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:28.721005+0000) 2024-02-19T20:14:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:28.780+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:28.780+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:28.780+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:29.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:28.780+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:29.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:28.789+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:29.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:28.789+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:29.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:28.789+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:29.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:28.790+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:29.301 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:29.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:29.302 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:29.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:29.302 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:29.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:29.302 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:29.080+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:29.635 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:14:29.306+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55208 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:14:29.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:29.303+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:29.304571+0000) 2024-02-19T20:14:29.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:29.303+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:29.304714+0000) 2024-02-19T20:14:29.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:29.305+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:29.306269+0000) 2024-02-19T20:14:29.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:29.438+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:29.439921+0000) 2024-02-19T20:14:30.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:29.633+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:29.634874+0000) 2024-02-19T20:14:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:29.751+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:29.751+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:29.751+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:29.752+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:30.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:29.720+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:29.721151+0000) 2024-02-19T20:14:30.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:29.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:30.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:29.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:30.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:29.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:30.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:29.811+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:30.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:30.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:30.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:30.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:30.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:30.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:30.318 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:30.047+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:30.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:30.441+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:30.442377+0000) 2024-02-19T20:14:31.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:30.636+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:30.637108+0000) 2024-02-19T20:14:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:30.743+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:30.743+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:30.743+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:31.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:30.744+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:31.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:30.722+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:30.723521+0000) 2024-02-19T20:14:31.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:30.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:31.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:30.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:31.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:30.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:31.023 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:30.815+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:31.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:31.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:31.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:31.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:31.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:31.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:31.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:31.025+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:31.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:31.444+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:31.445022+0000) 2024-02-19T20:14:32.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:31.639+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:31.640701+0000) 2024-02-19T20:14:32.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:31.755+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:32.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:31.755+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:32.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:31.755+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:32.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:31.756+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:32.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:31.725+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:31.726097+0000) 2024-02-19T20:14:32.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:31.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:32.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:31.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:32.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:31.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:32.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:31.776+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:32.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:32.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:32.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:32.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:32.017+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:32.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:32.446+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:32.447583+0000) 2024-02-19T20:14:33.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:32.642+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:32.643265+0000) 2024-02-19T20:14:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:32.729+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:32.729+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:32.729+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:33.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:32.729+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:33.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:32.726+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:32.727722+0000) 2024-02-19T20:14:33.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:32.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:33.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:32.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:33.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:32.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:33.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:32.757+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:33.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:33.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:33.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:33.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:33.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:33.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:33.318 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:33.063+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:33.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:33.446+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:33.447753+0000) 2024-02-19T20:14:34.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:33.642+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:33.643438+0000) 2024-02-19T20:14:34.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:33.726+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:33.727922+0000) 2024-02-19T20:14:34.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:33.755+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:34.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:33.755+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:34.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:33.755+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:34.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:33.756+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:34.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:33.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:34.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:33.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:34.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:33.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:34.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:33.706+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:34.309 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:34.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:34.309 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:34.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:34.309 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:34.105+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:34.309 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:34.106+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:34.643 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:14:34.313+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55248 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:14:34.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:34.310+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:34.311411+0000) 2024-02-19T20:14:34.644 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:34.311+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:34.312566+0000) 2024-02-19T20:14:34.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:34.310+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:34.311749+0000) 2024-02-19T20:14:34.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:34.446+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:34.447943+0000) 2024-02-19T20:14:35.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:34.642+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:34.643661+0000) 2024-02-19T20:14:35.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:34.727+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:34.728094+0000) 2024-02-19T20:14:35.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:34.758+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:35.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:34.758+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:35.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:34.758+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:35.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:34.759+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:35.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:34.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:35.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:34.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:35.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:34.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:35.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:34.697+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:35.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:35.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:35.318 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:35.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:35.318 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:35.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:35.318 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:35.059+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:35.647 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:35.448+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:35.449870+0000) 2024-02-19T20:14:36.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:35.646+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:35.647041+0000) 2024-02-19T20:14:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:35.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:35.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:35.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:36.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:35.670+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:36.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:35.729+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:35.730749+0000) 2024-02-19T20:14:36.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:35.767+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:36.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:35.767+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:36.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:35.767+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:36.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:35.767+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:36.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:36.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:36.318 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:36.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:36.318 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:36.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:36.318 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:36.039+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:36.649 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:36.450+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:36.451814+0000) 2024-02-19T20:14:37.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:36.648+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:36.649137+0000) 2024-02-19T20:14:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:36.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:36.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:37.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:36.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:37.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:36.669+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:37.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:36.731+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:36.732747+0000) 2024-02-19T20:14:37.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:36.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:37.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:36.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:37.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:36.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:37.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:36.764+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:37.318 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:37.042+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:37.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:37.042+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:37.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:37.042+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:37.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:37.043+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:37.570 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:37.452+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:37.453672+0000) 2024-02-19T20:14:38.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:37.650+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:37.651129+0000) 2024-02-19T20:14:38.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:37.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:38.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:37.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:38.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:37.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:38.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:37.632+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:38.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:37.733+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:37.734352+0000) 2024-02-19T20:14:38.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:37.752+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:38.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:37.752+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:38.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:37.752+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:38.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:37.752+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:38.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:38.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:38.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:38.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:38.004+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:38.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:38.452+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:38.453870+0000) 2024-02-19T20:14:38.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:38.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:38.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:38.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:38.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:38.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:38.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:38.587+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:38.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:38.650+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:38.651302+0000) 2024-02-19T20:14:38.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:38.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:38.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:38.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:38.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:38.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:38.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:38.728+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:38.989 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:38.728+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:38.989 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:38.728+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:38.989 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:38.728+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:38.989 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:38.733+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:38.734532+0000) 2024-02-19T20:14:39.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:38.987+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:39.651 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:14:39.320+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55288 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:14:39.652 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:39.317+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:39.318805+0000) 2024-02-19T20:14:39.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:39.317+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:39.318810+0000) 2024-02-19T20:14:39.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:39.453+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:39.454001+0000) 2024-02-19T20:14:39.653 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:39.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:39.653 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:39.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:39.653 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:39.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:39.653 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:39.558+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:39.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:39.318+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:39.319733+0000) 2024-02-19T20:14:39.958 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:39.650+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:39.651450+0000) 2024-02-19T20:14:39.958 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:39.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:39.959 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:39.733+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:39.734787+0000) 2024-02-19T20:14:39.959 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:39.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:39.959 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:39.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:39.959 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:39.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:39.959 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:39.766+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:39.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:39.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:39.958+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:40.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:40.455+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:40.456436+0000) 2024-02-19T20:14:40.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:40.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:40.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:40.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:40.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:40.527+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:40.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:40.529+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:40.949 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:40.653+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:40.654729+0000) 2024-02-19T20:14:40.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:40.720+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:40.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:40.720+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:40.950 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:40.720+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:40.950 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:40.720+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:40.950 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:40.735+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:40.736899+0000) 2024-02-19T20:14:41.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:40.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:40.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:40.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:40.949+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:41.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:41.457+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:41.458598+0000) 2024-02-19T20:14:41.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:41.499+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:41.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:41.500+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:41.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:41.500+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:41.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:41.501+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:41.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:41.656+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:41.657271+0000) 2024-02-19T20:14:41.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:41.959+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:41.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:41.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:41.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:41.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:41.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:41.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:41.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:41.721+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:41.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:41.738+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:41.739462+0000) 2024-02-19T20:14:42.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:41.959+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:41.959+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:41.960+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:42.625 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:42.457+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:42.625 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:42.457+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:42.625 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:42.457+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:42.625 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:42.457+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:42.626 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:42.459+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:42.460292+0000) 2024-02-19T20:14:42.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:42.657+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:42.658952+0000) 2024-02-19T20:14:42.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:42.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:42.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:42.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:42.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:42.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:42.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:42.681+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:42.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:42.740+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:42.741064+0000) 2024-02-19T20:14:43.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:42.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:42.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:42.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:43.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:42.925+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:43.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:43.441+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:43.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:43.441+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:43.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:43.441+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:43.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:43.442+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:43.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:43.459+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:43.460507+0000) 2024-02-19T20:14:43.899 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:43.658+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:43.659199+0000) 2024-02-19T20:14:43.899 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:43.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:43.899 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:43.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:43.900 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:43.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:43.900 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:43.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:43.900 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:43.697+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:43.900 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:43.740+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:43.741232+0000) 2024-02-19T20:14:44.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:43.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:43.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:43.899+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:44.659 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:14:44.327+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55329 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:14:44.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:44.325+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:44.326066+0000) 2024-02-19T20:14:44.660 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:44.324+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:44.325748+0000) 2024-02-19T20:14:44.660 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:44.325+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:44.326125+0000) 2024-02-19T20:14:44.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:44.449+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:44.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:44.449+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:44.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:44.449+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:44.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:44.450+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:44.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:44.459+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:44.460692+0000) 2024-02-19T20:14:45.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:44.658+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:44.659506+0000) 2024-02-19T20:14:45.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:44.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:45.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:44.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:45.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:44.905+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:45.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:44.906+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:45.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:44.659+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:45.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:44.659+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:45.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:44.659+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:45.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:44.659+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:45.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:44.740+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:44.741430+0000) 2024-02-19T20:14:45.660 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:45.461+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:45.462721+0000) 2024-02-19T20:14:45.660 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:45.500+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:45.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:45.500+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:45.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:45.500+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:45.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:45.501+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:45.659+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:45.659+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:45.659+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:45.659+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:45.742+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:45.743439+0000) 2024-02-19T20:14:46.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:45.661+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:45.662806+0000) 2024-02-19T20:14:46.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:45.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:46.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:45.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:46.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:45.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:46.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:45.860+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:46.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:46.463+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:46.464899+0000) 2024-02-19T20:14:46.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:46.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:46.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:46.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:46.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:46.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:46.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:46.545+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:46.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:46.665+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:46.666322+0000) 2024-02-19T20:14:46.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:46.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:46.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:46.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:46.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:46.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:46.891 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:46.660+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:46.891 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:46.660+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:46.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:46.660+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:46.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:46.660+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:46.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:46.744+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:46.745501+0000) 2024-02-19T20:14:47.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:46.890+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:47.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:47.466+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:47.467720+0000) 2024-02-19T20:14:47.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:47.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:47.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:47.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:47.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:47.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:47.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:47.538+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:48.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:47.667+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:47.668016+0000) 2024-02-19T20:14:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:47.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:47.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:47.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:47.867+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:48.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:47.696+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:47.696+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:47.696+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:47.697+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:47.746+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:47.747257+0000) 2024-02-19T20:14:48.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:48.466+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:48.467892+0000) 2024-02-19T20:14:48.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:48.515+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:48.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:48.515+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:48.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:48.515+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:48.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:48.516+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:48.667+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:48.668204+0000) 2024-02-19T20:14:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:48.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:48.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:48.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:49.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:48.907+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:49.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:48.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:49.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:48.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:49.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:48.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:49.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:48.666+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:49.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:48.746+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:48.747432+0000) 2024-02-19T20:14:49.668 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:14:49.334+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55370 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:14:49.669 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:49.331+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:49.332441+0000) 2024-02-19T20:14:49.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:49.332+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:49.333190+0000) 2024-02-19T20:14:49.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:49.332+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:49.333075+0000) 2024-02-19T20:14:49.670 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:49.467+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:49.468072+0000) 2024-02-19T20:14:49.670 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:49.564+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:49.670 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:49.564+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:49.670 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:49.564+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:49.670 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:49.565+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:50.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:49.667+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:49.668397+0000) 2024-02-19T20:14:50.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:49.867+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:50.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:49.867+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:50.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:49.867+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:50.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:49.868+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:50.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:49.670+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:50.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:49.670+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:50.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:49.670+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:50.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:49.671+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:50.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:49.746+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:49.747609+0000) 2024-02-19T20:14:50.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:50.469+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:50.470028+0000) 2024-02-19T20:14:50.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:50.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:50.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:50.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:50.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:50.598+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:50.890 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:50.669+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:50.670968+0000) 2024-02-19T20:14:50.890 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:50.599+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:50.891 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:50.644+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:50.891 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:50.644+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:50.891 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:50.644+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:50.891 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:50.644+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:50.891 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:50.749+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:50.750025+0000) 2024-02-19T20:14:51.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:50.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:51.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:50.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:51.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:50.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:51.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:50.890+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:51.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:51.471+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:51.472249+0000) 2024-02-19T20:14:51.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:51.574+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:51.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:51.574+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:51.576 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:51.574+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:51.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:51.574+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:51.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:51.673+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:51.674518+0000) 2024-02-19T20:14:51.921 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:51.667+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:51.921 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:51.667+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:51.921 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:51.667+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:51.921 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:51.668+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:51.921 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:51.751+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:51.752086+0000) 2024-02-19T20:14:52.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:51.918+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:51.918+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:51.918+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:51.920+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:52.576 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:52.473+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:52.474292+0000) 2024-02-19T20:14:52.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:52.675+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:52.676553+0000) 2024-02-19T20:14:52.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:52.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:52.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:52.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:52.918 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:52.575+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:52.918 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:52.576+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:52.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:52.625+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:52.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:52.625+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:52.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:52.625+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:52.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:52.626+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:52.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:52.753+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:52.754219+0000) 2024-02-19T20:14:53.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:52.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:52.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:52.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:52.917+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:53.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:53.473+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:53.474461+0000) 2024-02-19T20:14:53.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:53.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:53.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:53.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:53.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:53.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:53.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:53.568+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:53.948 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:53.675+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:53.676730+0000) 2024-02-19T20:14:53.948 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:53.946+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:53.948 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:53.946+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:53.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:53.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:53.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:53.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:53.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:53.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:53.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:53.624+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:53.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:53.753+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:53.754440+0000) 2024-02-19T20:14:54.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:53.946+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:54.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:53.947+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:54.642 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:14:54.341+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55409 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:14:54.643 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:54.338+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:54.339632+0000) 2024-02-19T20:14:54.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:54.339+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:54.340262+0000) 2024-02-19T20:14:54.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:54.641+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:54.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:54.641+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:54.644 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:54.641+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:54.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:54.339+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:54.340498+0000) 2024-02-19T20:14:54.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:54.473+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:54.474637+0000) 2024-02-19T20:14:54.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:54.541+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:54.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:54.541+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:54.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:54.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:54.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:54.543+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:54.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:54.675+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:54.676930+0000) 2024-02-19T20:14:54.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:54.642+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:54.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:54.753+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:54.754605+0000) 2024-02-19T20:14:55.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:54.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:54.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:54.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:54.983+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:55.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:55.475+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:55.476723+0000) 2024-02-19T20:14:55.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:55.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:55.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:55.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:55.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:55.537+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:55.680 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:55.538+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:55.941 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:55.678+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:55.679108+0000) 2024-02-19T20:14:55.942 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:55.679+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:55.942 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:55.679+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:55.942 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:55.679+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:55.942 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:55.680+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:55.942 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:55.755+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:55.756769+0000) 2024-02-19T20:14:56.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:55.940+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:56.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:55.940+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:56.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:55.940+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:56.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:55.941+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:56.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:56.477+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:56.478937+0000) 2024-02-19T20:14:56.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:56.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:56.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:56.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:56.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:56.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:56.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:56.515+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:56.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:56.680+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:56.681852+0000) 2024-02-19T20:14:56.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:56.959+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:56.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:56.959+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:56.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:56.959+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:56.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:56.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:56.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:56.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:56.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:56.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:56.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:56.714+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:56.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:56.759+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:56.760190+0000) 2024-02-19T20:14:57.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:56.960+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:57.684 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:57.480+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:57.481031+0000) 2024-02-19T20:14:57.684 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:57.515+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:57.685 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:57.515+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:57.685 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:57.515+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:57.685 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:57.515+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:57.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:57.683+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:57.684262+0000) 2024-02-19T20:14:57.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:57.937+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:57.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:57.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:57.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:57.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:57.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:57.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:57.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:57.751+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:57.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:57.761+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:57.762318+0000) 2024-02-19T20:14:58.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:57.937+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:57.937+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:57.938+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:58.685 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:58.480+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:58.481181+0000) 2024-02-19T20:14:58.685 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:58.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:58.685 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:58.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:58.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:58.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:58.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:58.492+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:59.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:58.683+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:58.684440+0000) 2024-02-19T20:14:59.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:58.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:59.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:58.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:59.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:58.925+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:14:59.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:58.926+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:14:59.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:58.743+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:59.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:58.743+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:59.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:58.743+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:14:59.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:58.743+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:14:59.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:58.761+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:58.762475+0000) 2024-02-19T20:14:59.685 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:14:59.348+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55450 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:14:59.685 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:59.346+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:59.347174+0000) 2024-02-19T20:14:59.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:59.345+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:59.346737+0000) 2024-02-19T20:14:59.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:59.346+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:59.347017+0000) 2024-02-19T20:14:59.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:59.480+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:59.481345+0000) 2024-02-19T20:14:59.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:59.516+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:59.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:59.516+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:59.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:59.516+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:14:59.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:14:59.517+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:59.683+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:59.684660+0000) 2024-02-19T20:15:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:59.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:59.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:59.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:14:59.886+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:00.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:59.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:00.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:59.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:00.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:59.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:00.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:59.759+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:00.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:14:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:14:59.761+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:59.762647+0000) 2024-02-19T20:15:00.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:00.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:00.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:00.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:00.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:00.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:00.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:00.473+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:00.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:00.482+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:00.483348+0000) 2024-02-19T20:15:01.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:00.685+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:00.686803+0000) 2024-02-19T20:15:01.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:00.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:01.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:00.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:01.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:00.893+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:01.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:00.893+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:01.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:00.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:01.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:00.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:01.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:00.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:01.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:00.722+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:01.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:00.764+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:00.765972+0000) 2024-02-19T20:15:01.690 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:01.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:01.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:01.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:01.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:01.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:01.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:01.426+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:01.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:01.484+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:01.485620+0000) 2024-02-19T20:15:02.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:01.689+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:01.690261+0000) 2024-02-19T20:15:02.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:01.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:02.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:01.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:02.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:01.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:02.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:01.885+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:02.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:01.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:02.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:01.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:02.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:01.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:02.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:01.756+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:02.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:01.767+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:01.768076+0000) 2024-02-19T20:15:02.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:02.384+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:02.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:02.384+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:02.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:02.384+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:02.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:02.385+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:02.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:02.486+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:02.487231+0000) 2024-02-19T20:15:03.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:02.691+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:02.692118+0000) 2024-02-19T20:15:03.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:02.868+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:03.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:02.868+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:03.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:02.868+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:03.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:02.869+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:03.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:02.755+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:03.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:02.755+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:03.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:02.755+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:03.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:02.756+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:03.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:02.769+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:02.770884+0000) 2024-02-19T20:15:03.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:03.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:03.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:03.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:03.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:03.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:03.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:03.350+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:03.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:03.486+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:03.487421+0000) 2024-02-19T20:15:03.842 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:03.691+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:03.692347+0000) 2024-02-19T20:15:03.842 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:03.770+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:03.771043+0000) 2024-02-19T20:15:03.842 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:03.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:03.842 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:03.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:03.842 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:03.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:03.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:03.771+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:04.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:03.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:03.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:03.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:03.841+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:04.692 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:15:04.355+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55489 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:15:04.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:04.352+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:04.353943+0000) 2024-02-19T20:15:04.693 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:04.353+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:04.354257+0000) 2024-02-19T20:15:04.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:04.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:04.694 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:04.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:04.694 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:04.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:04.694 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:04.351+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:04.694 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:04.353+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:04.354298+0000) 2024-02-19T20:15:04.694 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:04.486+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:04.487610+0000) 2024-02-19T20:15:05.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:04.691+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:04.692532+0000) 2024-02-19T20:15:05.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:04.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:05.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:04.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:05.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:04.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:05.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:04.848+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:05.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:04.770+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:04.771134+0000) 2024-02-19T20:15:05.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:04.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:05.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:04.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:05.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:04.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:05.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:04.782+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:05.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:05.329+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:05.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:05.329+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:05.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:05.329+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:05.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:05.329+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:05.696 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:05.489+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:05.490116+0000) 2024-02-19T20:15:06.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:05.693+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:05.694909+0000) 2024-02-19T20:15:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:05.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:05.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:05.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:06.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:05.876+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:06.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:05.772+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:05.773746+0000) 2024-02-19T20:15:06.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:05.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:06.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:05.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:06.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:05.818+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:06.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:05.818+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:06.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:06.361+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:06.698 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:06.361+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:06.698 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:06.361+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:06.698 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:06.361+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:06.698 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:06.492+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:06.493626+0000) 2024-02-19T20:15:07.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:06.696+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:06.697437+0000) 2024-02-19T20:15:07.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:06.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:07.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:06.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:07.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:06.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:07.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:06.830+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:07.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:06.775+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:06.776364+0000) 2024-02-19T20:15:07.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:06.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:07.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:06.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:07.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:06.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:07.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:06.861+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:07.699 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:07.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:07.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:07.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:07.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:07.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:07.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:07.373+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:07.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:07.494+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:07.495405+0000) 2024-02-19T20:15:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:07.698+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:07.699299+0000) 2024-02-19T20:15:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:07.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:07.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:07.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:08.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:07.814+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:08.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:07.777+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:07.778026+0000) 2024-02-19T20:15:08.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:07.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:08.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:07.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:08.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:07.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:08.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:07.872+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:08.699 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:08.409+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:08.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:08.409+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:08.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:08.409+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:08.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:08.410+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:08.700 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:08.494+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:08.495638+0000) 2024-02-19T20:15:09.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:08.698+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:08.699495+0000) 2024-02-19T20:15:09.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:08.790+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:09.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:08.790+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:09.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:08.790+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:09.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:08.791+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:09.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:08.777+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:08.778198+0000) 2024-02-19T20:15:09.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:08.852+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:09.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:08.852+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:09.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:08.852+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:09.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:08.853+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:09.700 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:15:09.363+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55530 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:15:09.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:09.359+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:09.360276+0000) 2024-02-19T20:15:09.703 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:09.359+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:09.360974+0000) 2024-02-19T20:15:09.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:09.359+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:09.360519+0000) 2024-02-19T20:15:09.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:09.382+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:09.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:09.382+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:09.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:09.382+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:09.704 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:09.383+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:09.704 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:09.494+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:09.495802+0000) 2024-02-19T20:15:10.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:09.777+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:09.778427+0000) 2024-02-19T20:15:10.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:09.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:10.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:09.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:10.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:09.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:10.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:09.863+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:09.698+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:09.699740+0000) 2024-02-19T20:15:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:09.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:09.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:09.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:10.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:09.744+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:10.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:10.351+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:10.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:10.351+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:10.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:10.352+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:10.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:10.352+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:10.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:10.497+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:10.498267+0000) 2024-02-19T20:15:11.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:10.779+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:10.780617+0000) 2024-02-19T20:15:11.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:10.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:11.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:10.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:11.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:10.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:11.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:10.909+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:10.701+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:10.702479+0000) 2024-02-19T20:15:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:10.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:10.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:10.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:11.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:10.764+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:11.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:11.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:11.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:11.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:11.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:11.338+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:11.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:11.339+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:11.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:11.500+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:11.501771+0000) 2024-02-19T20:15:12.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:11.782+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:11.783164+0000) 2024-02-19T20:15:12.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:11.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:12.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:11.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:12.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:11.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:12.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:11.896+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:12.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:11.704+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:11.705054+0000) 2024-02-19T20:15:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:11.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:11.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:11.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:12.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:11.795+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:12.708 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:12.314+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:12.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:12.314+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:12.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:12.314+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:12.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:12.315+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:12.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:12.503+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:12.504650+0000) 2024-02-19T20:15:13.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:12.784+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:12.785207+0000) 2024-02-19T20:15:13.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:12.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:13.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:12.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:13.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:12.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:13.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:12.850+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:13.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:12.706+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:12.707144+0000) 2024-02-19T20:15:13.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:12.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:13.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:12.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:13.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:12.754+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:13.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:12.755+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:13.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:13.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:13.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:13.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:13.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:13.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:13.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:13.341+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:13.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:13.503+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:13.504830+0000) 2024-02-19T20:15:14.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:13.706+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:13.707324+0000) 2024-02-19T20:15:14.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:13.724+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:14.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:13.724+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:14.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:13.724+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:14.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:13.724+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:14.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:13.785+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:13.785449+0000) 2024-02-19T20:15:14.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:13.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:14.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:13.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:14.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:13.812+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:14.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:13.813+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:14.695 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:15:14.369+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55569 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:15:14.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:14.367+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:14.368198+0000) 2024-02-19T20:15:14.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:14.368+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:14.369018+0000) 2024-02-19T20:15:14.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:14.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:14.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:14.303+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:14.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:14.303+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:14.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:14.303+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:14.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:14.304+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:14.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:14.368+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:14.369315+0000) 2024-02-19T20:15:14.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:14.504+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:14.505000+0000) 2024-02-19T20:15:15.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:14.784+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:14.785654+0000) 2024-02-19T20:15:15.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:14.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:15.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:14.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:15.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:14.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:15.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:14.856+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:15.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:14.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:15.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:14.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:15.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:14.694+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:15.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:14.706+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:14.707524+0000) 2024-02-19T20:15:15.322 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:15.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:15.322 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:15.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:15.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:15.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:15.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:15.294+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:15.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:15.506+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:15.508566+0000) 2024-02-19T20:15:16.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:15.707+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:15.709746+0000) 2024-02-19T20:15:16.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:15.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:16.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:15.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:16.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:15.714+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:16.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:15.714+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:16.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:15.788+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:15.789193+0000) 2024-02-19T20:15:16.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:15.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:16.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:15.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:16.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:15.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:16.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:15.899+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:16.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:16.304+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:16.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:16.304+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:16.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:16.304+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:16.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:16.305+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:16.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:16.511+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:16.512054+0000) 2024-02-19T20:15:17.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:16.711+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:16.712259+0000) 2024-02-19T20:15:17.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:16.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:17.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:16.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:17.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:16.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:17.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:16.752+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:16.791+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:16.792819+0000) 2024-02-19T20:15:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:16.878+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:17.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:16.878+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:17.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:16.878+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:17.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:16.879+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:17.322 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:17.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:17.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:17.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:17.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:17.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:17.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:17.277+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:17.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:17.512+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:17.513749+0000) 2024-02-19T20:15:17.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:17.713+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:17.714733+0000) 2024-02-19T20:15:17.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:17.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:17.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:17.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:17.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:17.761+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:17.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:17.761+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:17.999 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:17.793+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:17.794591+0000) 2024-02-19T20:15:18.000 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:17.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:18.000 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:17.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:18.000 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:17.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:18.000 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:17.927+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:18.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:18.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:18.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:18.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:18.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:18.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:18.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:18.266+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:18.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:18.512+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:18.513932+0000) 2024-02-19T20:15:19.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:18.713+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:18.714953+0000) 2024-02-19T20:15:19.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:18.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:19.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:18.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:19.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:18.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:19.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:18.765+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:19.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:18.793+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:18.794863+0000) 2024-02-19T20:15:19.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:18.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:19.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:18.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:19.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:18.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:19.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:18.919+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:19.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:19.274+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:19.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:19.274+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:19.324 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:19.274+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:19.324 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:19.275+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:19.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:15:19.376+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55610 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:15:19.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:19.374+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:19.375217+0000) 2024-02-19T20:15:19.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:19.375+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:19.376810+0000) 2024-02-19T20:15:19.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:19.373+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:19.374468+0000) 2024-02-19T20:15:19.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:19.513+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:19.514123+0000) 2024-02-19T20:15:20.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:19.714+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:19.715149+0000) 2024-02-19T20:15:20.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:19.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:20.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:19.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:20.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:19.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:20.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:19.781+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:20.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:19.794+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:19.795039+0000) 2024-02-19T20:15:20.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:19.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:20.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:19.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:20.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:19.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:20.022 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:19.957+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:20.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:20.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:20.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:20.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:20.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:20.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:20.324 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:20.256+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:20.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:20.515+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:20.516795+0000) 2024-02-19T20:15:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:20.717+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:20.718595+0000) 2024-02-19T20:15:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:20.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:20.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:20.786+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:21.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:20.787+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:21.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:20.796+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:20.797633+0000) 2024-02-19T20:15:21.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:20.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:21.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:20.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:21.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:20.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:21.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:20.967+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:21.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:21.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:21.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:21.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:21.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:21.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:21.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:21.265+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:21.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:21.517+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:21.518869+0000) 2024-02-19T20:15:21.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:21.720+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:21.721224+0000) 2024-02-19T20:15:21.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:21.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:21.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:21.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:21.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:21.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:21.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:21.833+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:21.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:21.799+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:21.800184+0000) 2024-02-19T20:15:21.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:21.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:21.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:21.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:21.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:21.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:22.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:21.983+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:22.723 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:22.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:22.723 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:22.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:22.723 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:22.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:22.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:22.309+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:22.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:22.521+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:22.522020+0000) 2024-02-19T20:15:23.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:22.800+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:22.801965+0000) 2024-02-19T20:15:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:22.722+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:22.723159+0000) 2024-02-19T20:15:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:22.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:22.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:23.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:22.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:23.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:22.871+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:23.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:23.272+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:23.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:23.272+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:23.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:23.272+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:23.324 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:23.273+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:23.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:23.031+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:23.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:23.031+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:23.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:23.031+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:23.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:23.032+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:23.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:23.521+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:23.522178+0000) 2024-02-19T20:15:23.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:23.722+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:23.723310+0000) 2024-02-19T20:15:23.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:23.801+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:23.802133+0000) 2024-02-19T20:15:24.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:23.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:23.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:23.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:23.881+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:24.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:24.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:24.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:24.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:24.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:24.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:24.053+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:24.723 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:15:24.383+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55649 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:15:24.724 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:24.380+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:24.381899+0000) 2024-02-19T20:15:24.724 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:24.380+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:24.381697+0000) 2024-02-19T20:15:24.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:24.315+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:24.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:24.315+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:24.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:24.315+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:24.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:24.316+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:24.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:24.381+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:24.382057+0000) 2024-02-19T20:15:24.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:24.521+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:24.522362+0000) 2024-02-19T20:15:25.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:24.722+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:24.723510+0000) 2024-02-19T20:15:25.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:24.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:25.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:24.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:25.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:24.875+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:25.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:24.876+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:25.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:24.801+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:24.802314+0000) 2024-02-19T20:15:25.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:25.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:25.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:25.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:25.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:25.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:25.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:25.017+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:25.726 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:25.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:25.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:25.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:25.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:25.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:25.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:25.353+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:25.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:25.523+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:25.524899+0000) 2024-02-19T20:15:26.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:25.804+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:25.805594+0000) 2024-02-19T20:15:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:25.725+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:25.726556+0000) 2024-02-19T20:15:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:25.914+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:25.914+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:25.914+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:26.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:25.915+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:26.323 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:26.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:26.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:26.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:26.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:26.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:26.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:26.020+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:26.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:26.380+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:26.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:26.380+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:26.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:26.380+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:26.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:26.380+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:26.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:26.526+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:26.527539+0000) 2024-02-19T20:15:26.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:26.727+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:26.728717+0000) 2024-02-19T20:15:26.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:26.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:26.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:26.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:26.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:26.870+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:26.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:26.871+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:26.975 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:26.807+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:26.808876+0000) 2024-02-19T20:15:26.975 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:26.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:26.975 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:26.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:26.975 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:26.973+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:27.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:26.973+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:27.731 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:27.377+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:27.732 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:27.377+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:27.732 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:27.377+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:27.732 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:27.378+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:27.732 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:27.528+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:27.529714+0000) 2024-02-19T20:15:28.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:27.810+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:27.811639+0000) 2024-02-19T20:15:28.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:27.730+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:27.731468+0000) 2024-02-19T20:15:28.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:27.891+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:28.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:27.891+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:28.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:27.891+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:28.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:27.892+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:28.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:28.021+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:28.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:28.021+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:28.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:28.021+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:28.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:28.022+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:28.732 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:28.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:28.732 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:28.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:28.732 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:28.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:28.732 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:28.365+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:28.732 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:28.528+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:28.529883+0000) 2024-02-19T20:15:28.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:28.730+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:28.731681+0000) 2024-02-19T20:15:28.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:28.926+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:28.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:28.926+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:28.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:28.926+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:28.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:28.927+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:28.999 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:28.810+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:28.811833+0000) 2024-02-19T20:15:28.999 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:28.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:28.999 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:28.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:29.000 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:28.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:29.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:28.997+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:29.732 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:15:29.390+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55690 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:15:29.732 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:29.387+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:29.388759+0000) 2024-02-19T20:15:29.733 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:29.386+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:29.387956+0000) 2024-02-19T20:15:29.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:29.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:29.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:29.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:29.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:29.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:29.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:29.348+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:29.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:29.388+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:29.389076+0000) 2024-02-19T20:15:29.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:29.529+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:29.530076+0000) 2024-02-19T20:15:30.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:29.730+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:29.731854+0000) 2024-02-19T20:15:30.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:29.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:30.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:29.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:30.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:29.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:30.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:29.924+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:30.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:29.811+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:29.812032+0000) 2024-02-19T20:15:30.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:29.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:30.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:29.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:30.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:29.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:30.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:29.957+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:30.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:30.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:30.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:30.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:30.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:30.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:30.736 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:30.355+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:30.736 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:30.531+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:30.532285+0000) 2024-02-19T20:15:30.993 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:30.734+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:30.735301+0000) 2024-02-19T20:15:30.993 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:30.900+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:30.993 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:30.900+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:30.993 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:30.900+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:30.994 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:30.901+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:30.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:30.814+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:30.815600+0000) 2024-02-19T20:15:30.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:30.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:30.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:30.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:31.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:30.992+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:31.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:30.992+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:31.738 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:31.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:31.738 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:31.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:31.738 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:31.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:31.738 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:31.329+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:31.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:31.533+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:31.534416+0000) 2024-02-19T20:15:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:31.817+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:31.818049+0000) 2024-02-19T20:15:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:31.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:31.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:31.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:32.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:31.985+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:31.736+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:31.737855+0000) 2024-02-19T20:15:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:31.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:31.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:31.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:32.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:31.934+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:32.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:32.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:32.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:32.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:32.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:32.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:32.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:32.356+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:32.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:32.535+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:32.536427+0000) 2024-02-19T20:15:33.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:32.820+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:32.821039+0000) 2024-02-19T20:15:33.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:32.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:33.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:32.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:33.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:32.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:33.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:32.984+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:32.738+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:32.739882+0000) 2024-02-19T20:15:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:32.958+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:32.958+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:32.958+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:33.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:32.959+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:33.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:33.332+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:33.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:33.333+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:33.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:33.333+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:33.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:33.333+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:33.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:33.535+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:33.536610+0000) 2024-02-19T20:15:33.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:33.820+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:33.821206+0000) 2024-02-19T20:15:33.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:33.739+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:33.740109+0000) 2024-02-19T20:15:33.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:33.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:33.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:33.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:33.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:33.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:34.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:33.955+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:34.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:33.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:34.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:33.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:34.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:33.990+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:34.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:33.991+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:34.740 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:15:34.397+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55729 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:15:34.741 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:34.395+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:34.396371+0000) 2024-02-19T20:15:34.741 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:34.394+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:34.395233+0000) 2024-02-19T20:15:34.741 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:34.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:34.741 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:34.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:34.742 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:34.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:34.742 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:34.360+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:34.742 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:34.394+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:34.395893+0000) 2024-02-19T20:15:34.742 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:34.535+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:34.536778+0000) 2024-02-19T20:15:35.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:34.820+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:34.821360+0000) 2024-02-19T20:15:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:34.739+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:34.740404+0000) 2024-02-19T20:15:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:34.964+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:34.964+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:34.964+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:34.964+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:35.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:35.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:35.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:35.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:35.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:35.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:35.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:35.029+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:35.743 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:35.346+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:35.743 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:35.346+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:35.743 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:35.346+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:35.743 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:35.347+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:35.743 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:35.538+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:35.539230+0000) 2024-02-19T20:15:36.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:35.822+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:35.823932+0000) 2024-02-19T20:15:36.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:35.741+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:35.742833+0000) 2024-02-19T20:15:36.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:35.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:36.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:35.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:36.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:35.915+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:36.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:35.916+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:36.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:36.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:36.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:36.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:36.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:36.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:36.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:36.023+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:36.745 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:36.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:36.745 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:36.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:36.746 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:36.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:36.746 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:36.336+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:36.746 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:36.540+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:36.541533+0000) 2024-02-19T20:15:37.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:36.825+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:36.826387+0000) 2024-02-19T20:15:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:36.744+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:36.745349+0000) 2024-02-19T20:15:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:36.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:36.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:36.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:36.906+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:37.315 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:37.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:37.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:37.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:37.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:37.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:37.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:37.067+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:37.748 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:37.314+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:37.748 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:37.314+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:37.749 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:37.314+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:37.749 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:37.315+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:37.749 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:37.542+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:37.543770+0000) 2024-02-19T20:15:38.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:37.827+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:37.828041+0000) 2024-02-19T20:15:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:37.747+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:37.748370+0000) 2024-02-19T20:15:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:37.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:37.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:37.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:37.957+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:38.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:38.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:38.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:38.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:38.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:38.105+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:38.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:38.105+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:38.748 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:38.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:38.749 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:38.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:38.749 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:38.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:38.749 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:38.332+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:38.749 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:38.542+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:38.543920+0000) 2024-02-19T20:15:39.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:38.827+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:38.828213+0000) 2024-02-19T20:15:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:38.747+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:38.748548+0000) 2024-02-19T20:15:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:38.964+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:38.964+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:38.964+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:39.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:38.965+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:39.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:39.117+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:39.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:39.117+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:39.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:39.117+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:39.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:39.117+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:39.749 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:15:39.404+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55770 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:15:39.749 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:39.402+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:39.403199+0000) 2024-02-19T20:15:39.750 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:39.402+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:39.403014+0000) 2024-02-19T20:15:39.750 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:39.343+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:39.750 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:39.343+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:39.750 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:39.343+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:39.750 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:39.343+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:39.750 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:39.401+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:39.402617+0000) 2024-02-19T20:15:39.750 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:39.543+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:39.544091+0000) 2024-02-19T20:15:40.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:39.827+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:39.828379+0000) 2024-02-19T20:15:40.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:39.747+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:39.748769+0000) 2024-02-19T20:15:40.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:39.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:40.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:39.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:40.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:39.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:39.995+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:40.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:40.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:40.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:40.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:40.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:40.107+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:40.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:40.108+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:40.752 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:40.332+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:40.752 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:40.332+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:40.752 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:40.332+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:40.752 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:40.333+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:40.752 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:40.545+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:40.546513+0000) 2024-02-19T20:15:41.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:40.829+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:40.830391+0000) 2024-02-19T20:15:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:40.750+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:40.751770+0000) 2024-02-19T20:15:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:40.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:40.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:40.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:40.991+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:41.325 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:41.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:41.325 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:41.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:41.325 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:41.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:41.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:41.292+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:41.326 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:41.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:41.326 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:41.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:41.326 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:41.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:41.326 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:41.072+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:41.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:41.548+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:41.549847+0000) 2024-02-19T20:15:42.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:41.752+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:41.753936+0000) 2024-02-19T20:15:42.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:42.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:42.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:41.831+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:41.832405+0000) 2024-02-19T20:15:42.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:42.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:42.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:42.004+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:42.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:42.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:42.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:42.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:42.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:42.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:42.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:42.045+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:42.552 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:42.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:42.552 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:42.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:42.552 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:42.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:42.553 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:42.270+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:42.835 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:42.754+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:42.755809+0000) 2024-02-19T20:15:42.835 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:42.550+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:42.551566+0000) 2024-02-19T20:15:43.245 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:43.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:43.245 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:43.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:43.246 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:43.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:43.246 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:43.052+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:43.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:42.834+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:42.835218+0000) 2024-02-19T20:15:43.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:43.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:43.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:43.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:43.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:43.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:43.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:43.015+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:43.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:43.258+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:43.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:43.258+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:43.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:43.258+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:43.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:43.258+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:43.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:43.550+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:43.551761+0000) 2024-02-19T20:15:44.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:43.755+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:43.756021+0000) 2024-02-19T20:15:44.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:43.834+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:43.835379+0000) 2024-02-19T20:15:44.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:43.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:44.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:43.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:44.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:43.999+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:44.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:43.999+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:44.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:44.053+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:44.326 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:44.053+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:44.326 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:44.053+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:44.326 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:44.054+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:44.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:44.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:44.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:44.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:44.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:44.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:44.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:44.212+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:44.756 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:15:44.411+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55809 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:15:44.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:44.408+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:44.409774+0000) 2024-02-19T20:15:44.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:44.409+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:44.410267+0000) 2024-02-19T20:15:44.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:44.409+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:44.410146+0000) 2024-02-19T20:15:44.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:44.550+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:44.551930+0000) 2024-02-19T20:15:45.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:44.755+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:44.756198+0000) 2024-02-19T20:15:45.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:44.834+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:44.835544+0000) 2024-02-19T20:15:45.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:44.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:45.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:44.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:45.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:44.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:45.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:44.981+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:45.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:45.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:45.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:45.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:45.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:45.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:45.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:45.221+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:45.326 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:45.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:45.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:45.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:45.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:45.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:45.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:45.052+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:45.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:45.553+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:45.554329+0000) 2024-02-19T20:15:46.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:45.757+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:45.758421+0000) 2024-02-19T20:15:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:45.837+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:45.838148+0000) 2024-02-19T20:15:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:45.947+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:45.947+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:46.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:45.947+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:46.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:45.948+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:46.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:46.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:46.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:46.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:46.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:46.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:46.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:46.261+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:46.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:46.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:46.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:46.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:46.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:46.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:46.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:46.044+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:46.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:46.555+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:46.556914+0000) 2024-02-19T20:15:47.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:46.760+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:46.761925+0000) 2024-02-19T20:15:47.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:46.839+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:46.840770+0000) 2024-02-19T20:15:47.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:46.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:47.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:46.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:47.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:46.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:47.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:46.913+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:47.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:47.217+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:47.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:47.217+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:47.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:47.217+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:47.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:47.218+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:47.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:47.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:47.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:47.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:47.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:47.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:47.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:47.020+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:47.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:47.558+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:47.558993+0000) 2024-02-19T20:15:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:47.763+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:47.764167+0000) 2024-02-19T20:15:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:48.004+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:48.004+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:48.004+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:48.005+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:47.841+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:47.842615+0000) 2024-02-19T20:15:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:47.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:47.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:47.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:48.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:47.884+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:48.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:48.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:48.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:48.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:48.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:48.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:48.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:48.212+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:48.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:48.558+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:48.559143+0000) 2024-02-19T20:15:49.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:48.763+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:48.764380+0000) 2024-02-19T20:15:49.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:49.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:49.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:49.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:49.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:49.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:49.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:48.841+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:48.842819+0000) 2024-02-19T20:15:49.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:48.876+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:49.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:48.876+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:49.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:48.876+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:49.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:48.877+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:49.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:49.017+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:49.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:49.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:49.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:49.256+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:49.764 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:15:49.418+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55850 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:15:49.765 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:49.415+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:49.416525+0000) 2024-02-19T20:15:49.765 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:49.415+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:49.416844+0000) 2024-02-19T20:15:49.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:49.415+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:49.416305+0000) 2024-02-19T20:15:49.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:49.558+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:49.559296+0000) 2024-02-19T20:15:50.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:49.763+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:49.764566+0000) 2024-02-19T20:15:50.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:50.011+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:50.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:50.011+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:50.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:50.011+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:50.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:50.012+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:50.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:49.842+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:49.843011+0000) 2024-02-19T20:15:50.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:49.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:50.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:49.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:50.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:49.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:50.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:49.881+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:50.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:50.279+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:50.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:50.279+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:50.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:50.279+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:50.327 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:50.280+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:50.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:50.560+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:50.561347+0000) 2024-02-19T20:15:51.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:50.766+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:50.766962+0000) 2024-02-19T20:15:51.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:50.844+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:50.845543+0000) 2024-02-19T20:15:51.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:50.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:51.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:50.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:51.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:50.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:51.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:50.927+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:51.306 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:51.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:51.306 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:51.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:51.307 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:51.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:51.307 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:51.029+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:51.564 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:51.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:51.564 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:51.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:51.564 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:51.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:51.564 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:51.306+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:51.847 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:51.562+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:51.563946+0000) 2024-02-19T20:15:51.848 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:51.768+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:51.769543+0000) 2024-02-19T20:15:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:52.069+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:52.069+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:52.069+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:52.070+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:51.846+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:51.847630+0000) 2024-02-19T20:15:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:51.923+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:52.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:51.923+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:52.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:51.923+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:52.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:51.923+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:52.334+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:52.334+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:52.334+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:52.335+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:52.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:52.565+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:52.566178+0000) 2024-02-19T20:15:53.056 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:52.771+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:52.772411+0000) 2024-02-19T20:15:53.056 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:52.848+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:52.849791+0000) 2024-02-19T20:15:53.056 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:52.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:53.056 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:52.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:53.056 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:52.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:53.057 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:52.875+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:53.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:53.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:53.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:53.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:53.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:53.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:53.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:53.097+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:53.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:53.326+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:53.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:53.326+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:53.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:53.326+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:53.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:53.327+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:53.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:53.565+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:53.566325+0000) 2024-02-19T20:15:53.850 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:53.771+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:53.772614+0000) 2024-02-19T20:15:54.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:54.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:54.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:54.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:54.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:54.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:54.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:54.088+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:53.848+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:53.849968+0000) 2024-02-19T20:15:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:53.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:53.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:53.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:54.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:53.889+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:54.544 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:15:54.424+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55889 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:15:54.545 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:54.422+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:54.423387+0000) 2024-02-19T20:15:54.545 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:54.422+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:54.423958+0000) 2024-02-19T20:15:54.545 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:54.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:54.545 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:54.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:54.546 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:54.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:54.546 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:54.314+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:54.546 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:54.423+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:54.423998+0000) 2024-02-19T20:15:54.846 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:54.565+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:54.566521+0000) 2024-02-19T20:15:54.846 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:54.771+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:54.772844+0000) 2024-02-19T20:15:55.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:54.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:55.097 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:54.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:55.098 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:54.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:55.098 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:54.845+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:55.098 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:54.849+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:54.850128+0000) 2024-02-19T20:15:55.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:55.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:55.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:55.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:55.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:55.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:55.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:55.097+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:55.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:55.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:55.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:55.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:55.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:55.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:55.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:55.364+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:55.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:55.567+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:55.568650+0000) 2024-02-19T20:15:56.056 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:55.773+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:55.774906+0000) 2024-02-19T20:15:56.056 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:55.851+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:55.852080+0000) 2024-02-19T20:15:56.057 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:55.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:56.057 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:55.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:56.057 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:55.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:56.057 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:55.860+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:56.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:56.104+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:56.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:56.104+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:56.328 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:56.104+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:56.328 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:56.105+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:56.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:56.334+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:56.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:56.334+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:56.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:56.334+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:56.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:56.335+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:56.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:56.570+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:56.570975+0000) 2024-02-19T20:15:57.056 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:56.776+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:56.777029+0000) 2024-02-19T20:15:57.057 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:56.853+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:56.854095+0000) 2024-02-19T20:15:57.057 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:56.881+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:57.057 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:56.881+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:57.057 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:56.881+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:57.058 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:56.881+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:57.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:57.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:57.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:57.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:57.328 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:57.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:57.328 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:57.063+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:57.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:57.343+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:57.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:57.343+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:57.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:57.343+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:57.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:57.344+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:57.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:57.573+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:57.574005+0000) 2024-02-19T20:15:58.051 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:57.778+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:57.779219+0000) 2024-02-19T20:15:58.052 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:57.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:58.052 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:57.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:58.052 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:57.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:58.052 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:57.838+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:58.052 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:57.854+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:57.855853+0000) 2024-02-19T20:15:58.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:58.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:58.328 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:58.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:58.328 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:58.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:58.328 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:58.051+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:58.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:58.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:58.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:58.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:58.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:58.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:58.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:58.364+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:58.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:58.573+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:58.574169+0000) 2024-02-19T20:15:59.051 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:58.778+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:58.779383+0000) 2024-02-19T20:15:59.051 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:58.855+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:58.856028+0000) 2024-02-19T20:15:59.051 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:58.877+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:59.051 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:58.877+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:59.052 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:58.877+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:15:59.052 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:58.877+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:59.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:59.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:59.328 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:59.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:59.328 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:59.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:15:59.328 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:59.051+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:15:59.769 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:15:59.433+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55930 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:15:59.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:59.429+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:59.430065+0000) 2024-02-19T20:15:59.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:59.428+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:59.429681+0000) 2024-02-19T20:15:59.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:59.351+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:59.351+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:59.351+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:15:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:59.352+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:15:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:59.428+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:59.429887+0000) 2024-02-19T20:15:59.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:15:59.573+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:59.574339+0000) 2024-02-19T20:16:00.057 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:15:59.778+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:59.779552+0000) 2024-02-19T20:16:00.057 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:00.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:00.057 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:00.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:00.057 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:00.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:00.057 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:00.002+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:00.058 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:59.828+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:00.058 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:59.828+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:00.058 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:59.828+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:00.058 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:59.829+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:00.058 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:15:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:15:59.855+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:59.856127+0000) 2024-02-19T20:16:00.577 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:00.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:00.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:00.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:00.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:00.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:00.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:00.311+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:00.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:00.576+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:00.577368+0000) 2024-02-19T20:16:00.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:00.780+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:00.781614+0000) 2024-02-19T20:16:00.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:00.781+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:00.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:00.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:00.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:00.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:00.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:00.782+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:01.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:00.857+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:00.858115+0000) 2024-02-19T20:16:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:01.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:01.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:01.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:01.010+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:01.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:01.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:01.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:01.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:01.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:01.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:01.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:01.354+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:01.860 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:01.579+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:01.580030+0000) 2024-02-19T20:16:01.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:01.783+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:01.784026+0000) 2024-02-19T20:16:01.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:01.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:01.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:01.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:01.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:01.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:01.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:01.820+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:02.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:01.859+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:01.860201+0000) 2024-02-19T20:16:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:02.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:02.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:02.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:02.023+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:02.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:02.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:02.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:02.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:02.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:02.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:02.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:02.337+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:02.863 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:02.580+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:02.581801+0000) 2024-02-19T20:16:02.863 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:02.784+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:02.785778+0000) 2024-02-19T20:16:02.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:02.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:02.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:02.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:02.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:02.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:02.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:02.797+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:03.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:02.862+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:02.863101+0000) 2024-02-19T20:16:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:03.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:03.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:03.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:03.030+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:03.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:03.376+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:03.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:03.376+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:03.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:03.376+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:03.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:03.377+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:03.834 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:03.580+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:03.582006+0000) 2024-02-19T20:16:03.834 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:03.784+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:03.786013+0000) 2024-02-19T20:16:03.834 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:03.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:03.835 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:03.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:03.835 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:03.831+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:04.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:03.983+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:03.983+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:03.983+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:03.984+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:04.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:03.832+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:04.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:03.862+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:03.863382+0000) 2024-02-19T20:16:04.582 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:16:04.440+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 55969 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:16:04.583 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:04.438+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:04.439001+0000) 2024-02-19T20:16:04.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:04.438+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:04.439681+0000) 2024-02-19T20:16:04.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:04.413+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:04.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:04.413+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:04.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:04.413+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:04.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:04.414+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:04.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:04.437+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:04.438188+0000) 2024-02-19T20:16:04.863 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:04.785+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:04.786174+0000) 2024-02-19T20:16:04.864 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:04.581+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:04.582198+0000) 2024-02-19T20:16:04.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:04.794+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:04.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:04.794+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:04.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:04.794+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:04.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:04.794+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:05.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:04.862+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:04.863593+0000) 2024-02-19T20:16:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:04.964+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:04.964+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:04.964+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:04.965+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:05.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:05.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:05.585 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:05.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:05.585 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:05.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:05.585 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:05.419+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:05.835 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:05.583+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:05.584509+0000) 2024-02-19T20:16:05.835 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:05.787+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:05.788472+0000) 2024-02-19T20:16:06.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:05.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:05.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:05.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:06.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:05.967+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:06.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:05.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:06.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:05.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:06.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:05.833+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:06.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:05.834+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:06.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:05.865+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:05.866093+0000) 2024-02-19T20:16:06.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:06.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:06.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:06.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:06.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:06.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:06.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:06.432+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:06.844 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:06.585+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:06.586714+0000) 2024-02-19T20:16:06.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:06.790+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:06.791242+0000) 2024-02-19T20:16:07.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:06.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:06.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:06.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:07.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:06.987+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:06.843+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:07.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:06.843+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:06.843+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:06.843+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:07.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:06.868+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:06.869656+0000) 2024-02-19T20:16:07.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:07.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:07.589 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:07.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:07.589 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:07.467+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:07.589 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:07.468+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:07.872 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:07.587+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:07.588697+0000) 2024-02-19T20:16:07.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:07.793+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:07.794393+0000) 2024-02-19T20:16:07.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:07.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:07.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:07.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:07.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:07.802+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:07.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:07.802+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:08.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:07.871+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:07.872676+0000) 2024-02-19T20:16:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:07.997+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:07.997+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:07.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:07.998+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:08.589 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:08.449+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:08.589 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:08.449+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:08.589 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:08.449+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:08.589 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:08.449+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:08.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:08.587+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:08.588906+0000) 2024-02-19T20:16:08.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:08.793+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:08.794613+0000) 2024-02-19T20:16:08.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:08.786+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:08.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:08.786+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:08.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:08.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:08.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:08.787+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:09.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:08.871+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:08.872849+0000) 2024-02-19T20:16:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:08.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:08.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:08.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:08.979+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:09.589 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:16:09.447+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56010 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:16:09.589 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:09.445+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:09.446192+0000) 2024-02-19T20:16:09.590 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:09.443+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:09.444753+0000) 2024-02-19T20:16:09.590 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:09.433+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:09.590 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:09.433+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:09.590 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:09.433+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:09.590 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:09.434+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:09.591 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:09.445+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:09.446305+0000) 2024-02-19T20:16:09.873 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:09.588+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:09.589094+0000) 2024-02-19T20:16:09.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:09.793+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:09.794820+0000) 2024-02-19T20:16:09.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:09.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:09.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:09.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:09.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:09.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:09.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:09.792+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:10.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:09.872+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:09.873025+0000) 2024-02-19T20:16:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:09.930+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:09.930+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:09.930+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:10.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:09.930+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:10.592 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:10.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:10.592 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:10.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:10.593 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:10.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:10.593 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:10.451+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:10.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:10.796+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:10.797026+0000) 2024-02-19T20:16:10.876 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:10.591+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:10.592402+0000) 2024-02-19T20:16:10.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:10.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:10.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:10.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:10.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:10.787+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:10.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:10.788+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:11.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:10.874+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:10.875407+0000) 2024-02-19T20:16:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:10.942+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:10.942+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:10.942+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:11.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:10.943+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:11.595 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:11.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:11.595 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:11.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:11.595 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:11.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:11.595 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:11.452+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:11.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:11.594+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:11.594978+0000) 2024-02-19T20:16:11.879 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:11.798+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:11.799029+0000) 2024-02-19T20:16:11.880 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:11.748+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:11.880 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:11.749+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:11.880 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:11.749+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:11.880 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:11.749+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:12.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:11.878+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:11.879105+0000) 2024-02-19T20:16:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:11.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:11.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:11.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:12.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:11.905+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:12.597 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:12.437+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:12.597 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:12.437+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:12.597 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:12.437+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:12.597 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:12.438+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:12.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:12.596+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:12.597187+0000) 2024-02-19T20:16:12.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:12.799+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:12.800809+0000) 2024-02-19T20:16:12.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:12.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:12.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:12.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:12.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:12.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:12.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:12.737+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:13.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:12.881+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:12.882099+0000) 2024-02-19T20:16:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:12.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:12.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:12.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:13.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:12.942+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:13.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:13.422+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:13.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:13.422+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:13.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:13.422+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:13.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:13.423+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:13.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:13.596+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:13.597366+0000) 2024-02-19T20:16:13.882 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:13.800+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:13.801037+0000) 2024-02-19T20:16:13.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:13.695+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:13.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:13.695+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:13.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:13.695+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:13.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:13.695+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:14.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:13.881+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:13.882279+0000) 2024-02-19T20:16:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:13.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:13.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:13.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:13.965+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:14.597 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:16:14.454+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56049 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:16:14.598 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:14.451+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:14.452578+0000) 2024-02-19T20:16:14.598 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:14.451+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:14.452485+0000) 2024-02-19T20:16:14.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:14.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:14.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:14.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:14.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:14.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:14.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:14.421+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:14.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:14.451+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:14.452892+0000) 2024-02-19T20:16:14.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:14.596+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:14.597550+0000) 2024-02-19T20:16:14.883 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:14.800+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:14.801250+0000) 2024-02-19T20:16:14.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:14.685+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:14.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:14.685+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:14.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:14.685+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:14.884 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:14.685+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:15.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:14.881+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:14.882450+0000) 2024-02-19T20:16:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:14.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:14.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:14.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:14.948+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:15.601 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:15.429+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:15.601 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:15.429+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:15.601 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:15.429+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:15.601 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:15.430+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:15.885 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:15.598+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:15.599713+0000) 2024-02-19T20:16:15.886 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:15.802+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:15.803254+0000) 2024-02-19T20:16:15.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:15.684+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:15.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:15.684+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:15.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:15.684+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:15.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:15.684+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:15.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:15.884+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:15.885343+0000) 2024-02-19T20:16:16.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:15.955+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:16.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:15.955+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:15.955+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:16.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:15.956+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:16.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:16.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:16.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:16.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:16.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:16.472+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:16.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:16.473+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:16.887 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:16.600+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:16.601965+0000) 2024-02-19T20:16:16.888 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:16.804+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:16.805209+0000) 2024-02-19T20:16:16.888 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:16.659+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:16.888 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:16.659+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:16.888 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:16.659+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:16.889 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:16.659+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:17.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:16.886+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:16.887682+0000) 2024-02-19T20:16:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:16.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:16.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:16.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:17.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:16.967+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:17.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:17.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:17.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:17.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:17.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:17.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:17.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:17.432+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:17.891 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:17.603+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:17.604139+0000) 2024-02-19T20:16:17.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:17.806+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:17.807020+0000) 2024-02-19T20:16:17.891 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:17.664+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:17.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:17.664+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:17.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:17.664+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:17.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:17.665+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:18.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:17.889+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:17.890878+0000) 2024-02-19T20:16:18.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:17.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:17.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:17.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:17.956+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:18.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:18.429+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:18.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:18.429+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:18.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:18.429+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:18.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:18.430+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:18.891 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:18.603+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:18.604292+0000) 2024-02-19T20:16:18.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:18.806+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:18.807177+0000) 2024-02-19T20:16:18.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:18.624+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:18.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:18.624+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:18.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:18.624+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:18.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:18.625+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:19.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:18.890+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:18.891061+0000) 2024-02-19T20:16:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:18.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:18.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:18.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:18.911+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:19.598 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:16:19.460+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56090 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:16:19.598 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:19.459+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:19.460656+0000) 2024-02-19T20:16:19.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:19.404+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:19.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:19.404+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:19.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:19.404+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:19.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:19.405+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:19.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:19.458+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:19.459051+0000) 2024-02-19T20:16:19.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:19.459+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:19.460319+0000) 2024-02-19T20:16:19.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:19.597+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:19.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:19.597+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:19.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:19.597+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:19.891 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:19.603+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:19.604498+0000) 2024-02-19T20:16:19.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:19.806+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:19.807361+0000) 2024-02-19T20:16:19.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:19.597+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:20.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:19.890+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:19.891239+0000) 2024-02-19T20:16:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:19.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:19.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:19.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:19.950+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:20.606 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:20.565+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:20.607 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:20.565+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:20.607 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:20.565+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:20.607 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:20.565+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:20.607 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:20.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:20.607 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:20.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:20.607 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:20.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:20.608 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:20.359+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:20.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:20.605+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:20.606494+0000) 2024-02-19T20:16:20.893 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:20.808+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:20.809625+0000) 2024-02-19T20:16:21.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:20.892+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:20.893299+0000) 2024-02-19T20:16:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:20.927+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:20.928+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:20.928+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:21.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:20.929+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:21.609 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:21.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:21.609 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:21.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:21.609 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:21.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:21.609 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:21.364+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:21.610 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:21.547+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:21.610 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:21.547+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:21.610 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:21.547+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:21.610 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:21.548+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:21.896 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:21.607+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:21.608857+0000) 2024-02-19T20:16:21.897 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:21.810+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:21.811815+0000) 2024-02-19T20:16:22.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:21.895+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:21.896687+0000) 2024-02-19T20:16:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:21.942+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:21.942+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:21.942+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:21.942+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:22.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:22.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:22.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:22.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:22.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:22.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:22.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:22.350+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:22.613 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:22.574+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:22.613 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:22.574+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:22.613 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:22.574+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:22.613 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:22.575+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:22.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:22.611+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:22.612021+0000) 2024-02-19T20:16:22.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:22.812+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:22.813641+0000) 2024-02-19T20:16:23.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:22.898+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:22.899738+0000) 2024-02-19T20:16:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:22.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:22.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:22.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:22.979+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:23.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:23.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:23.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:23.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:23.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:23.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:23.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:23.379+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:23.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:23.812+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:23.813840+0000) 2024-02-19T20:16:23.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:23.611+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:23.612175+0000) 2024-02-19T20:16:23.900 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:23.598+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:23.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:23.598+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:23.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:23.598+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:23.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:23.598+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:24.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:23.898+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:23.899917+0000) 2024-02-19T20:16:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:23.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:23.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:23.966+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:24.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:23.967+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:24.587 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:16:24.468+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56129 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:16:24.588 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:24.465+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:24.466901+0000) 2024-02-19T20:16:24.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:24.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:24.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:24.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:24.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:24.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:24.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:24.364+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:24.589 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:24.464+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:24.465928+0000) 2024-02-19T20:16:24.589 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:24.465+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:24.466687+0000) 2024-02-19T20:16:24.589 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:24.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:24.589 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:24.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:24.589 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:24.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:24.589 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:24.574+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:24.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:24.611+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:24.612332+0000) 2024-02-19T20:16:24.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:24.813+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:24.814066+0000) 2024-02-19T20:16:25.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:24.899+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:24.900132+0000) 2024-02-19T20:16:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:24.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:24.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:24.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:24.986+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:25.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:25.336+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:25.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:25.336+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:25.615 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:25.336+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:25.615 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:25.337+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:25.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:25.552+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:25.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:25.552+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:25.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:25.552+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:25.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:25.552+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:25.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:25.613+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:25.614300+0000) 2024-02-19T20:16:25.902 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:25.815+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:25.816026+0000) 2024-02-19T20:16:26.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:25.901+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:25.902100+0000) 2024-02-19T20:16:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:25.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:25.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:25.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:25.955+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:26.597 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:26.381+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:26.597 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:26.381+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:26.597 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:26.381+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:26.597 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:26.382+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:26.904 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:26.615+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:26.616949+0000) 2024-02-19T20:16:26.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:26.817+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:26.818049+0000) 2024-02-19T20:16:26.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:26.595+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:26.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:26.595+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:26.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:26.595+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:26.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:26.596+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:27.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:26.903+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:26.904247+0000) 2024-02-19T20:16:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:26.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:26.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:26.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:26.932+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:27.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:27.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:27.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:27.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:27.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:27.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:27.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:27.568+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:27.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:27.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:27.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:27.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:27.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:27.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:27.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:27.351+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:27.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:27.618+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:27.619890+0000) 2024-02-19T20:16:27.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:27.818+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:27.819851+0000) 2024-02-19T20:16:28.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:27.905+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:27.906268+0000) 2024-02-19T20:16:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:27.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:27.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:27.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:27.912+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:28.620 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:28.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:28.620 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:28.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:28.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:28.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:28.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:28.324+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:28.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:28.542+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:28.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:28.542+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:28.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:28.542+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:28.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:28.543+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:28.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:28.619+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:28.620054+0000) 2024-02-19T20:16:28.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:28.819+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:28.820025+0000) 2024-02-19T20:16:29.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:28.905+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:28.906440+0000) 2024-02-19T20:16:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:28.918+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:28.918+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:28.918+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:29.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:28.919+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:29.593 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:16:29.475+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56170 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:16:29.594 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:29.472+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:29.473952+0000) 2024-02-19T20:16:29.594 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:29.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:29.594 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:29.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:29.595 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:29.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:29.595 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:29.313+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:29.595 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:29.473+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:29.474955+0000) 2024-02-19T20:16:29.595 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:29.472+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:29.473658+0000) 2024-02-19T20:16:29.595 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:29.560+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:29.595 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:29.560+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:29.596 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:29.560+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:29.596 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:29.560+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:29.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:29.619+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:29.620260+0000) 2024-02-19T20:16:29.907 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:29.819+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:29.820205+0000) 2024-02-19T20:16:30.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:29.905+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:29.906650+0000) 2024-02-19T20:16:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:29.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:29.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:29.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:29.953+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:30.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:30.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:30.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:30.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:30.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:30.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:30.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:30.313+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:30.823 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:30.621+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:30.622248+0000) 2024-02-19T20:16:30.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:30.565+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:30.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:30.565+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:30.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:30.565+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:30.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:30.565+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:31.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:30.908+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:30.909540+0000) 2024-02-19T20:16:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:30.822+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:30.823721+0000) 2024-02-19T20:16:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:30.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:30.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:30.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:30.980+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:31.625 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:31.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:31.625 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:31.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:31.625 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:31.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:31.625 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:31.331+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:31.625 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:31.564+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:31.626 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:31.564+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:31.626 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:31.564+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:31.626 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:31.565+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:31.912 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:31.623+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:31.624792+0000) 2024-02-19T20:16:31.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:31.825+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:31.826456+0000) 2024-02-19T20:16:32.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:31.911+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:31.912095+0000) 2024-02-19T20:16:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:31.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:31.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:31.981+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:31.981+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:32.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:32.545+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:32.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:32.545+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:32.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:32.545+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:32.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:32.546+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:32.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:32.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:32.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:32.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:32.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:32.312+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:32.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:32.313+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:32.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:32.827+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:32.828446+0000) 2024-02-19T20:16:32.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:32.625+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:32.626886+0000) 2024-02-19T20:16:33.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:32.913+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:32.914323+0000) 2024-02-19T20:16:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:32.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:32.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:32.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:32.977+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:33.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:33.327+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:33.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:33.327+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:33.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:33.327+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:33.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:33.328+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:33.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:33.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:33.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:33.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:33.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:33.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:33.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:33.554+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:33.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:33.827+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:33.828639+0000) 2024-02-19T20:16:33.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:33.626+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:33.627069+0000) 2024-02-19T20:16:34.227 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:33.913+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:33.914500+0000) 2024-02-19T20:16:34.227 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:33.987+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:34.227 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:33.987+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:34.228 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:33.987+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:34.228 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:33.988+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:34.477 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:34.317+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:34.477 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:34.317+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:34.477 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:34.317+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:34.478 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:34.317+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:34.769 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:16:34.482+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56209 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:16:34.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:34.479+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:34.480283+0000) 2024-02-19T20:16:34.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:34.479+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:34.480923+0000) 2024-02-19T20:16:34.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:34.626+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:34.627223+0000) 2024-02-19T20:16:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:34.479+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:34.480278+0000) 2024-02-19T20:16:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:34.561+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:34.561+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:34.561+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:34.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:34.562+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:35.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:34.913+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:34.914672+0000) 2024-02-19T20:16:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:34.827+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:34.828866+0000) 2024-02-19T20:16:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:35.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:35.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:35.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:35.022+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:35.629 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:35.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:35.629 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:35.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:35.630 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:35.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:35.630 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:35.344+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:35.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:35.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:35.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:35.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:35.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:35.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:35.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:35.546+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:35.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:35.628+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:35.629204+0000) 2024-02-19T20:16:35.917 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:35.831+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:35.832395+0000) 2024-02-19T20:16:36.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:35.916+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:35.917028+0000) 2024-02-19T20:16:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:35.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:35.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:35.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:35.989+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:36.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:36.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:36.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:36.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:36.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:36.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:36.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:36.313+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:36.836 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:36.630+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:36.631479+0000) 2024-02-19T20:16:36.836 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:36.581+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:36.836 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:36.581+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:36.836 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:36.581+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:36.837 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:36.582+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:37.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:36.918+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:36.919569+0000) 2024-02-19T20:16:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:36.834+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:36.835905+0000) 2024-02-19T20:16:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:37.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:37.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:37.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:37.020+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:37.563 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:37.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:37.563 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:37.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:37.563 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:37.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:37.564 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:37.291+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:37.839 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:37.632+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:37.633324+0000) 2024-02-19T20:16:37.839 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:37.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:37.839 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:37.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:37.839 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:37.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:37.840 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:37.562+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:38.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:37.921+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:37.922020+0000) 2024-02-19T20:16:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:37.838+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:37.839023+0000) 2024-02-19T20:16:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:38.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:38.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:38.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:38.053+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:38.591 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:38.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:38.591 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:38.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:38.591 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:38.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:38.591 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:38.294+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:38.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:38.838+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:38.839205+0000) 2024-02-19T20:16:38.923 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:38.632+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:38.633513+0000) 2024-02-19T20:16:38.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:38.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:38.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:38.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:38.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:38.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:38.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:38.590+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:39.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:38.921+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:38.922173+0000) 2024-02-19T20:16:39.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:39.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:39.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:39.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:39.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:39.092+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:39.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:39.093+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:39.519 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:16:39.489+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56250 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:16:39.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:39.486+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:39.487504+0000) 2024-02-19T20:16:39.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:39.486+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:39.487198+0000) 2024-02-19T20:16:39.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:39.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:39.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:39.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:39.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:39.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:39.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:39.266+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:39.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:39.487+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:39.488362+0000) 2024-02-19T20:16:39.839 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:39.632+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:39.633720+0000) 2024-02-19T20:16:39.840 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:39.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:39.840 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:39.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:39.840 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:39.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:39.840 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:39.558+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:40.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:39.921+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:39.922343+0000) 2024-02-19T20:16:40.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:39.838+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:39.839352+0000) 2024-02-19T20:16:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:40.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:40.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:40.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:40.059+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:40.548 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:40.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:40.549 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:40.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:40.549 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:40.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:40.549 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:40.283+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:40.843 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:40.636+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:40.637242+0000) 2024-02-19T20:16:40.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:40.547+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:40.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:40.547+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:40.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:40.547+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:40.844 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:40.547+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:41.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:40.923+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:40.924369+0000) 2024-02-19T20:16:41.110 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:40.841+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:40.842902+0000) 2024-02-19T20:16:41.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:41.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:41.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:41.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:41.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:41.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:41.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:41.260+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:41.423 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:41.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:41.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:41.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:41.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:41.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:41.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:41.109+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:41.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:41.639+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:41.640061+0000) 2024-02-19T20:16:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:41.566+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:41.566+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:41.566+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:41.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:41.566+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:42.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:41.844+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:41.845432+0000) 2024-02-19T20:16:42.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:41.925+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:41.926474+0000) 2024-02-19T20:16:42.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:42.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:42.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:42.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:42.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:42.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:42.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:42.227+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:42.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:42.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:42.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:42.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:42.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:42.157+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:42.425 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:42.158+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:42.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:42.641+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:42.642486+0000) 2024-02-19T20:16:42.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:42.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:42.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:42.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:42.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:42.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:42.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:42.555+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:43.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:42.928+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:42.928976+0000) 2024-02-19T20:16:43.143 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:42.847+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:42.848350+0000) 2024-02-19T20:16:43.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:43.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:43.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:43.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:43.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:43.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:43.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:43.239+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:43.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:43.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:43.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:43.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:43.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:43.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:43.424 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:43.142+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:43.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:43.641+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:43.642657+0000) 2024-02-19T20:16:43.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:43.534+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:43.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:43.534+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:43.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:43.534+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:43.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:43.535+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:44.061 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:43.847+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:43.848549+0000) 2024-02-19T20:16:44.062 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:43.928+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:43.929146+0000) 2024-02-19T20:16:44.333 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:44.250+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:44.333 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:44.250+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:44.333 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:44.250+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:44.333 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:44.251+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:44.333 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:44.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:44.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:44.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:44.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:44.141+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:44.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:44.142+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:44.616 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:16:44.496+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56289 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:16:44.616 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:44.494+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:44.495312+0000) 2024-02-19T20:16:44.616 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:44.493+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:44.494879+0000) 2024-02-19T20:16:44.617 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:44.494+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:44.495669+0000) 2024-02-19T20:16:44.617 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:44.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:44.617 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:44.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:44.617 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:44.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:44.617 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:44.530+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:44.929 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:44.641+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:44.642838+0000) 2024-02-19T20:16:44.930 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:44.847+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:44.848747+0000) 2024-02-19T20:16:45.252 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:44.928+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:44.929326+0000) 2024-02-19T20:16:45.252 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:45.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:45.253 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:45.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:45.253 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:45.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:45.253 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:45.107+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:45.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:45.488+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:45.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:45.488+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:45.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:45.488+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:45.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:45.488+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:45.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:45.251+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:45.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:45.251+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:45.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:45.251+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:45.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:45.252+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:45.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:45.643+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:45.644920+0000) 2024-02-19T20:16:46.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:45.849+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:45.850860+0000) 2024-02-19T20:16:46.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:45.930+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:45.931514+0000) 2024-02-19T20:16:46.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:46.229+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:46.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:46.229+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:46.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:46.229+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:46.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:46.230+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:46.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:46.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:46.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:46.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:46.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:46.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:46.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:46.140+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:46.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:46.646+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:46.647535+0000) 2024-02-19T20:16:46.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:46.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:46.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:46.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:46.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:46.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:46.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:46.505+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:47.151 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:46.932+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:46.933507+0000) 2024-02-19T20:16:47.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:46.851+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:46.852904+0000) 2024-02-19T20:16:47.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:47.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:47.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:47.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:47.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:47.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:47.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:47.151+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:47.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:47.217+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:47.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:47.217+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:47.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:47.217+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:47.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:47.217+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:47.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:47.649+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:47.650730+0000) 2024-02-19T20:16:47.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:47.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:47.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:47.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:47.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:47.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:47.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:47.502+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:48.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:47.934+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:47.935487+0000) 2024-02-19T20:16:48.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:47.854+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:47.855261+0000) 2024-02-19T20:16:48.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:48.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:48.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:48.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:48.427 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:48.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:48.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:48.140+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:48.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:48.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:48.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:48.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:48.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:48.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:48.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:48.201+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:48.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:48.649+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:48.650920+0000) 2024-02-19T20:16:48.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:48.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:48.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:48.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:48.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:48.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:48.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:48.479+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:49.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:48.854+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:48.855415+0000) 2024-02-19T20:16:49.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:48.934+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:48.935683+0000) 2024-02-19T20:16:49.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:49.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:49.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:49.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:49.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:49.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:49.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:49.237+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:49.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:49.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:49.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:49.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:49.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:49.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:49.429 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:49.119+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:49.769 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:16:49.503+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56330 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:16:49.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:49.500+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:49.501645+0000) 2024-02-19T20:16:49.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:49.499+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:49.500952+0000) 2024-02-19T20:16:49.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:49.650+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:49.651068+0000) 2024-02-19T20:16:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:49.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:49.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:49.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:49.474+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:49.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:49.500+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:49.501814+0000) 2024-02-19T20:16:50.151 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:49.854+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:49.855588+0000) 2024-02-19T20:16:50.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:49.934+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:49.935873+0000) 2024-02-19T20:16:50.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:50.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:50.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:50.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:50.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:50.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:50.428 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:50.151+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:50.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:50.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:50.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:50.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:50.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:50.257+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:50.429 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:50.258+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:50.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:50.652+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:50.653069+0000) 2024-02-19T20:16:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:50.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:50.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:50.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:50.488+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:51.111 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:50.857+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:50.858905+0000) 2024-02-19T20:16:51.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:50.936+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:50.937962+0000) 2024-02-19T20:16:51.430 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:51.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:51.430 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:51.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:51.431 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:51.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:51.431 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:51.111+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:51.431 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:51.295+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:51.431 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:51.295+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:51.431 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:51.295+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:51.432 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:51.296+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:51.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:51.653+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:51.655436+0000) 2024-02-19T20:16:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:51.534+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:51.534+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:51.534+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:51.535+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:52.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:51.938+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:51.940815+0000) 2024-02-19T20:16:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:51.859+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:51.861549+0000) 2024-02-19T20:16:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:52.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:52.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:52.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:52.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:52.106+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:52.657 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:52.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:52.657 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:52.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:52.658 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:52.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:52.658 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:52.528+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:52.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:52.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:52.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:52.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:52.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:52.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:52.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:52.338+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:52.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:52.656+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:52.657348+0000) 2024-02-19T20:16:52.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:52.862+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:52.863443+0000) 2024-02-19T20:16:53.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:52.943+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:52.944138+0000) 2024-02-19T20:16:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:53.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:53.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:53.110+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:53.110+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:53.657 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:53.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:53.658 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:53.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:53.658 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:53.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:53.658 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:53.523+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:53.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:53.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:53.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:53.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:53.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:53.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:53.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:53.355+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:53.659 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:53.656+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:53.657519+0000) 2024-02-19T20:16:53.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:53.862+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:53.863711+0000) 2024-02-19T20:16:54.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:53.943+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:53.944323+0000) 2024-02-19T20:16:54.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:54.120+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:54.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:54.120+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:54.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:54.120+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:54.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:54.121+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:54.627 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:16:54.509+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56369 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:16:54.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:54.507+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:54.508515+0000) 2024-02-19T20:16:54.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:54.387+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:54.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:54.387+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:54.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:54.387+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:54.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:54.388+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:54.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:54.507+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:54.508172+0000) 2024-02-19T20:16:54.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:54.506+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:54.507590+0000) 2024-02-19T20:16:54.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:54.514+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:54.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:54.514+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:54.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:54.514+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:54.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:54.514+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:54.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:54.656+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:54.657684+0000) 2024-02-19T20:16:54.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:54.862+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:54.863940+0000) 2024-02-19T20:16:55.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:54.943+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:54.944509+0000) 2024-02-19T20:16:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:55.087+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:55.087+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:55.087+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:55.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:55.088+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:55.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:55.383+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:55.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:55.383+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:55.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:55.383+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:55.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:55.384+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:55.662 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:55.538+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:55.662 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:55.538+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:55.662 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:55.538+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:55.662 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:55.539+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:55.948 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:55.659+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:55.660918+0000) 2024-02-19T20:16:55.948 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:55.864+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:55.865916+0000) 2024-02-19T20:16:56.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:55.946+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:55.947883+0000) 2024-02-19T20:16:56.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:56.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:56.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:56.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:56.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:56.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:56.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:56.045+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:56.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:56.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:56.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:56.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:56.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:56.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:56.522 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:56.351+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:56.868 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:56.662+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:56.663445+0000) 2024-02-19T20:16:56.868 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:56.521+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:56.868 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:56.521+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:56.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:56.521+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:56.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:56.521+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:57.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:56.950+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:56.951365+0000) 2024-02-19T20:16:57.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:56.867+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:56.868168+0000) 2024-02-19T20:16:57.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:57.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:57.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:57.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:57.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:57.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:57.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:57.052+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:57.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:57.306+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:57.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:57.306+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:57.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:57.306+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:57.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:57.307+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:57.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:57.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:57.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:57.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:57.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:57.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:57.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:57.487+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:57.953 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:57.870+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:57.871408+0000) 2024-02-19T20:16:57.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:57.664+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:57.665447+0000) 2024-02-19T20:16:58.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:57.952+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:57.953360+0000) 2024-02-19T20:16:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:58.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:58.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:58.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:58.037+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:58.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:58.325+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:58.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:58.325+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:58.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:58.325+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:58.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:58.326+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:58.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:58.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:58.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:58.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:58.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:58.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:58.667 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:58.453+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:58.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:58.664+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:58.665635+0000) 2024-02-19T20:16:58.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:58.870+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:58.871581+0000) 2024-02-19T20:16:59.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:58.952+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:58.953539+0000) 2024-02-19T20:16:59.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:59.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:59.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:59.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:59.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:59.024+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:16:59.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:59.025+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:16:59.635 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:16:59.516+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56410 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:16:59.636 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:59.514+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:59.515785+0000) 2024-02-19T20:16:59.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:59.351+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:59.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:59.351+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:59.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:59.351+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:16:59.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:59.351+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:59.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:59.514+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:59.515614+0000) 2024-02-19T20:16:59.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:59.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:59.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:59.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:59.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:59.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:16:59.638 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:59.436+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:16:59.638 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:59.513+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:59.514752+0000) 2024-02-19T20:16:59.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:16:59.664+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:59.665853+0000) 2024-02-19T20:16:59.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:16:59.871+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:59.872040+0000) 2024-02-19T20:17:00.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:16:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:16:59.953+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:59.954016+0000) 2024-02-19T20:17:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:00.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:00.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:00.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:00.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:00.061+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:00.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:00.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:00.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:00.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:00.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:00.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:00.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:00.453+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:00.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:00.343+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:00.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:00.343+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:00.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:00.343+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:00.670 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:00.344+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:00.670 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:00.667+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:00.668325+0000) 2024-02-19T20:17:00.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:00.873+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:00.874026+0000) 2024-02-19T20:17:01.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:00.956+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:00.957318+0000) 2024-02-19T20:17:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:01.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:01.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:01.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:01.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:01.046+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:01.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:01.390+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:01.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:01.390+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:01.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:01.390+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:01.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:01.391+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:01.671 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:01.485+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:01.672 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:01.485+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:01.672 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:01.485+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:01.672 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:01.485+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:01.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:01.669+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:01.670905+0000) 2024-02-19T20:17:01.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:01.876+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:01.877130+0000) 2024-02-19T20:17:02.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:01.959+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:01.960764+0000) 2024-02-19T20:17:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:02.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:02.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:02.059+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:02.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:02.060+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:02.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:02.507+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:02.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:02.507+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:02.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:02.507+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:02.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:02.508+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:02.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:02.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:02.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:02.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:02.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:02.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:02.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:02.396+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:02.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:02.672+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:02.673181+0000) 2024-02-19T20:17:02.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:02.878+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:02.879824+0000) 2024-02-19T20:17:03.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:02.962+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:02.963861+0000) 2024-02-19T20:17:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:03.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:03.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:03.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:03.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:03.038+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:03.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:03.534+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:03.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:03.534+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:03.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:03.534+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:03.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:03.535+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:03.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:03.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:03.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:03.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:03.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:03.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:03.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:03.422+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:03.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:03.672+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:03.673363+0000) 2024-02-19T20:17:03.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:03.879+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:03.880015+0000) 2024-02-19T20:17:04.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:03.963+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:03.964061+0000) 2024-02-19T20:17:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:03.993+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:03.993+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:03.993+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:04.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:03.993+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:04.644 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:17:04.523+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56450 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:17:04.645 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:04.521+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:04.522415+0000) 2024-02-19T20:17:04.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:04.391+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:04.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:04.391+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:04.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:04.391+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:04.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:04.392+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:04.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:04.521+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:04.522552+0000) 2024-02-19T20:17:04.646 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:04.520+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:04.521775+0000) 2024-02-19T20:17:04.646 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:04.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:04.646 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:04.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:04.646 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:04.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:04.646 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:04.573+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:04.947 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:04.672+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:04.673645+0000) 2024-02-19T20:17:04.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:04.879+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:04.880207+0000) 2024-02-19T20:17:05.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:04.963+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:04.964243+0000) 2024-02-19T20:17:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:04.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:04.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:04.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:04.955+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:05.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:05.362+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:05.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:05.362+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:05.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:05.362+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:05.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:05.363+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:05.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:05.567+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:05.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:05.567+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:05.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:05.567+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:05.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:05.567+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:05.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:05.674+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:05.675619+0000) 2024-02-19T20:17:05.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:05.882+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:05.883707+0000) 2024-02-19T20:17:05.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:05.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:05.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:05.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:05.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:05.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:06.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:05.939+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:06.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:05.965+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:05.966848+0000) 2024-02-19T20:17:06.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:06.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:06.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:06.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:06.354+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:06.355+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:06.885 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:06.677+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:06.678968+0000) 2024-02-19T20:17:06.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:06.518+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:06.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:06.518+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:06.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:06.518+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:06.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:06.519+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:07.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:06.968+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:06.969002+0000) 2024-02-19T20:17:07.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:06.884+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:06.885764+0000) 2024-02-19T20:17:07.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:06.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:07.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:06.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:07.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:06.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:07.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:06.946+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:07.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:07.345+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:07.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:07.345+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:07.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:07.345+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:07.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:07.346+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:07.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:07.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:07.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:07.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:07.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:07.476+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:07.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:07.681+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:07.682217+0000) 2024-02-19T20:17:07.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:07.886+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:07.887777+0000) 2024-02-19T20:17:08.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:07.971+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:07.972353+0000) 2024-02-19T20:17:08.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:07.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:08.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:07.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:08.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:07.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:08.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:07.950+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:08.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:08.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:08.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:08.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:08.332+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:08.442+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:08.442+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:08.442+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:08.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:08.442+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:08.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:08.681+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:08.682405+0000) 2024-02-19T20:17:08.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:08.886+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:08.887971+0000) 2024-02-19T20:17:09.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:08.971+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:08.972501+0000) 2024-02-19T20:17:09.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:08.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:09.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:08.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:09.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:08.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:09.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:08.955+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:09.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:09.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:09.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:09.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:09.306+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:09.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:09.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:09.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:09.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:09.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:09.424+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:09.888 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:17:09.531+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56490 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:17:09.888 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:09.528+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:09.529711+0000) 2024-02-19T20:17:09.889 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:09.528+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:09.529457+0000) 2024-02-19T20:17:09.889 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:09.528+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:09.529920+0000) 2024-02-19T20:17:09.889 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:09.681+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:09.682596+0000) 2024-02-19T20:17:10.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:09.971+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:09.972678+0000) 2024-02-19T20:17:10.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:09.887+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:09.888147+0000) 2024-02-19T20:17:10.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:09.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:10.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:09.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:10.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:09.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:10.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:09.960+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:10.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:10.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:10.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:10.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:10.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:10.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:10.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:10.440+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:10.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:10.317+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:10.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:10.317+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:10.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:10.317+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:10.521 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:10.318+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:10.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:10.683+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:10.684727+0000) 2024-02-19T20:17:10.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:10.889+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:10.890655+0000) 2024-02-19T20:17:10.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:10.955+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:10.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:10.955+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:10.958 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:10.955+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:11.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:10.956+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:11.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:10.974+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:10.975088+0000) 2024-02-19T20:17:11.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:11.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:11.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:11.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:11.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:11.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:11.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:11.325+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:11.456+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:11.456+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:11.456+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:11.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:11.457+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:11.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:11.685+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:11.686828+0000) 2024-02-19T20:17:11.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:11.891+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:11.892716+0000) 2024-02-19T20:17:12.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:11.976+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:11.977234+0000) 2024-02-19T20:17:12.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:11.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:12.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:11.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:12.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:11.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:12.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:11.985+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:12.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:12.298+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:12.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:12.298+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:12.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:12.298+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:12.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:12.299+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:12.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:12.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:12.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:12.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:12.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:12.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:12.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:12.432+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:12.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:12.687+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:12.688907+0000) 2024-02-19T20:17:12.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:12.894+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:12.895140+0000) 2024-02-19T20:17:12.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:12.940+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:12.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:12.940+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:12.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:12.940+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:13.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:12.941+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:13.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:12.978+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:12.979434+0000) 2024-02-19T20:17:13.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:13.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:13.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:13.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:13.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:13.340+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:13.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:13.341+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:13.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:13.425+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:13.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:13.425+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:13.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:13.425+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:13.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:13.426+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:13.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:13.688+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:13.689061+0000) 2024-02-19T20:17:14.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:13.978+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:13.979615+0000) 2024-02-19T20:17:14.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:13.894+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:13.895291+0000) 2024-02-19T20:17:14.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:13.928+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:14.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:13.928+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:14.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:13.928+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:14.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:13.929+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:14.446 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:14.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:14.446 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:14.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:14.446 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:14.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:14.446 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:14.314+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:14.769 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:17:14.539+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56529 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:17:14.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:14.535+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:14.536974+0000) 2024-02-19T20:17:14.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:14.536+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:14.537095+0000) 2024-02-19T20:17:14.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:14.688+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:14.689215+0000) 2024-02-19T20:17:14.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:14.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:14.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:14.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:14.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:14.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:14.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:14.455+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:14.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:14.535+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:14.536652+0000) 2024-02-19T20:17:15.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:14.978+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:14.979811+0000) 2024-02-19T20:17:15.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:14.894+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:14.895477+0000) 2024-02-19T20:17:15.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:14.946+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:15.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:14.946+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:15.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:14.946+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:15.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:14.947+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:15.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:15.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:15.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:15.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:15.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:15.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:15.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:15.291+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:15.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:15.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:15.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:15.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:15.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:15.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:15.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:15.471+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:15.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:15.896+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:15.897549+0000) 2024-02-19T20:17:15.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:15.690+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:15.691369+0000) 2024-02-19T20:17:16.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:15.981+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:15.982505+0000) 2024-02-19T20:17:16.252 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:15.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:16.252 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:15.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:16.252 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:15.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:16.252 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:15.977+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:16.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:16.250+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:16.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:16.250+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:16.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:16.250+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:16.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:16.251+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:16.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:16.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:16.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:16.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:16.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:16.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:16.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:16.475+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:16.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:16.692+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:16.693689+0000) 2024-02-19T20:17:16.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:16.898+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:16.899967+0000) 2024-02-19T20:17:17.240 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:16.984+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:16.985092+0000) 2024-02-19T20:17:17.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:16.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:17.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:16.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:17.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:16.977+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:17.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:16.978+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:17.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:17.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:17.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:17.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:17.475+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:17.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:17.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:17.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:17.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:17.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:17.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:17.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:17.240+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:17.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:17.695+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:17.696016+0000) 2024-02-19T20:17:17.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:17.901+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:17.901972+0000) 2024-02-19T20:17:18.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:17.986+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:17.987540+0000) 2024-02-19T20:17:18.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:18.017+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:18.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:18.017+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:18.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:18.017+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:18.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:18.018+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:18.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:18.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:18.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:18.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:18.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:18.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:18.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:18.226+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:18.902 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:18.695+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:18.696192+0000) 2024-02-19T20:17:18.902 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:18.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:18.902 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:18.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:18.903 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:18.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:18.903 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:18.520+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:19.222 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:18.901+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:18.902180+0000) 2024-02-19T20:17:19.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:19.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:19.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:19.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:19.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:19.003+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:19.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:19.004+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:19.223 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:18.986+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:18.987709+0000) 2024-02-19T20:17:19.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:19.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:19.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:19.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:19.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:19.221+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:19.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:19.222+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:19.902 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:17:19.545+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56569 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:17:19.903 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:19.542+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:19.543879+0000) 2024-02-19T20:17:19.903 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:19.695+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:19.696383+0000) 2024-02-19T20:17:19.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:19.543+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:19.544360+0000) 2024-02-19T20:17:19.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:19.901+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:19.902394+0000) 2024-02-19T20:17:19.904 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:19.525+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:19.904 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:19.525+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:19.904 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:19.525+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:19.904 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:19.526+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:19.904 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:19.543+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:19.544327+0000) 2024-02-19T20:17:20.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:19.986+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:19.987907+0000) 2024-02-19T20:17:20.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:20.000+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:20.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:20.000+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:20.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:20.000+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:20.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:20.001+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:20.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:20.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:20.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:20.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:20.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:20.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:20.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:20.202+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:20.904 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:20.697+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:20.698432+0000) 2024-02-19T20:17:20.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:20.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:20.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:20.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:20.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:20.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:20.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:20.559+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:21.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:20.989+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:20.990496+0000) 2024-02-19T20:17:21.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:20.903+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:20.904553+0000) 2024-02-19T20:17:21.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:21.011+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:21.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:21.011+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:21.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:21.011+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:21.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:21.012+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:21.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:21.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:21.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:21.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:21.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:21.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:21.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:21.179+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:21.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:21.701+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:21.702073+0000) 2024-02-19T20:17:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:21.516+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:21.516+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:21.516+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:21.516+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:22.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:21.992+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:21.993095+0000) 2024-02-19T20:17:22.263 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:22.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:22.263 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:22.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:22.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:22.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:22.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:22.144+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:22.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:21.906+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:21.907150+0000) 2024-02-19T20:17:22.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:22.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:22.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:22.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:22.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:22.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:22.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:22.002+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:22.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:22.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:22.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:22.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:22.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:22.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:22.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:22.511+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:22.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:22.703+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:22.704495+0000) 2024-02-19T20:17:22.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:22.908+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:22.909478+0000) 2024-02-19T20:17:22.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:22.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:23.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:22.995+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:22.996448+0000) 2024-02-19T20:17:23.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:22.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:23.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:22.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:23.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:22.954+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:23.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:23.179+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:23.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:23.179+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:23.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:23.179+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:23.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:23.180+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:23.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:23.703+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:23.704698+0000) 2024-02-19T20:17:23.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:23.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:23.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:23.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:23.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:23.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:23.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:23.513+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:24.066 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:23.995+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:23.996646+0000) 2024-02-19T20:17:24.066 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:23.908+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:23.909722+0000) 2024-02-19T20:17:24.066 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:23.943+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:24.066 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:23.943+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:24.067 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:23.943+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:24.067 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:23.944+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:24.330 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:24.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:24.330 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:24.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:24.331 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:24.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:24.331 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:24.204+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:24.671 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:17:24.552+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56609 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:17:24.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:24.550+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:24.551332+0000) 2024-02-19T20:17:24.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:24.550+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:24.551336+0000) 2024-02-19T20:17:24.672 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:24.550+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:24.551694+0000) 2024-02-19T20:17:24.672 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:24.552+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:24.672 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:24.552+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:24.672 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:24.552+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:24.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:24.553+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:24.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:24.703+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:24.704905+0000) 2024-02-19T20:17:24.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:24.908+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:24.909938+0000) 2024-02-19T20:17:24.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:24.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:24.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:24.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:24.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:24.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:25.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:24.955+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:25.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:24.995+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:24.996839+0000) 2024-02-19T20:17:25.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:25.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:25.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:25.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:25.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:25.155+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:25.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:25.156+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:25.908 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:25.706+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:25.707048+0000) 2024-02-19T20:17:25.908 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:25.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:25.908 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:25.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:25.908 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:25.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:25.909 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:25.529+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:26.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:25.997+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:25.998906+0000) 2024-02-19T20:17:26.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:25.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:26.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:25.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:26.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:25.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:26.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:25.907+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:26.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:25.910+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:25.911766+0000) 2024-02-19T20:17:26.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:26.160+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:26.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:26.160+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:26.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:26.160+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:26.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:26.162+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:26.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:26.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:26.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:26.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:26.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:26.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:26.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:26.553+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:27.001 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:26.709+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:26.710546+0000) 2024-02-19T20:17:27.001 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:26.912+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:26.913772+0000) 2024-02-19T20:17:27.001 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:26.913+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:27.001 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:26.913+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:27.001 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:26.913+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:27.002 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:26.913+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:27.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:26.999+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:27.000886+0000) 2024-02-19T20:17:27.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:27.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:27.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:27.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:27.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:27.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:27.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:27.203+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:27.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:27.711+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:27.712721+0000) 2024-02-19T20:17:27.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:27.535+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:27.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:27.535+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:27.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:27.535+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:27.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:27.536+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:28.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:28.001+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:28.002970+0000) 2024-02-19T20:17:28.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:27.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:28.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:27.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:28.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:27.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:28.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:27.881+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:28.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:27.916+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:27.917218+0000) 2024-02-19T20:17:28.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:28.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:28.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:28.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:28.457 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:28.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:28.457 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:28.184+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:28.713 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:28.570+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:28.713 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:28.570+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:28.713 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:28.570+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:28.713 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:28.570+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:29.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:28.711+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:28.712907+0000) 2024-02-19T20:17:29.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:28.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:29.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:28.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:29.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:28.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:29.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:28.908+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:29.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:28.916+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:28.917384+0000) 2024-02-19T20:17:29.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:29.002+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:29.003135+0000) 2024-02-19T20:17:29.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:29.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:29.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:29.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:29.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:29.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:29.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:29.157+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:29.868 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:17:29.559+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56650 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:17:29.868 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:29.557+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:29.557991+0000) 2024-02-19T20:17:29.869 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:29.712+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:29.713092+0000) 2024-02-19T20:17:29.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:29.556+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:29.557755+0000) 2024-02-19T20:17:29.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:29.867+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:29.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:29.867+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:29.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:29.867+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:29.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:29.551+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:29.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:29.551+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:29.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:29.551+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:29.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:29.551+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:29.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:29.557+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:29.558301+0000) 2024-02-19T20:17:30.144 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:30.002+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:30.003309+0000) 2024-02-19T20:17:30.145 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:29.867+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:30.145 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:29.916+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:29.917527+0000) 2024-02-19T20:17:30.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:30.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:30.457 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:30.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:30.457 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:30.143+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:30.457 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:30.144+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:30.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:30.595+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:30.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:30.595+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:30.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:30.595+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:30.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:30.596+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:31.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:30.714+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:30.715690+0000) 2024-02-19T20:17:31.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:30.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:31.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:30.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:31.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:30.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:31.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:30.859+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:31.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:30.919+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:30.920077+0000) 2024-02-19T20:17:31.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:31.005+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:31.006881+0000) 2024-02-19T20:17:31.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:31.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:31.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:31.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:31.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:31.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:31.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:31.192+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:31.867 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:31.716+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:31.717935+0000) 2024-02-19T20:17:31.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:31.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:31.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:31.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:31.868 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:31.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:31.868 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:31.548+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:32.223 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:32.009+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:32.010396+0000) 2024-02-19T20:17:32.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:31.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:32.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:31.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:32.224 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:31.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:32.224 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:31.867+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:32.224 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:31.921+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:31.922107+0000) 2024-02-19T20:17:32.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:32.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:32.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:32.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:32.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:32.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:32.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:32.223+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:32.882 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:32.719+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:32.720248+0000) 2024-02-19T20:17:32.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:32.576+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:32.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:32.576+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:32.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:32.576+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:32.883 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:32.577+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:33.212 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:33.011+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:33.012737+0000) 2024-02-19T20:17:33.212 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:32.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:33.212 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:32.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:33.212 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:32.881+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:33.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:32.881+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:33.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:32.923+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:32.924093+0000) 2024-02-19T20:17:33.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:33.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:33.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:33.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:33.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:33.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:33.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:33.212+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:33.848 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:33.719+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:33.720393+0000) 2024-02-19T20:17:33.848 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:33.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:33.848 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:33.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:33.849 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:33.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:33.849 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:33.556+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:34.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:34.011+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:34.012935+0000) 2024-02-19T20:17:34.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:33.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:34.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:33.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:34.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:33.846+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:34.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:33.847+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:34.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:33.923+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:33.924253+0000) 2024-02-19T20:17:34.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:34.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:34.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:34.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:34.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:34.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:34.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:34.261+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:34.891 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:17:34.566+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56690 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:17:34.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:34.563+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:34.564299+0000) 2024-02-19T20:17:34.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:34.564+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:34.565198+0000) 2024-02-19T20:17:34.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:34.719+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:34.720604+0000) 2024-02-19T20:17:34.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:34.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:34.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:34.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:34.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:34.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:34.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:34.550+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:34.893 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:34.563+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:34.564168+0000) 2024-02-19T20:17:35.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:35.012+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:35.013118+0000) 2024-02-19T20:17:35.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:34.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:35.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:34.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:35.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:34.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:35.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:34.890+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:35.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:34.923+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:34.924406+0000) 2024-02-19T20:17:35.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:35.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:35.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:35.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:35.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:35.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:35.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:35.216+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:35.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:35.721+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:35.722706+0000) 2024-02-19T20:17:35.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:35.501+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:35.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:35.501+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:35.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:35.501+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:35.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:35.501+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:36.203 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:36.014+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:36.014998+0000) 2024-02-19T20:17:36.203 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:35.892+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:36.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:35.892+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:36.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:35.892+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:36.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:35.893+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:36.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:35.925+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:35.926977+0000) 2024-02-19T20:17:36.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:36.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:36.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:36.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:36.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:36.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:36.464 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:36.203+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:36.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:36.483+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:36.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:36.483+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:36.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:36.483+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:36.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:36.483+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:37.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:36.725+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:36.726043+0000) 2024-02-19T20:17:37.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:36.856+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:37.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:36.856+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:37.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:36.856+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:37.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:36.857+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:37.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:36.927+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:36.928949+0000) 2024-02-19T20:17:37.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:37.016+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:37.017618+0000) 2024-02-19T20:17:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:37.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:37.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:37.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:37.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:37.214+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:37.729 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:37.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:37.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:37.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:37.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:37.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:37.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:37.472+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:38.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:37.728+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:37.729425+0000) 2024-02-19T20:17:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:37.851+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:37.851+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:37.851+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:37.852+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:37.930+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:37.931338+0000) 2024-02-19T20:17:38.339 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:38.247+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:38.339 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:38.247+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:38.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:38.247+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:38.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:38.247+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:38.340 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:38.018+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:38.019736+0000) 2024-02-19T20:17:38.729 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:38.481+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:38.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:38.481+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:38.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:38.481+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:38.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:38.481+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:39.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:38.728+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:38.729610+0000) 2024-02-19T20:17:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:38.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:38.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:38.842+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:38.843+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:38.931+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:38.931496+0000) 2024-02-19T20:17:39.339 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:39.018+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:39.019919+0000) 2024-02-19T20:17:39.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:39.240+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:39.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:39.240+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:39.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:39.240+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:39.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:39.240+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:39.592 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:17:39.572+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56731 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:17:39.592 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:39.570+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:39.572046+0000) 2024-02-19T20:17:39.593 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:39.569+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:39.571325+0000) 2024-02-19T20:17:39.593 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:39.516+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:39.593 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:39.516+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:39.593 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:39.516+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:39.593 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:39.517+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:39.593 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:39.570+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:39.572162+0000) 2024-02-19T20:17:39.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:39.727+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:39.729765+0000) 2024-02-19T20:17:39.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:39.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:39.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:39.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:39.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:39.799+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:39.932 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:39.800+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:40.210 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:39.930+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:39.931741+0000) 2024-02-19T20:17:40.211 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:40.018+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:40.020124+0000) 2024-02-19T20:17:40.464 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:40.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:40.464 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:40.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:40.464 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:40.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:40.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:40.210+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:40.732 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:40.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:40.732 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:40.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:40.733 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:40.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:40.733 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:40.548+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:41.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:40.731+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:40.732286+0000) 2024-02-19T20:17:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:40.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:40.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:40.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:40.817+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:40.933+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:40.934279+0000) 2024-02-19T20:17:41.339 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:41.021+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:41.022512+0000) 2024-02-19T20:17:41.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:41.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:41.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:41.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:41.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:41.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:41.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:41.257+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:41.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:41.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:41.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:41.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:41.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:41.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:41.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:41.537+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:42.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:41.733+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:41.734839+0000) 2024-02-19T20:17:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:41.810+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:41.810+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:41.810+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:41.811+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:41.935+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:41.936629+0000) 2024-02-19T20:17:42.290 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:42.023+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:42.024918+0000) 2024-02-19T20:17:42.737 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:42.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:42.737 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:42.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:42.737 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:42.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:42.738 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:42.290+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:42.738 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:42.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:42.738 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:42.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:42.738 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:42.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:42.738 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:42.522+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:43.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:42.736+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:42.737140+0000) 2024-02-19T20:17:43.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:42.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:43.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:42.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:43.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:42.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:43.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:42.820+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:43.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:42.937+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:42.938818+0000) 2024-02-19T20:17:43.340 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:43.025+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:43.026912+0000) 2024-02-19T20:17:43.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:43.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:43.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:43.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:43.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:43.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:43.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:43.272+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:43.737 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:43.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:43.738 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:43.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:43.738 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:43.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:43.738 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:43.490+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:44.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:43.736+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:43.737314+0000) 2024-02-19T20:17:44.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:43.805+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:44.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:43.805+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:44.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:43.805+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:44.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:43.806+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:44.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:43.938+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:43.939052+0000) 2024-02-19T20:17:44.340 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:44.026+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:44.027096+0000) 2024-02-19T20:17:44.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:44.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:44.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:44.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:44.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:44.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:44.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:44.262+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:44.597 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:17:44.580+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56773 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:17:44.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:44.577+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:44.578388+0000) 2024-02-19T20:17:44.598 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:44.578+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:44.579522+0000) 2024-02-19T20:17:44.599 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:44.496+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:44.599 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:44.496+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:44.599 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:44.496+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:44.599 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:44.496+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:44.599 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:44.577+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:44.578321+0000) 2024-02-19T20:17:44.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:44.736+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:44.737469+0000) 2024-02-19T20:17:44.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:44.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:44.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:44.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:44.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:44.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:44.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:44.839+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:45.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:44.938+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:44.939220+0000) 2024-02-19T20:17:45.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:45.026+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:45.027345+0000) 2024-02-19T20:17:45.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:45.272+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:45.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:45.272+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:45.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:45.272+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:45.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:45.273+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:45.813 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:45.738+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:45.739621+0000) 2024-02-19T20:17:45.813 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:45.539+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:45.814 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:45.539+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:45.814 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:45.539+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:45.814 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:45.539+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:46.068 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:46.029+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:46.030048+0000) 2024-02-19T20:17:46.069 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:45.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:46.069 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:45.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:46.069 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:45.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:46.069 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:45.813+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:46.069 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:45.940+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:45.941395+0000) 2024-02-19T20:17:46.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:46.279+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:46.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:46.279+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:46.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:46.279+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:46.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:46.280+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:46.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:46.543+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:46.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:46.543+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:46.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:46.543+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:46.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:46.544+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:46.740+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:46.741866+0000) 2024-02-19T20:17:47.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:46.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:47.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:46.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:47.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:46.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:47.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:46.813+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:47.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:46.942+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:46.943422+0000) 2024-02-19T20:17:47.340 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:47.032+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:47.033579+0000) 2024-02-19T20:17:47.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:47.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:47.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:47.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:47.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:47.265+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:47.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:47.265+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:47.744 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:47.561+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:47.744 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:47.561+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:47.744 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:47.561+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:47.744 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:47.562+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:48.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:47.743+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:47.744187+0000) 2024-02-19T20:17:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:47.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:47.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:47.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:47.764+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:47.944+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:47.945429+0000) 2024-02-19T20:17:48.307 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:48.036+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:48.037046+0000) 2024-02-19T20:17:48.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:48.306+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:48.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:48.306+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:48.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:48.306+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:48.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:48.306+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:48.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:48.743+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:48.744343+0000) 2024-02-19T20:17:48.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:48.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:48.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:48.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:48.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:48.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:48.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:48.750+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:48.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:48.577+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:48.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:48.577+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:48.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:48.577+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:48.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:48.578+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:49.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:48.944+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:48.945603+0000) 2024-02-19T20:17:49.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:49.036+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:49.037209+0000) 2024-02-19T20:17:49.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:49.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:49.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:49.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:49.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:49.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:49.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:49.283+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:49.945 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:17:49.588+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56814 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:17:49.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:49.584+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:49.585359+0000) 2024-02-19T20:17:49.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:49.743+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:49.744524+0000) 2024-02-19T20:17:49.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:49.585+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:49.586026+0000) 2024-02-19T20:17:49.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:49.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:49.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:49.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:49.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:49.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:49.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:49.734+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:49.947 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:49.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:49.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:49.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:49.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:49.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:49.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:49.537+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:49.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:49.585+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:49.586167+0000) 2024-02-19T20:17:50.239 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:49.944+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:49.945796+0000) 2024-02-19T20:17:50.239 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:50.036+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:50.037403+0000) 2024-02-19T20:17:50.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:50.238+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:50.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:50.238+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:50.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:50.238+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:50.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:50.239+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:50.948 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:50.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:50.948 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:50.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:50.948 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:50.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:50.948 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:50.709+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:50.949 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:50.746+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:50.747072+0000) 2024-02-19T20:17:50.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:50.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:50.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:50.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:50.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:50.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:50.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:50.576+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:51.252 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:50.947+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:50.948077+0000) 2024-02-19T20:17:51.253 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:51.038+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:51.039490+0000) 2024-02-19T20:17:51.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:51.251+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:51.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:51.251+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:51.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:51.251+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:51.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:51.252+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:51.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:51.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:51.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:51.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:51.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:51.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:51.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:51.686+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:51.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:51.748+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:51.749174+0000) 2024-02-19T20:17:51.952 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:51.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:51.952 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:51.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:51.952 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:51.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:51.952 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:51.530+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:52.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:51.949+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:51.950743+0000) 2024-02-19T20:17:52.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:52.040+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:52.041836+0000) 2024-02-19T20:17:52.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:52.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:52.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:52.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:52.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:52.256+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:52.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:52.257+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:52.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:52.750+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:52.751325+0000) 2024-02-19T20:17:52.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:52.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:52.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:52.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:52.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:52.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:52.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:52.669+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:52.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:52.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:52.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:52.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:52.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:52.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:52.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:52.510+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:53.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:52.951+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:52.952898+0000) 2024-02-19T20:17:53.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:53.042+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:53.043847+0000) 2024-02-19T20:17:53.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:53.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:53.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:53.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:53.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:53.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:53.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:53.282+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:53.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:53.750+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:53.751491+0000) 2024-02-19T20:17:53.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:53.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:53.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:53.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:53.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:53.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:53.954 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:53.549+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:53.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:53.663+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:53.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:53.663+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:53.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:53.663+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:53.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:53.664+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:54.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:53.952+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:53.953069+0000) 2024-02-19T20:17:54.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:54.043+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:54.044042+0000) 2024-02-19T20:17:54.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:54.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:54.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:54.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:54.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:54.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:54.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:54.328+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:54.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:54.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:54.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:54.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:54.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:54.506+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:54.953 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:17:54.595+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56854 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:17:54.954 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:54.592+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:54.593277+0000) 2024-02-19T20:17:54.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:54.592+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:54.593769+0000) 2024-02-19T20:17:54.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:54.750+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:54.751687+0000) 2024-02-19T20:17:54.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:54.593+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:54.594524+0000) 2024-02-19T20:17:54.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:54.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:54.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:54.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:54.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:54.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:54.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:54.616+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:55.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:54.952+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:54.953364+0000) 2024-02-19T20:17:55.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:55.043+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:55.044210+0000) 2024-02-19T20:17:55.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:55.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:55.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:55.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:55.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:55.328+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:55.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:55.329+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:55.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:55.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:55.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:55.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:55.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:55.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:55.512+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:55.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:55.752+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:55.753650+0000) 2024-02-19T20:17:55.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:55.632+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:55.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:55.632+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:55.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:55.632+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:55.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:55.633+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:56.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:55.955+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:55.956856+0000) 2024-02-19T20:17:56.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:56.045+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:56.046313+0000) 2024-02-19T20:17:56.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:56.311+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:56.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:56.311+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:56.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:56.311+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:56.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:56.312+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:56.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:56.478+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:56.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:56.478+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:56.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:56.478+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:56.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:56.479+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:56.959 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:56.754+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:56.755799+0000) 2024-02-19T20:17:56.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:56.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:56.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:56.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:56.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:56.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:56.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:56.636+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:57.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:56.958+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:56.959292+0000) 2024-02-19T20:17:57.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:57.048+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:57.049510+0000) 2024-02-19T20:17:57.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:57.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:57.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:57.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:57.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:57.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:57.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:57.360+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:57.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:57.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:57.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:57.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:57.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:57.509+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:57.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:57.510+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:57.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:57.756+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:57.757785+0000) 2024-02-19T20:17:57.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:57.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:57.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:57.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:57.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:57.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:57.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:57.673+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:58.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:57.960+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:57.961588+0000) 2024-02-19T20:17:58.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:58.050+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:58.051564+0000) 2024-02-19T20:17:58.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:58.360+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:58.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:58.360+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:58.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:58.360+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:58.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:58.361+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:58.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:58.757+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:58.758004+0000) 2024-02-19T20:17:58.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:58.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:58.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:58.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:58.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:58.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:58.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:58.646+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:58.962 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:58.538+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:58.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:58.538+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:58.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:58.538+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:58.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:58.538+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:59.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:58.960+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:58.961795+0000) 2024-02-19T20:17:59.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:59.050+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:59.051739+0000) 2024-02-19T20:17:59.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:59.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:59.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:59.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:59.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:59.367+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:17:59.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:59.368+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:59.962 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:17:59.602+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56895 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:17:59.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:59.600+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:59.601235+0000) 2024-02-19T20:17:59.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:17:59.757+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:59.758169+0000) 2024-02-19T20:17:59.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:59.599+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:59.600788+0000) 2024-02-19T20:17:59.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:59.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:59.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:59.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:59.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:59.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:17:59.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:59.635+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:17:59.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:59.551+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:59.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:59.551+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:59.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:59.551+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:17:59.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:59.552+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:17:59.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:17:59.599+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:59.600306+0000) 2024-02-19T20:18:00.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:17:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:17:59.961+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:59.961978+0000) 2024-02-19T20:18:00.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:00.050+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:00.051942+0000) 2024-02-19T20:18:00.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:00.391+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:00.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:00.391+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:00.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:00.391+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:00.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:00.392+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:00.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:00.760+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:00.761727+0000) 2024-02-19T20:18:00.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:00.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:00.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:00.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:00.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:00.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:00.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:00.647+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:00.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:00.582+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:00.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:00.582+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:00.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:00.582+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:00.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:00.583+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:01.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:00.963+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:00.964452+0000) 2024-02-19T20:18:01.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:01.054+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:01.055270+0000) 2024-02-19T20:18:01.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:01.439+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:01.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:01.439+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:01.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:01.439+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:01.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:01.440+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:01.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:01.762+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:01.763810+0000) 2024-02-19T20:18:01.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:01.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:01.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:01.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:01.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:01.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:01.968 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:01.685+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:01.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:01.563+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:01.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:01.563+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:01.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:01.563+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:01.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:01.563+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:02.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:01.965+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:01.966613+0000) 2024-02-19T20:18:02.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:02.056+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:02.057407+0000) 2024-02-19T20:18:02.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:02.444+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:02.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:02.444+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:02.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:02.444+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:02.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:02.445+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:02.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:02.764+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:02.765907+0000) 2024-02-19T20:18:02.969 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:02.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:02.969 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:02.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:02.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:02.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:02.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:02.640+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:02.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:02.583+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:02.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:02.583+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:02.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:02.583+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:02.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:02.584+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:03.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:02.968+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:02.969029+0000) 2024-02-19T20:18:03.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:03.058+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:03.059868+0000) 2024-02-19T20:18:03.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:03.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:03.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:03.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:03.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:03.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:03.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:03.421+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:03.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:03.765+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:03.766115+0000) 2024-02-19T20:18:03.969 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:03.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:03.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:03.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:03.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:03.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:03.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:03.645+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:03.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:03.625+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:03.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:03.625+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:03.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:03.625+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:03.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:03.625+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:04.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:03.968+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:03.969208+0000) 2024-02-19T20:18:04.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:04.059+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:04.060022+0000) 2024-02-19T20:18:04.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:04.434+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:04.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:04.434+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:04.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:04.434+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:04.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:04.434+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:04.969 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:18:04.610+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56935 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:18:04.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:04.608+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:04.609279+0000) 2024-02-19T20:18:04.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:04.765+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:04.766282+0000) 2024-02-19T20:18:04.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:04.607+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:04.608441+0000) 2024-02-19T20:18:04.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:04.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:04.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:04.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:04.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:04.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:04.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:04.663+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:04.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:04.968+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:04.969403+0000) 2024-02-19T20:18:04.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:04.606+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:04.607853+0000) 2024-02-19T20:18:04.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:04.610+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:04.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:04.610+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:04.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:04.610+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:04.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:04.610+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:05.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:05.059+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:05.060193+0000) 2024-02-19T20:18:05.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:05.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:05.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:05.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:05.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:05.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:05.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:05.441+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:05.972 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:05.769+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:05.770064+0000) 2024-02-19T20:18:05.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:05.592+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:05.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:05.592+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:05.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:05.592+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:05.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:05.592+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:05.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:05.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:05.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:05.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:05.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:05.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:05.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:05.620+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:05.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:05.971+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:05.972469+0000) 2024-02-19T20:18:06.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:06.062+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:06.063509+0000) 2024-02-19T20:18:06.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:06.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:06.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:06.418+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:06.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:06.419+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:06.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:06.771+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:06.772234+0000) 2024-02-19T20:18:06.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:06.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:06.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:06.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:06.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:06.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:06.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:06.640+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:06.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:06.974+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:06.975113+0000) 2024-02-19T20:18:06.976 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:06.624+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:06.976 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:06.624+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:06.976 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:06.624+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:06.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:06.624+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:07.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:07.064+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:07.065682+0000) 2024-02-19T20:18:07.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:07.397+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:07.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:07.397+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:07.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:07.397+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:07.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:07.397+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:07.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:07.773+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:07.774881+0000) 2024-02-19T20:18:07.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:07.607+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:07.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:07.607+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:07.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:07.607+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:07.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:07.607+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:07.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:07.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:07.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:07.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:07.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:07.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:07.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:07.649+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:07.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:07.976+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:07.977346+0000) 2024-02-19T20:18:08.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:08.068+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:08.069202+0000) 2024-02-19T20:18:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:08.400+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:08.400+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:08.400+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:08.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:08.400+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:08.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:08.774+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:08.775017+0000) 2024-02-19T20:18:08.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:08.622+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:08.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:08.622+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:08.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:08.622+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:08.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:08.623+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:08.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:08.620+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:08.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:08.620+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:08.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:08.620+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:08.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:08.620+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:09.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:08.976+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:08.977526+0000) 2024-02-19T20:18:09.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:09.068+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:09.069358+0000) 2024-02-19T20:18:09.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:09.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:09.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:09.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:09.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:09.418+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:09.977 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:18:09.617+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 56976 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:18:09.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:09.614+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:09.615896+0000) 2024-02-19T20:18:09.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:09.774+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:09.775177+0000) 2024-02-19T20:18:09.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:09.614+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:09.615339+0000) 2024-02-19T20:18:09.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:09.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:09.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:09.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:09.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:09.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:09.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:09.657+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:09.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:09.593+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:09.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:09.593+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:09.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:09.593+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:09.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:09.593+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:09.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:09.614+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:09.615818+0000) 2024-02-19T20:18:10.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:09.976+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:09.977717+0000) 2024-02-19T20:18:10.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:10.068+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:10.069549+0000) 2024-02-19T20:18:10.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:10.395+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:10.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:10.395+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:10.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:10.395+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:10.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:10.396+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:10.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:10.776+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:10.777741+0000) 2024-02-19T20:18:10.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:10.609+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:10.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:10.609+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:10.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:10.609+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:10.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:10.610+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:10.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:10.577+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:10.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:10.577+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:10.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:10.577+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:10.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:10.578+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:11.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:10.979+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:10.980077+0000) 2024-02-19T20:18:11.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:11.070+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:11.071535+0000) 2024-02-19T20:18:11.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:11.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:11.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:11.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:11.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:11.350+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:11.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:11.351+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:11.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:11.607+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:11.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:11.607+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:11.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:11.607+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:11.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:11.607+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:11.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:11.780+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:11.781194+0000) 2024-02-19T20:18:11.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:11.608+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:11.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:11.608+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:11.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:11.608+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:11.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:11.609+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:12.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:11.982+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:11.983548+0000) 2024-02-19T20:18:12.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:12.073+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:12.074192+0000) 2024-02-19T20:18:12.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:12.342+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:12.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:12.342+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:12.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:12.342+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:12.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:12.343+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:12.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:12.782+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:12.783714+0000) 2024-02-19T20:18:12.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:12.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:12.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:12.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:12.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:12.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:12.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:12.569+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:12.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:12.572+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:12.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:12.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:12.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:12.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:12.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:12.573+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:13.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:12.985+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:12.986177+0000) 2024-02-19T20:18:13.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:13.075+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:13.076626+0000) 2024-02-19T20:18:13.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:13.393+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:13.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:13.393+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:13.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:13.393+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:13.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:13.394+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:13.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:13.782+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:13.783907+0000) 2024-02-19T20:18:13.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:13.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:13.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:13.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:13.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:13.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:13.987 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:13.574+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:13.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:13.577+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:13.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:13.577+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:13.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:13.577+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:13.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:13.577+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:14.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:13.985+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:13.986337+0000) 2024-02-19T20:18:14.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:14.075+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:14.076847+0000) 2024-02-19T20:18:14.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:14.375+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:14.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:14.375+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:14.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:14.375+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:14.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:14.376+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:14.853 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:18:14.624+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57016 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:18:14.853 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:14.621+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:14.622261+0000) 2024-02-19T20:18:14.854 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:14.783+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:14.784097+0000) 2024-02-19T20:18:14.854 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:14.540+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:14.854 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:14.540+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:14.854 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:14.540+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:14.854 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:14.541+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:14.854 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:14.621+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:14.622253+0000) 2024-02-19T20:18:14.855 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:14.563+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:14.855 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:14.564+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:14.855 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:14.564+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:14.855 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:14.564+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:14.855 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:14.620+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:14.621868+0000) 2024-02-19T20:18:15.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:14.985+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:14.986543+0000) 2024-02-19T20:18:15.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:15.076+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:15.077026+0000) 2024-02-19T20:18:15.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:15.382+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:15.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:15.382+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:15.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:15.382+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:15.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:15.383+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:15.517+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:15.517+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:15.517+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:15.517+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:15.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:15.610+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:15.610+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:15.610+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:15.610+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:16.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:15.785+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:15.786103+0000) 2024-02-19T20:18:16.072 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:15.987+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:15.988571+0000) 2024-02-19T20:18:16.343 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:16.078+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:16.079153+0000) 2024-02-19T20:18:16.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:16.393+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:16.393+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:16.393+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:16.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:16.394+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:16.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:16.468+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:16.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:16.468+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:16.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:16.468+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:16.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:16.468+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:16.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:16.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:16.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:16.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:16.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:16.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:16.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:16.576+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:17.072 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:16.990+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:16.991121+0000) 2024-02-19T20:18:17.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:16.787+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:16.788513+0000) 2024-02-19T20:18:17.343 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:17.080+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:17.081335+0000) 2024-02-19T20:18:17.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:17.436+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:17.436+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:17.436+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:17.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:17.436+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:17.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:17.422+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:17.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:17.422+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:17.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:17.422+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:17.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:17.422+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:17.537+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:17.537+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:17.537+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:17.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:17.537+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:18.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:17.789+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:17.790688+0000) 2024-02-19T20:18:18.072 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:17.992+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:17.993341+0000) 2024-02-19T20:18:18.343 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:18.083+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:18.084922+0000) 2024-02-19T20:18:18.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:18.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:18.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:18.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:18.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:18.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:18.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:18.488+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:18.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:18.443+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:18.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:18.443+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:18.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:18.443+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:18.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:18.444+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:18.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:18.409+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:18.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:18.409+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:18.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:18.409+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:18.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:18.409+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:19.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:18.789+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:18.790864+0000) 2024-02-19T20:18:19.072 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:18.992+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:18.993510+0000) 2024-02-19T20:18:19.344 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:19.084+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:19.085084+0000) 2024-02-19T20:18:19.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:19.458+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:19.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:19.458+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:19.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:19.458+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:19.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:19.459+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:19.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:19.415+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:19.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:19.415+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:19.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:19.415+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:19.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:19.416+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:19.628 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:19.470+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:19.628 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:19.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:19.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:19.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:19.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:19.471+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:19.901 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:18:19.631+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57057 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:18:19.902 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:19.628+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:19.629731+0000) 2024-02-19T20:18:19.902 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:19.628+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:19.629192+0000) 2024-02-19T20:18:19.903 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:19.629+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:19.630339+0000) 2024-02-19T20:18:19.903 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:19.790+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:19.791030+0000) 2024-02-19T20:18:20.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:19.992+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:19.993701+0000) 2024-02-19T20:18:20.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:20.084+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:20.085232+0000) 2024-02-19T20:18:20.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:20.405+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:20.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:20.405+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:20.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:20.405+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:20.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:20.406+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:20.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:20.445+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:20.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:20.445+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:20.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:20.445+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:20.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:20.446+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:20.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:20.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:20.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:20.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:20.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:20.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:20.522 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:20.484+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:21.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:20.792+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:20.793583+0000) 2024-02-19T20:18:21.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:20.996+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:20.997239+0000) 2024-02-19T20:18:21.344 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:21.086+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:21.087966+0000) 2024-02-19T20:18:21.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:21.406+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:21.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:21.406+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:21.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:21.406+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:21.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:21.407+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:21.488+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:21.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:21.488+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:21.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:21.488+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:21.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:21.488+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:21.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:21.464+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:21.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:21.464+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:21.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:21.464+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:21.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:21.465+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:22.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:21.796+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:21.797089+0000) 2024-02-19T20:18:22.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:21.998+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:21.999210+0000) 2024-02-19T20:18:22.344 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:22.089+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:22.090168+0000) 2024-02-19T20:18:22.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:22.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:22.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:22.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:22.454+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:22.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:22.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:22.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:22.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:22.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:22.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:22.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:22.372+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:22.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:22.453+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:22.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:22.453+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:22.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:22.453+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:22.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:22.453+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:23.072 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:23.000+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:23.001896+0000) 2024-02-19T20:18:23.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:22.799+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:22.800338+0000) 2024-02-19T20:18:23.344 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:23.092+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:23.093792+0000) 2024-02-19T20:18:23.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:23.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:23.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:23.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:23.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:23.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:23.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:23.484+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:23.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:23.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:23.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:23.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:23.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:23.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:23.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:23.379+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:23.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:23.440+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:23.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:23.440+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:23.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:23.440+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:23.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:23.441+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:24.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:23.799+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:23.800539+0000) 2024-02-19T20:18:24.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:24.001+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:24.002077+0000) 2024-02-19T20:18:24.344 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:24.093+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:24.094001+0000) 2024-02-19T20:18:24.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:24.422+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:24.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:24.422+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:24.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:24.422+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:24.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:24.423+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:24.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:24.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:24.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:24.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:24.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:24.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:24.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:24.487+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:24.635 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:24.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:24.635 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:24.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:24.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:24.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:24.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:24.528+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:24.948 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:18:24.638+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57097 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:18:24.949 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:24.637+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:24.638234+0000) 2024-02-19T20:18:24.949 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:24.636+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:24.637258+0000) 2024-02-19T20:18:24.949 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:24.799+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:24.800723+0000) 2024-02-19T20:18:24.950 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:24.635+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:24.636514+0000) 2024-02-19T20:18:25.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:25.001+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:25.002231+0000) 2024-02-19T20:18:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:25.093+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:25.094145+0000) 2024-02-19T20:18:25.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:25.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:25.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:25.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:25.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:25.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:25.530 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:25.417+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:25.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:25.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:25.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:25.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:25.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:25.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:25.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:25.512+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:25.804 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:25.528+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:25.804 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:25.528+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:25.804 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:25.528+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:25.804 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:25.529+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:26.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:25.803+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:25.803965+0000) 2024-02-19T20:18:26.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:26.003+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:26.004339+0000) 2024-02-19T20:18:26.344 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:26.095+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:26.096245+0000) 2024-02-19T20:18:26.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:26.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:26.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:26.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:26.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:26.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:26.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:26.490+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:26.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:26.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:26.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:26.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:26.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:26.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:26.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:26.463+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:26.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:26.572+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:26.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:26.572+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:26.771 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:26.572+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:26.772 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:26.572+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:27.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:26.805+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:26.806150+0000) 2024-02-19T20:18:27.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:27.004+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:27.006938+0000) 2024-02-19T20:18:27.344 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:27.096+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:27.098796+0000) 2024-02-19T20:18:27.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:27.535+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:27.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:27.535+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:27.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:27.535+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:27.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:27.536+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:27.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:27.435+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:27.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:27.435+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:27.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:27.435+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:27.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:27.435+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:27.467+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:27.467+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:27.467+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:27.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:27.467+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:28.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:27.806+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:27.808227+0000) 2024-02-19T20:18:28.074 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:28.008+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:28.008990+0000) 2024-02-19T20:18:28.344 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:28.100+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:28.101104+0000) 2024-02-19T20:18:28.769 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:28.582+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:28.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:28.582+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:28.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:28.582+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:28.770 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:28.583+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:28.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:28.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:28.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:28.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:28.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:28.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:28.771 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:28.460+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:28.416+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:28.416+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:28.416+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:28.771 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:28.416+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:29.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:28.807+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:28.808459+0000) 2024-02-19T20:18:29.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:29.008+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:29.009191+0000) 2024-02-19T20:18:29.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:29.100+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:29.101305+0000) 2024-02-19T20:18:29.607 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:29.433+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:29.607 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:29.433+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:29.607 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:29.433+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:29.607 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:29.434+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:29.608 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:29.374+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:29.608 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:29.374+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:29.608 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:29.374+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:29.608 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:29.374+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:29.996 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:18:29.646+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57137 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:18:29.996 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:29.642+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:29.643801+0000) 2024-02-19T20:18:29.997 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:29.643+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:29.644409+0000) 2024-02-19T20:18:29.997 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:29.807+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:29.808629+0000) 2024-02-19T20:18:29.997 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:29.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:29.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:29.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:29.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:29.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:29.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:29.606+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:29.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:29.643+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:29.644422+0000) 2024-02-19T20:18:30.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:30.008+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:30.009412+0000) 2024-02-19T20:18:30.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:30.100+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:30.101544+0000) 2024-02-19T20:18:30.637 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:30.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:30.638 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:30.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:30.638 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:30.417+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:30.638 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:30.418+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:30.638 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:30.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:30.638 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:30.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:30.639 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:30.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:30.639 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:30.454+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:31.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:30.809+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:30.810727+0000) 2024-02-19T20:18:31.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:30.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:31.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:30.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:31.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:30.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:31.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:30.637+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:31.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:31.102+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:31.103669+0000) 2024-02-19T20:18:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:31.011+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:31.012523+0000) 2024-02-19T20:18:31.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:31.461+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:31.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:31.461+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:31.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:31.461+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:31.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:31.462+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:31.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:31.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:31.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:31.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:31.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:31.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:31.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:31.439+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:32.014 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:31.811+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:31.812807+0000) 2024-02-19T20:18:32.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:31.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:32.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:31.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:32.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:31.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:32.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:31.674+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:32.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:32.013+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:32.014694+0000) 2024-02-19T20:18:32.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:32.104+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:32.105661+0000) 2024-02-19T20:18:32.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:32.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:32.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:32.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:32.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:32.503+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:32.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:32.504+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:32.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:32.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:32.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:32.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:32.703 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:32.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:32.703 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:32.476+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:33.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:32.814+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:32.815302+0000) 2024-02-19T20:18:33.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:32.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:33.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:32.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:33.018 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:32.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:33.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:32.701+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:33.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:33.107+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:33.108099+0000) 2024-02-19T20:18:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:33.017+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:33.018011+0000) 2024-02-19T20:18:33.548 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:33.485+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:33.548 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:33.485+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:33.549 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:33.485+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:33.549 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:33.486+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:33.802 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:33.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:33.803 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:33.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:33.803 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:33.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:33.803 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:33.662+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:33.803 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:33.547+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:33.803 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:33.547+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:33.803 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:33.547+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:33.804 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:33.548+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:34.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:33.814+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:33.815538+0000) 2024-02-19T20:18:34.074 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:34.017+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:34.018380+0000) 2024-02-19T20:18:34.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:34.107+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:34.108317+0000) 2024-02-19T20:18:34.648 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:34.562+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:34.648 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:34.562+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:34.648 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:34.562+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:34.648 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:34.563+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:34.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:34.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:34.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:34.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:34.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:34.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:34.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:34.458+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:35.018 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:18:34.653+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57178 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:18:35.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:34.650+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:34.651694+0000) 2024-02-19T20:18:35.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:34.650+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:34.651709+0000) 2024-02-19T20:18:35.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:34.814+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:34.815744+0000) 2024-02-19T20:18:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:34.650+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:34.651930+0000) 2024-02-19T20:18:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:34.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:34.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:34.711+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:35.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:34.711+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:35.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:35.017+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:35.018583+0000) 2024-02-19T20:18:35.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:35.107+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:35.108479+0000) 2024-02-19T20:18:35.557 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:35.491+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:35.557 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:35.492+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:35.557 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:35.492+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:35.557 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:35.492+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:35.818 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:35.556+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:35.818 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:35.556+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:35.818 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:35.556+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:35.818 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:35.557+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:35.819 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:35.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:35.819 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:35.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:35.819 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:35.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:35.819 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:35.745+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:36.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:35.817+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:35.818145+0000) 2024-02-19T20:18:36.074 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:36.020+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:36.021291+0000) 2024-02-19T20:18:36.345 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:36.110+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:36.111634+0000) 2024-02-19T20:18:36.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:36.508+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:36.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:36.508+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:36.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:36.508+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:36.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:36.509+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:36.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:36.469+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:36.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:36.469+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:36.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:36.469+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:36.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:36.470+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:37.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:36.819+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:36.820651+0000) 2024-02-19T20:18:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:36.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:36.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:36.766+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:37.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:36.767+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:37.346 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:37.022+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:37.023271+0000) 2024-02-19T20:18:37.346 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:37.113+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:37.114152+0000) 2024-02-19T20:18:37.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:37.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:37.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:37.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:37.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:37.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:37.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:37.506+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:37.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:37.424+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:37.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:37.424+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:37.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:37.424+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:37.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:37.424+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:38.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:37.822+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:37.823953+0000) 2024-02-19T20:18:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:37.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:37.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:37.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:38.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:37.718+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:38.346 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:38.024+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:38.025860+0000) 2024-02-19T20:18:38.346 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:38.115+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:38.116889+0000) 2024-02-19T20:18:38.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:38.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:38.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:38.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:38.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:38.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:38.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:38.545+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:38.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:38.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:38.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:38.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:38.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:38.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:38.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:38.415+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:39.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:38.823+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:38.824167+0000) 2024-02-19T20:18:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:38.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:38.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:38.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:39.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:38.719+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:39.346 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:39.025+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:39.026099+0000) 2024-02-19T20:18:39.346 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:39.116+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:39.117101+0000) 2024-02-19T20:18:39.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:39.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:39.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:39.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:39.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:39.531+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:39.656 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:39.532+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:39.656 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:39.377+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:39.656 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:39.377+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:39.656 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:39.377+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:39.656 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:39.377+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:40.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:18:39.660+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57218 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:18:40.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:39.658+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:39.659886+0000) 2024-02-19T20:18:40.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:39.658+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:39.659014+0000) 2024-02-19T20:18:40.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:39.823+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:39.824350+0000) 2024-02-19T20:18:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:39.657+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:39.658685+0000) 2024-02-19T20:18:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:39.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:39.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:39.708+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:40.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:39.709+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:40.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:40.025+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:40.026264+0000) 2024-02-19T20:18:40.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:40.116+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:40.117269+0000) 2024-02-19T20:18:40.335 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:40.333+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:40.335 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:40.333+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:40.703 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:40.333+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:40.703 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:40.333+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:40.704 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:40.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:40.704 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:40.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:40.704 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:40.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:40.704 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:40.578+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:41.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:40.825+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:40.826387+0000) 2024-02-19T20:18:41.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:40.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:40.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:40.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:41.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:40.703+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:41.323 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:41.027+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:41.028648+0000) 2024-02-19T20:18:41.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:41.118+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:41.119354+0000) 2024-02-19T20:18:41.603 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:41.322+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:41.603 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:41.322+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:41.603 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:41.322+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:41.603 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:41.323+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:42.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:41.707+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:42.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:41.707+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:41.707+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:42.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:41.708+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:42.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:41.601+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:42.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:41.601+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:42.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:41.601+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:42.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:41.602+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:42.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:41.827+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:41.828815+0000) 2024-02-19T20:18:42.346 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:42.030+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:42.031128+0000) 2024-02-19T20:18:42.347 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:42.120+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:42.121450+0000) 2024-02-19T20:18:42.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:42.353+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:42.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:42.353+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:42.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:42.353+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:42.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:42.354+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:43.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:42.640+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:43.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:42.640+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:43.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:42.640+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:43.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:42.641+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:43.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:42.830+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:42.831309+0000) 2024-02-19T20:18:43.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:42.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:43.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:42.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:43.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:42.715+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:43.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:42.716+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:43.346 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:43.032+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:43.033182+0000) 2024-02-19T20:18:43.347 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:43.123+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:43.124876+0000) 2024-02-19T20:18:43.684 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:43.391+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:43.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:43.391+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:43.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:43.391+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:43.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:43.391+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:44.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:43.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:44.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:43.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:44.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:43.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:44.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:43.703+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:44.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:43.683+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:44.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:43.683+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:44.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:43.683+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:44.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:43.684+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:44.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:43.830+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:43.831500+0000) 2024-02-19T20:18:44.346 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:44.032+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:44.033429+0000) 2024-02-19T20:18:44.347 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:44.124+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:44.125072+0000) 2024-02-19T20:18:44.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:44.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:44.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:44.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:44.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:44.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:44.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:44.427+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:45.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:44.666+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:44.667066+0000) 2024-02-19T20:18:45.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:18:44.667+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57259 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:18:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:44.665+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:44.666602+0000) 2024-02-19T20:18:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:44.716+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:44.716+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:45.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:44.716+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:45.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:44.717+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:45.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:44.830+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:44.831691+0000) 2024-02-19T20:18:45.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:44.664+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:44.665594+0000) 2024-02-19T20:18:45.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:44.704+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:45.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:44.704+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:45.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:44.704+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:45.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:44.705+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:45.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:45.032+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:45.033624+0000) 2024-02-19T20:18:45.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:45.124+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:45.125241+0000) 2024-02-19T20:18:45.723 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:45.393+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:45.723 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:45.393+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:45.724 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:45.393+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:45.724 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:45.393+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:46.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:45.721+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:46.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:45.721+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:46.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:45.721+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:46.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:45.722+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:46.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:45.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:46.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:45.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:46.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:45.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:45.767+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:46.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:45.832+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:45.833764+0000) 2024-02-19T20:18:46.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:46.034+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:46.035880+0000) 2024-02-19T20:18:46.347 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:46.126+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:46.127238+0000) 2024-02-19T20:18:46.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:46.399+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:46.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:46.399+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:46.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:46.399+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:46.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:46.400+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:47.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:46.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:47.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:46.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:47.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:46.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:47.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:46.745+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:46.748+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:46.748+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:47.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:46.748+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:47.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:46.749+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:47.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:46.834+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:46.835846+0000) 2024-02-19T20:18:47.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:47.038+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:47.039376+0000) 2024-02-19T20:18:47.347 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:47.128+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:47.129212+0000) 2024-02-19T20:18:47.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:47.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:47.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:47.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:47.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:47.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:47.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:47.433+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:48.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:47.730+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:47.730+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:47.730+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:47.731+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:47.837+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:47.838158+0000) 2024-02-19T20:18:48.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:47.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:47.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:47.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:48.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:47.702+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:48.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:48.041+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:48.042023+0000) 2024-02-19T20:18:48.347 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:48.130+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:48.131625+0000) 2024-02-19T20:18:48.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:48.465+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:48.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:48.465+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:48.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:48.465+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:48.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:48.465+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:49.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:48.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:48.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:48.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:49.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:48.673+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:48.769+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:48.769+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:48.769+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:49.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:48.770+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:49.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:48.837+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:48.838403+0000) 2024-02-19T20:18:49.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:49.041+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:49.042284+0000) 2024-02-19T20:18:49.347 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:49.130+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:49.131899+0000) 2024-02-19T20:18:49.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:49.429+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:49.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:49.429+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:49.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:49.429+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:49.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:49.429+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:50.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:18:49.674+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57299 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:18:50.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:49.672+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:49.673606+0000) 2024-02-19T20:18:50.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:49.671+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:49.672723+0000) 2024-02-19T20:18:50.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:49.710+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:50.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:49.710+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:50.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:49.710+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:50.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:49.711+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:50.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:49.672+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:49.673247+0000) 2024-02-19T20:18:50.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:49.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:50.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:49.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:50.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:49.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:50.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:49.810+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:50.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:49.837+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:49.838626+0000) 2024-02-19T20:18:50.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:50.041+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:50.042464+0000) 2024-02-19T20:18:50.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:50.131+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:50.132095+0000) 2024-02-19T20:18:50.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:50.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:50.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:50.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:50.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:50.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:50.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:50.439+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:51.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:50.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:51.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:50.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:51.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:50.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:51.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:50.719+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:50.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:50.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:51.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:50.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:51.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:50.836+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:51.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:50.839+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:50.840704+0000) 2024-02-19T20:18:51.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:51.043+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:51.044428+0000) 2024-02-19T20:18:51.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:51.133+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:51.134078+0000) 2024-02-19T20:18:51.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:51.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:51.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:51.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:51.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:51.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:51.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:51.450+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:52.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:51.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:52.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:51.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:52.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:51.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:52.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:51.713+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:51.841+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:51.842861+0000) 2024-02-19T20:18:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:51.859+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:51.859+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:52.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:51.859+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:52.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:51.860+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:52.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:52.045+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:52.046493+0000) 2024-02-19T20:18:52.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:52.135+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:52.136236+0000) 2024-02-19T20:18:52.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:52.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:52.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:52.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:52.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:52.449+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:52.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:52.449+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:53.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:52.755+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:53.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:52.755+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:53.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:52.755+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:53.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:52.756+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:53.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:52.844+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:52.845068+0000) 2024-02-19T20:18:53.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:52.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:53.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:52.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:53.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:52.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:53.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:52.861+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:53.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:53.048+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:53.049178+0000) 2024-02-19T20:18:53.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:53.138+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:53.139718+0000) 2024-02-19T20:18:53.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:53.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:53.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:53.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:53.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:53.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:53.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:53.460+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:54.049 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:53.844+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:53.845306+0000) 2024-02-19T20:18:54.049 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:53.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:54.050 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:53.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:54.050 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:53.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:54.050 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:53.855+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:54.050 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:53.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:54.050 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:53.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:54.050 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:53.785+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:54.050 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:53.786+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:54.348 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:54.048+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:54.049462+0000) 2024-02-19T20:18:54.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:54.139+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:54.140006+0000) 2024-02-19T20:18:54.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:54.416+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:54.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:54.416+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:54.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:54.416+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:54.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:54.416+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:55.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:18:54.681+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57340 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:18:55.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:54.679+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:54.680084+0000) 2024-02-19T20:18:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:54.679+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:54.680028+0000) 2024-02-19T20:18:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:54.844+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:54.845461+0000) 2024-02-19T20:18:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:54.894+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:54.894+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:54.894+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:54.894+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:55.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:54.678+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:54.679357+0000) 2024-02-19T20:18:55.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:54.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:55.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:54.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:55.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:54.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:55.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:54.835+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:55.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:55.139+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:55.140159+0000) 2024-02-19T20:18:55.348 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:55.048+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:55.049672+0000) 2024-02-19T20:18:55.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:55.455+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:55.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:55.455+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:55.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:55.455+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:55.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:55.456+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:56.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:55.846+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:55.847486+0000) 2024-02-19T20:18:56.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:55.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:56.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:55.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:56.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:55.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:56.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:55.849+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:56.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:55.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:56.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:55.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:56.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:55.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:56.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:55.830+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:56.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:56.051+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:56.052918+0000) 2024-02-19T20:18:56.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:56.141+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:56.142132+0000) 2024-02-19T20:18:56.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:56.417+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:56.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:56.417+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:56.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:56.417+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:56.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:56.417+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:57.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:56.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:57.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:56.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:57.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:56.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:57.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:56.833+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:57.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:57.055+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:57.056113+0000) 2024-02-19T20:18:57.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:56.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:57.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:56.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:57.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:56.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:57.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:56.827+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:57.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:56.849+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:56.850206+0000) 2024-02-19T20:18:57.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:57.143+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:57.144850+0000) 2024-02-19T20:18:57.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:57.369+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:57.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:57.369+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:57.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:57.369+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:57.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:57.370+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:58.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:57.851+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:57.852356+0000) 2024-02-19T20:18:58.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:57.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:58.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:57.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:58.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:57.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:58.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:57.857+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:58.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:57.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:58.150 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:57.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:58.150 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:57.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:58.150 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:57.848+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:58.150 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:58.057+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:58.058739+0000) 2024-02-19T20:18:58.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:58.147+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:58.148463+0000) 2024-02-19T20:18:58.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:58.399+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:58.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:58.399+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:58.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:58.399+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:58.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:58.399+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:59.077 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:58.816+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:59.077 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:58.816+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:59.077 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:58.816+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:18:59.077 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:58.816+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:18:59.077 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:59.058+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:59.059015+0000) 2024-02-19T20:18:59.078 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:58.851+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:58.852596+0000) 2024-02-19T20:18:59.078 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:58.897+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:59.078 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:58.897+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:59.078 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:58.897+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:18:59.078 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:58.898+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:18:59.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:59.147+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:59.148713+0000) 2024-02-19T20:18:59.684 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:59.394+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:59.684 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:59.394+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:59.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:59.394+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:18:59.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:59.395+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:00.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:18:59.688+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57380 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:19:00.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:18:59.686+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:59.687496+0000) 2024-02-19T20:19:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:59.685+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:59.686210+0000) 2024-02-19T20:19:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:59.828+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:00.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:59.828+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:00.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:59.828+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:00.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:18:59.829+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:00.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:59.686+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:59.687254+0000) 2024-02-19T20:19:00.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:59.851+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:59.852777+0000) 2024-02-19T20:19:00.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:59.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:00.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:59.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:00.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:59.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:00.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:18:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:18:59.929+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:00.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:00.058+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:00.059183+0000) 2024-02-19T20:19:00.310 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:00.147+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:00.148879+0000) 2024-02-19T20:19:00.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:00.379+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:00.379+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:00.379+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:00.380+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:01.077 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:00.854+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:00.854992+0000) 2024-02-19T20:19:01.077 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:00.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:01.077 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:00.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:01.077 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:00.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:01.078 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:00.978+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:01.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:00.810+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:01.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:00.810+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:01.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:00.810+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:01.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:00.811+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:01.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:01.060+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:01.061597+0000) 2024-02-19T20:19:01.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:01.150+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:01.151281+0000) 2024-02-19T20:19:01.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:01.359+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:01.359+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:01.359+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:01.360+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:02.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:01.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:02.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:01.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:02.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:01.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:02.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:01.796+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:02.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:01.856+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:01.857282+0000) 2024-02-19T20:19:02.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:01.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:02.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:01.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:02.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:01.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:02.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:01.946+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:02.314 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:02.062+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:02.063584+0000) 2024-02-19T20:19:02.315 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:02.153+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:02.154873+0000) 2024-02-19T20:19:02.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:02.313+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:02.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:02.313+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:02.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:02.313+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:02.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:02.314+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:03.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:02.858+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:02.859242+0000) 2024-02-19T20:19:03.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:02.976+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:03.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:02.976+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:03.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:02.976+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:03.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:02.977+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:03.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:02.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:03.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:02.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:03.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:02.834+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:03.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:02.835+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:03.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:03.065+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:03.066156+0000) 2024-02-19T20:19:03.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:03.156+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:03.157269+0000) 2024-02-19T20:19:03.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:03.274+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:03.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:03.274+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:03.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:03.274+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:03.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:03.274+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:04.066 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:03.858+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:03.859468+0000) 2024-02-19T20:19:04.066 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:03.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:04.066 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:03.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:04.067 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:03.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:04.067 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:03.961+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:04.067 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:03.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:04.067 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:03.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:04.067 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:03.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:04.068 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:03.870+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:04.349 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:04.065+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:04.066402+0000) 2024-02-19T20:19:04.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:04.156+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:04.157569+0000) 2024-02-19T20:19:04.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:04.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:04.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:04.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:04.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:04.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:04.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:04.297+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:05.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:04.693+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:04.694583+0000) 2024-02-19T20:19:05.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:19:04.695+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57417 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:19:05.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:04.694+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:04.695249+0000) 2024-02-19T20:19:05.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:04.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:05.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:04.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:05.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:04.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:05.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:04.830+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:05.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:04.693+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:04.694224+0000) 2024-02-19T20:19:05.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:04.858+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:04.859668+0000) 2024-02-19T20:19:05.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:04.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:05.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:04.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:05.023 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:04.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:05.024 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:04.939+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:05.349 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:05.066+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:05.067119+0000) 2024-02-19T20:19:05.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:05.156+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:05.157748+0000) 2024-02-19T20:19:05.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:05.270+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:05.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:05.270+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:05.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:05.270+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:05.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:05.271+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:06.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:05.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:06.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:05.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:06.071 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:05.803+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:06.071 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:05.803+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:06.071 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:05.861+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:05.862080+0000) 2024-02-19T20:19:06.071 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:05.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:06.071 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:05.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:06.071 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:05.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:06.071 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:05.937+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:06.349 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:06.069+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:06.070479+0000) 2024-02-19T20:19:06.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:06.158+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:06.159685+0000) 2024-02-19T20:19:06.349 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:06.229+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:06.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:06.229+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:06.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:06.229+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:06.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:06.229+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:07.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:06.863+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:06.864441+0000) 2024-02-19T20:19:07.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:06.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:07.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:06.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:07.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:06.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:07.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:06.985+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:07.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:06.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:07.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:06.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:07.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:06.840+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:07.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:06.841+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:07.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:07.073+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:07.074025+0000) 2024-02-19T20:19:07.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:07.162+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:07.163160+0000) 2024-02-19T20:19:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:07.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:07.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:07.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:07.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:07.217+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:08.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:07.866+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:07.867057+0000) 2024-02-19T20:19:08.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:07.996+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:08.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:07.996+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:08.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:07.996+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:08.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:07.997+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:08.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:07.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:08.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:07.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:08.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:07.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:08.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:07.886+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:08.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:08.076+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:08.077527+0000) 2024-02-19T20:19:08.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:08.164+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:08.165325+0000) 2024-02-19T20:19:08.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:08.266+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:08.266+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:08.266+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:08.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:08.266+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:09.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:08.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:09.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:08.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:09.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:08.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:09.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:08.911+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:09.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:09.076+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:09.077906+0000) 2024-02-19T20:19:09.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:08.866+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:08.867374+0000) 2024-02-19T20:19:09.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:09.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:09.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:09.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:09.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:09.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:09.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:09.012+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:09.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:09.164+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:09.165612+0000) 2024-02-19T20:19:09.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:09.298+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:09.298+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:09.298+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:09.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:09.298+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:09.822 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:19:09.703+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57457 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:19:09.823 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:09.702+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:09.703097+0000) 2024-02-19T20:19:09.823 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:09.699+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:09.700949+0000) 2024-02-19T20:19:09.823 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:09.700+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:09.701195+0000) 2024-02-19T20:19:10.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:09.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:10.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:09.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:10.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:09.880+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:10.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:09.880+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:10.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:09.866+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:09.867578+0000) 2024-02-19T20:19:10.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:09.976+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:10.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:09.976+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:10.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:09.976+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:10.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:09.976+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:10.349 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:10.077+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:10.078095+0000) 2024-02-19T20:19:10.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:10.164+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:10.165767+0000) 2024-02-19T20:19:10.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:10.279+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:10.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:10.279+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:10.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:10.279+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:10.350 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:10.280+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:11.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:10.868+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:10.869655+0000) 2024-02-19T20:19:11.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:10.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:11.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:10.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:11.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:10.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:11.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:10.945+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:11.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:10.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:11.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:10.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:11.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:10.906+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:11.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:10.907+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:11.219 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:11.080+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:11.081478+0000) 2024-02-19T20:19:11.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:11.166+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:11.167815+0000) 2024-02-19T20:19:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:11.249+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:11.249+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:11.249+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:11.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:11.249+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:12.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:11.871+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:11.872467+0000) 2024-02-19T20:19:12.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:11.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:12.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:11.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:12.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:11.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:12.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:11.944+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:12.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:11.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:12.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:11.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:12.172 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:11.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:12.172 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:11.897+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:12.172 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:12.084+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:12.085067+0000) 2024-02-19T20:19:12.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:12.169+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:12.170130+0000) 2024-02-19T20:19:12.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:12.272+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:12.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:12.272+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:12.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:12.272+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:12.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:12.272+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:13.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:12.874+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:12.874987+0000) 2024-02-19T20:19:13.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:12.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:13.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:12.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:13.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:12.960+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:13.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:12.960+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:13.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:12.930+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:13.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:12.930+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:13.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:12.930+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:13.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:12.931+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:13.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:13.086+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:13.087145+0000) 2024-02-19T20:19:13.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:13.172+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:13.172745+0000) 2024-02-19T20:19:13.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:13.309+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:13.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:13.309+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:13.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:13.309+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:13.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:13.310+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:14.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:13.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:14.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:13.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:14.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:13.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:14.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:13.913+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:14.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:13.874+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:13.875304+0000) 2024-02-19T20:19:14.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:13.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:14.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:13.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:14.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:13.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:14.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:13.975+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:14.333 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:14.086+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:14.087503+0000) 2024-02-19T20:19:14.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:14.173+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:14.173199+0000) 2024-02-19T20:19:14.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:14.334+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:14.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:14.334+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:14.334 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:14.334+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:14.705 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:14.334+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:14.955 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:19:14.710+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57498 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:19:14.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:14.708+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:14.709779+0000) 2024-02-19T20:19:14.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:14.707+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:14.708171+0000) 2024-02-19T20:19:14.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:14.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:14.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:14.708+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:14.709542+0000) 2024-02-19T20:19:14.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:14.873+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:14.875533+0000) 2024-02-19T20:19:14.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:14.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:14.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:14.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:14.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:14.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:14.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:14.943+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:15.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:15.171+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:15.173488+0000) 2024-02-19T20:19:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:14.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:14.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:14.954+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:15.085+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:15.087733+0000) 2024-02-19T20:19:15.534 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:15.337+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:15.535 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:15.337+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:15.535 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:15.337+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:15.535 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:15.337+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:16.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:15.876+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:15.877650+0000) 2024-02-19T20:19:16.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:15.913+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:16.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:15.913+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:16.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:15.913+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:16.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:15.913+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:16.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:15.930+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:16.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:15.930+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:16.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:15.930+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:16.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:15.931+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:16.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:16.088+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:16.089840+0000) 2024-02-19T20:19:16.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:16.174+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:16.175936+0000) 2024-02-19T20:19:16.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:16.336+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:16.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:16.336+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:16.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:16.336+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:16.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:16.336+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:17.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:16.878+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:16.879969+0000) 2024-02-19T20:19:17.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:16.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:17.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:16.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:17.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:16.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:17.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:16.934+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:17.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:16.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:17.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:16.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:17.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:16.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:17.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:16.895+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:17.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:17.090+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:17.091909+0000) 2024-02-19T20:19:17.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:17.177+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:17.178292+0000) 2024-02-19T20:19:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:17.381+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:17.381+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:17.381+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:17.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:17.381+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:18.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:17.881+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:17.882411+0000) 2024-02-19T20:19:18.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:17.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:18.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:17.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:18.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:17.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:18.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:17.907+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:18.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:17.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:18.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:17.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:18.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:17.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:18.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:17.885+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:18.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:18.093+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:18.094121+0000) 2024-02-19T20:19:18.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:18.179+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:18.180872+0000) 2024-02-19T20:19:18.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:18.346+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:18.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:18.346+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:18.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:18.346+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:18.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:18.347+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:19.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:18.927+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:19.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:18.927+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:19.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:18.927+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:19.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:18.927+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:19.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:19.093+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:19.094474+0000) 2024-02-19T20:19:19.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:18.881+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:18.882915+0000) 2024-02-19T20:19:19.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:18.924+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:19.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:18.924+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:19.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:18.924+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:19.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:18.925+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:19.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:19.180+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:19.181216+0000) 2024-02-19T20:19:19.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:19.363+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:19.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:19.363+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:19.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:19.363+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:19.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:19.364+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:19.835 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:19:19.717+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57538 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:19:19.836 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:19.714+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:19.715280+0000) 2024-02-19T20:19:19.836 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:19.714+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:19.715373+0000) 2024-02-19T20:19:19.836 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:19.714+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:19.715761+0000) 2024-02-19T20:19:20.094 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:19.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:20.095 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:19.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:20.095 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:19.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:20.095 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:19.911+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:20.095 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:19.882+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:19.883179+0000) 2024-02-19T20:19:20.095 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:19.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:20.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:19.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:20.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:19.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:20.096 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:19.888+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:20.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:20.093+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:20.094718+0000) 2024-02-19T20:19:20.351 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:20.180+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:20.181408+0000) 2024-02-19T20:19:20.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:20.367+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:20.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:20.367+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:20.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:20.367+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:20.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:20.368+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:21.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:20.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:21.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:20.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:21.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:20.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:21.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:20.953+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:21.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:21.097+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:21.098228+0000) 2024-02-19T20:19:21.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:20.859+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:21.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:20.859+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:21.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:20.859+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:21.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:20.860+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:21.185 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:20.884+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:20.885243+0000) 2024-02-19T20:19:21.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:21.182+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:21.183444+0000) 2024-02-19T20:19:21.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:21.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:21.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:21.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:21.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:21.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:21.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:21.373+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:22.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:21.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:22.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:21.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:22.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:21.923+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:22.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:21.924+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:22.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:22.100+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:22.101738+0000) 2024-02-19T20:19:22.186 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:21.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:22.186 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:21.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:22.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:21.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:22.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:21.863+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:22.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:21.886+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:21.887561+0000) 2024-02-19T20:19:22.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:22.184+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:22.185623+0000) 2024-02-19T20:19:22.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:22.396+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:22.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:22.396+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:22.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:22.396+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:22.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:22.397+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:23.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:22.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:23.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:22.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:23.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:22.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:23.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:22.821+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:23.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:22.889+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:22.889985+0000) 2024-02-19T20:19:23.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:22.899+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:23.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:22.899+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:23.081 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:22.899+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:23.081 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:22.900+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:23.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:23.103+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:23.104155+0000) 2024-02-19T20:19:23.351 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:23.187+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:23.188769+0000) 2024-02-19T20:19:23.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:23.391+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:23.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:23.391+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:23.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:23.391+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:23.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:23.391+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:24.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:23.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:24.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:23.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:24.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:23.869+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:24.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:23.870+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:24.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:23.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:24.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:23.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:24.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:23.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:24.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:23.800+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:24.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:23.889+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:23.890276+0000) 2024-02-19T20:19:24.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:24.103+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:24.104435+0000) 2024-02-19T20:19:24.351 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:24.188+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:24.189127+0000) 2024-02-19T20:19:24.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:24.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:24.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:24.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:24.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:24.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:24.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:24.411+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:25.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:19:24.725+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57579 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:19:25.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:24.722+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:24.723082+0000) 2024-02-19T20:19:25.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:24.721+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:24.722938+0000) 2024-02-19T20:19:25.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:24.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:24.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:24.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:25.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:24.897+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:25.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:24.723+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:24.724158+0000) 2024-02-19T20:19:25.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:24.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:25.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:24.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:25.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:24.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:25.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:24.812+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:25.022 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:24.889+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:24.890462+0000) 2024-02-19T20:19:25.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:25.103+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:25.104639+0000) 2024-02-19T20:19:25.352 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:25.188+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:25.189294+0000) 2024-02-19T20:19:25.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:25.447+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:25.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:25.447+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:25.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:25.447+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:25.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:25.448+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:26.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:25.807+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:26.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:25.807+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:26.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:25.807+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:26.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:25.808+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:26.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:25.891+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:25.892567+0000) 2024-02-19T20:19:26.081 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:25.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:26.081 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:25.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:26.081 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:25.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:26.081 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:25.905+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:26.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:26.107+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:26.108138+0000) 2024-02-19T20:19:26.352 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:26.190+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:26.191294+0000) 2024-02-19T20:19:26.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:26.429+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:26.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:26.429+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:26.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:26.429+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:26.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:26.429+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:27.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:26.898+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:27.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:26.898+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:27.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:26.898+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:27.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:26.899+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:26.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:26.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:27.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:26.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:27.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:26.798+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:27.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:26.893+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:26.894624+0000) 2024-02-19T20:19:27.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:27.109+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:27.110555+0000) 2024-02-19T20:19:27.352 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:27.192+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:27.193286+0000) 2024-02-19T20:19:27.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:27.446+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:27.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:27.446+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:27.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:27.446+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:27.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:27.446+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:28.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:27.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:28.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:27.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:28.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:27.948+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:28.081 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:27.949+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:28.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:27.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:28.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:27.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:28.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:27.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:28.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:27.786+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:28.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:27.895+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:27.896831+0000) 2024-02-19T20:19:28.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:28.112+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:28.113884+0000) 2024-02-19T20:19:28.352 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:28.194+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:28.195486+0000) 2024-02-19T20:19:28.766 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:28.413+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:28.766 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:28.413+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:28.766 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:28.413+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:28.766 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:28.413+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:29.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:28.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:28.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:28.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:28.765+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:28.896+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:28.897162+0000) 2024-02-19T20:19:29.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:28.950+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:29.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:28.951+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:29.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:28.951+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:29.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:28.951+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:29.352 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:29.113+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:29.114197+0000) 2024-02-19T20:19:29.352 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:29.194+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:29.195773+0000) 2024-02-19T20:19:29.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:29.455+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:29.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:29.455+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:29.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:29.455+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:29.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:29.455+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:30.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:19:29.732+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57619 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:19:30.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:29.729+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:29.730088+0000) 2024-02-19T20:19:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:29.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:29.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:30.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:29.718+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:29.719+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:29.729+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:29.730886+0000) 2024-02-19T20:19:30.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:29.896+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:29.897343+0000) 2024-02-19T20:19:30.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:29.729+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:29.730811+0000) 2024-02-19T20:19:30.021 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:29.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:30.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:29.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:30.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:29.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:30.022 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:29.957+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:30.352 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:30.113+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:30.114358+0000) 2024-02-19T20:19:30.352 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:30.195+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:30.196018+0000) 2024-02-19T20:19:30.622 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:30.481+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:30.623 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:30.481+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:30.623 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:30.481+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:30.623 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:30.482+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:30.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:30.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:30.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:30.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:30.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:30.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:30.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:30.722+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:31.198 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:30.917+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:31.198 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:30.917+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:31.198 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:30.917+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:31.198 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:30.918+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:31.199 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:31.115+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:31.116437+0000) 2024-02-19T20:19:31.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:30.898+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:30.899670+0000) 2024-02-19T20:19:31.504 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:31.197+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:31.198157+0000) 2024-02-19T20:19:31.504 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:31.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:31.504 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:31.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:31.504 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:31.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:31.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:31.503+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:31.769 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:31.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:31.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:31.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:31.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:31.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:31.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:31.688+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:32.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:31.900+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:31.901782+0000) 2024-02-19T20:19:32.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:31.955+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:32.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:31.955+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:32.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:31.955+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:32.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:31.956+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:32.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:32.118+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:32.119709+0000) 2024-02-19T20:19:32.467 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:32.200+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:32.201015+0000) 2024-02-19T20:19:32.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:32.466+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:32.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:32.466+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:32.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:32.466+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:32.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:32.467+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:32.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:32.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:32.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:32.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:32.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:32.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:32.770 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:32.673+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:33.204 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:32.903+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:32.904018+0000) 2024-02-19T20:19:33.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:32.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:33.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:32.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:33.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:32.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:33.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:32.981+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:33.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:33.121+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:33.122085+0000) 2024-02-19T20:19:33.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:33.203+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:33.204286+0000) 2024-02-19T20:19:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:33.450+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:33.450+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:33.450+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:33.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:33.450+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:33.805 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:33.637+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:33.805 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:33.637+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:33.805 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:33.637+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:33.806 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:33.638+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:34.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:33.903+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:33.904417+0000) 2024-02-19T20:19:34.081 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:34.018+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:34.081 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:34.018+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:34.081 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:34.018+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:34.081 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:34.019+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:34.352 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:34.121+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:34.122488+0000) 2024-02-19T20:19:34.353 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:34.203+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:34.204732+0000) 2024-02-19T20:19:34.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:34.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:34.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:34.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:34.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:34.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:34.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:34.432+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:34.904 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:19:34.739+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57660 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:19:34.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:34.736+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:34.737270+0000) 2024-02-19T20:19:34.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:34.736+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:34.737789+0000) 2024-02-19T20:19:34.905 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:34.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:34.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:34.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:34.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:34.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:34.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:34.637+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:34.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:34.736+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:34.737137+0000) 2024-02-19T20:19:34.906 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:34.903+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:34.904634+0000) 2024-02-19T20:19:35.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:35.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:35.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:35.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:35.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:35.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:35.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:35.023+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:35.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:35.121+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:35.122703+0000) 2024-02-19T20:19:35.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:35.203+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:35.204900+0000) 2024-02-19T20:19:35.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:35.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:35.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:35.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:35.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:35.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:35.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:35.428+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:35.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:35.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:35.908 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:35.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:35.908 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:35.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:35.908 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:35.662+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:36.208 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:35.906+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:35.907846+0000) 2024-02-19T20:19:36.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:36.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:36.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:36.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:36.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:36.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:36.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:36.047+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:36.209 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:36.125+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:36.126017+0000) 2024-02-19T20:19:36.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:36.207+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:36.208090+0000) 2024-02-19T20:19:36.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:36.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:36.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:36.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:36.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:36.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:36.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:36.412+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:36.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:36.667+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:36.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:36.667+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:36.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:36.667+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:36.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:36.668+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:37.211 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:36.909+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:36.910597+0000) 2024-02-19T20:19:37.211 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:37.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:37.211 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:37.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:37.211 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:37.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:37.212 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:37.083+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:37.212 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:37.128+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:37.129338+0000) 2024-02-19T20:19:37.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:37.210+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:37.211234+0000) 2024-02-19T20:19:37.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:37.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:37.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:37.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:37.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:37.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:37.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:37.458+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:37.913 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:37.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:37.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:37.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:37.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:37.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:37.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:37.648+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:38.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:37.912+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:37.913680+0000) 2024-02-19T20:19:38.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:38.125+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:38.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:38.125+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:38.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:38.125+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:38.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:38.125+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:38.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:38.131+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:38.132677+0000) 2024-02-19T20:19:38.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:38.212+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:38.213546+0000) 2024-02-19T20:19:38.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:38.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:38.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:38.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:38.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:38.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:38.523 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:38.412+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:38.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:38.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:38.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:38.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:38.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:38.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:38.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:38.630+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:39.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:38.913+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:38.914030+0000) 2024-02-19T20:19:39.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:39.132+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:39.133118+0000) 2024-02-19T20:19:39.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:39.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:39.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:39.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:39.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:39.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:39.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:39.148+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:39.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:39.212+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:39.213933+0000) 2024-02-19T20:19:39.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:39.382+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:39.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:39.382+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:39.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:39.382+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:39.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:39.383+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:39.867 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:19:39.746+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57700 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:19:39.867 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:39.743+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:39.744277+0000) 2024-02-19T20:19:39.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:39.743+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:39.744957+0000) 2024-02-19T20:19:39.868 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:39.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:39.868 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:39.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:39.868 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:39.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:39.868 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:39.633+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:39.868 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:39.745+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:39.746347+0000) 2024-02-19T20:19:40.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:39.913+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:39.914243+0000) 2024-02-19T20:19:40.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:40.213+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:40.214117+0000) 2024-02-19T20:19:40.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:40.377+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:40.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:40.377+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:40.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:40.377+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:40.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:40.378+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:40.132+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:40.133220+0000) 2024-02-19T20:19:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:40.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:40.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:40.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:40.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:40.174+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:40.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:40.175+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:40.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:40.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:40.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:40.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:40.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:40.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:40.918 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:40.675+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:41.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:40.915+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:40.916818+0000) 2024-02-19T20:19:41.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:41.134+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:41.135794+0000) 2024-02-19T20:19:41.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:41.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:41.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:41.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:41.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:41.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:41.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:41.167+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:41.215+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:41.216257+0000) 2024-02-19T20:19:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:41.389+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:41.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:41.389+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:41.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:41.389+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:41.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:41.389+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:41.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:41.673+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:41.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:41.673+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:41.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:41.673+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:41.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:41.673+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:41.919 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:41.917+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:41.918907+0000) 2024-02-19T20:19:42.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:42.138+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:42.139329+0000) 2024-02-19T20:19:42.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:42.203+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:42.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:42.203+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:42.205 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:42.203+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:42.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:42.203+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:42.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:42.218+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:42.219854+0000) 2024-02-19T20:19:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:42.397+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:42.397+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:42.397+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:42.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:42.397+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:42.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:42.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:42.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:42.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:42.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:42.669+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:42.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:42.670+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:43.222 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:42.920+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:42.921446+0000) 2024-02-19T20:19:43.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:43.140+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:43.141876+0000) 2024-02-19T20:19:43.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:43.161+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:43.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:43.161+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:43.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:43.161+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:43.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:43.161+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:43.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:43.221+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:43.222616+0000) 2024-02-19T20:19:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:43.395+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:43.395+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:43.395+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:43.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:43.395+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:43.806 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:43.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:43.806 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:43.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:43.806 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:43.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:43.806 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:43.671+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:44.082 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:43.920+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:43.921885+0000) 2024-02-19T20:19:44.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:44.222+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:44.223026+0000) 2024-02-19T20:19:44.354 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:44.141+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:44.142229+0000) 2024-02-19T20:19:44.354 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:44.173+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:44.354 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:44.173+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:44.354 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:44.173+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:44.354 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:44.174+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:44.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:44.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:44.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:44.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:44.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:44.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:44.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:44.432+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:45.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:19:44.753+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57741 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:19:45.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:44.750+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:44.751359+0000) 2024-02-19T20:19:45.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:44.750+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:44.751960+0000) 2024-02-19T20:19:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:44.713+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:45.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:44.713+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:45.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:44.713+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:45.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:44.713+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:45.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:44.750+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:44.751890+0000) 2024-02-19T20:19:45.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:44.921+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:44.922077+0000) 2024-02-19T20:19:45.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:45.222+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:45.223187+0000) 2024-02-19T20:19:45.354 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:45.141+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:45.142384+0000) 2024-02-19T20:19:45.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:45.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:45.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:45.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:45.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:45.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:45.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:45.216+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:45.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:45.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:45.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:45.412+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:45.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:45.413+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:45.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:45.413+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:46.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:45.717+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:46.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:45.717+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:46.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:45.717+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:46.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:45.718+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:46.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:45.923+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:45.924125+0000) 2024-02-19T20:19:46.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:46.224+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:46.225884+0000) 2024-02-19T20:19:46.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:46.143+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:46.144430+0000) 2024-02-19T20:19:46.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:46.231+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:46.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:46.231+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:46.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:46.231+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:46.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:46.231+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:46.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:46.371+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:46.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:46.371+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:46.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:46.371+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:46.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:46.372+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:47.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:46.716+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:47.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:46.716+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:46.716+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:46.717+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:47.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:46.925+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:46.926222+0000) 2024-02-19T20:19:47.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:47.227+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:47.228337+0000) 2024-02-19T20:19:47.354 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:47.146+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:47.147968+0000) 2024-02-19T20:19:47.354 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:47.229+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:47.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:47.229+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:47.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:47.229+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:47.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:47.230+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:47.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:47.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:47.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:47.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:47.693 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:47.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:47.693 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:47.418+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:48.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:47.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:48.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:47.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:47.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:47.692+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:48.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:47.927+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:47.928322+0000) 2024-02-19T20:19:48.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:48.230+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:48.231746+0000) 2024-02-19T20:19:48.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:48.149+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:48.150425+0000) 2024-02-19T20:19:48.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:48.222+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:48.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:48.222+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:48.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:48.222+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:48.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:48.223+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:48.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:48.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:48.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:48.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:48.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:48.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:48.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:48.455+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:49.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:48.734+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:49.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:48.734+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:48.734+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:48.735+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:49.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:48.927+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:48.928630+0000) 2024-02-19T20:19:49.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:49.231+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:49.232147+0000) 2024-02-19T20:19:49.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:49.149+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:49.150758+0000) 2024-02-19T20:19:49.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:49.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:49.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:49.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:49.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:49.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:49.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:49.259+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:49.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:49.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:49.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:49.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:49.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:49.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:49.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:49.437+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:50.019 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:19:49.760+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57781 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:19:50.020 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:49.757+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:49.758926+0000) 2024-02-19T20:19:50.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:49.758+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:49.759067+0000) 2024-02-19T20:19:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:49.757+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:49.758098+0000) 2024-02-19T20:19:50.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:49.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:50.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:49.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:50.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:49.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:50.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:49.775+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:50.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:49.927+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:49.928821+0000) 2024-02-19T20:19:50.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:50.231+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:50.232294+0000) 2024-02-19T20:19:50.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:50.149+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:50.150961+0000) 2024-02-19T20:19:50.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:50.215+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:50.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:50.215+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:50.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:50.215+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:50.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:50.216+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:50.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:50.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:50.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:50.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:50.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:50.420+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:50.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:50.421+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:51.082 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:50.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:51.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:50.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:51.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:50.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:51.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:50.788+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:51.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:50.929+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:50.930843+0000) 2024-02-19T20:19:51.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:51.234+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:51.235586+0000) 2024-02-19T20:19:51.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:51.151+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:51.152866+0000) 2024-02-19T20:19:51.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:51.192+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:51.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:51.192+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:51.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:51.192+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:51.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:51.192+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:51.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:51.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:51.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:51.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:51.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:51.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:51.423+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:52.082 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:51.827+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:52.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:51.827+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:52.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:51.827+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:52.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:51.828+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:52.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:51.931+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:51.932970+0000) 2024-02-19T20:19:52.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:52.237+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:52.238925+0000) 2024-02-19T20:19:52.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:52.154+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:52.155098+0000) 2024-02-19T20:19:52.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:52.218+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:52.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:52.218+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:52.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:52.218+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:52.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:52.219+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:52.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:52.456+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:52.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:52.456+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:52.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:52.456+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:52.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:52.457+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:53.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:52.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:53.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:52.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:53.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:52.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:53.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:52.852+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:53.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:52.935+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:52.936541+0000) 2024-02-19T20:19:53.475 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:53.240+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:53.241565+0000) 2024-02-19T20:19:53.475 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:53.156+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:53.157201+0000) 2024-02-19T20:19:53.475 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:53.223+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:53.476 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:53.223+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:53.476 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:53.223+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:53.476 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:53.224+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:53.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:53.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:53.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:53.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:53.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:53.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:53.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:53.474+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:54.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:53.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:54.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:53.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:54.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:53.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:54.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:53.832+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:54.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:53.936+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:53.937016+0000) 2024-02-19T20:19:54.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:54.241+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:54.242029+0000) 2024-02-19T20:19:54.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:54.156+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:54.157617+0000) 2024-02-19T20:19:54.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:54.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:54.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:54.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:54.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:54.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:54.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:54.179+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:54.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:54.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:54.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:54.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:54.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:54.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:54.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:54.454+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:55.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:54.764+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:54.765571+0000) 2024-02-19T20:19:55.020 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:19:54.767+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57822 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:19:55.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:54.764+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:54.765579+0000) 2024-02-19T20:19:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:54.764+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:54.765913+0000) 2024-02-19T20:19:55.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:54.832+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:54.832+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:54.832+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:54.833+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:55.021 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:54.936+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:54.937178+0000) 2024-02-19T20:19:55.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:55.241+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:55.242200+0000) 2024-02-19T20:19:55.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:55.156+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:55.157811+0000) 2024-02-19T20:19:55.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:55.202+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:55.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:55.202+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:55.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:55.202+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:55.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:55.203+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:55.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:55.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:55.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:55.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:55.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:55.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:55.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:55.473+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:56.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:55.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:56.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:55.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:56.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:55.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:56.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:55.800+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:56.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:55.939+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:55.940616+0000) 2024-02-19T20:19:56.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:56.243+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:56.244761+0000) 2024-02-19T20:19:56.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:56.159+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:56.160324+0000) 2024-02-19T20:19:56.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:56.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:56.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:56.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:56.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:56.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:56.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:56.178+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:56.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:56.504+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:56.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:56.504+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:56.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:56.504+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:56.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:56.505+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:57.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:56.838+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:57.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:56.838+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:57.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:56.838+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:57.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:56.838+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:57.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:56.941+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:56.942838+0000) 2024-02-19T20:19:57.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:57.245+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:57.246889+0000) 2024-02-19T20:19:57.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:57.162+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:57.163659+0000) 2024-02-19T20:19:57.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:57.194+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:57.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:57.194+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:57.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:57.194+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:57.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:57.195+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:57.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:57.477+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:57.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:57.477+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:57.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:57.477+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:57.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:57.477+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:58.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:57.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:58.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:57.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:58.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:57.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:58.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:57.861+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:58.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:57.943+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:57.944880+0000) 2024-02-19T20:19:58.461 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:58.247+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:58.248863+0000) 2024-02-19T20:19:58.462 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:58.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:58.462 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:58.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:58.462 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:58.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:58.462 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:58.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:58.462 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:58.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:58.462 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:58.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:58.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:58.165+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:58.166088+0000) 2024-02-19T20:19:58.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:58.165+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:58.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:58.461+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:59.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:58.817+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:59.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:58.817+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:59.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:58.817+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:19:59.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:58.818+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:19:59.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:58.944+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:58.945215+0000) 2024-02-19T20:19:59.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:59.248+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:59.249347+0000) 2024-02-19T20:19:59.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:59.165+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:59.166647+0000) 2024-02-19T20:19:59.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:59.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:59.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:59.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:59.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:59.204+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:19:59.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:59.205+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:19:59.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:59.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:59.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:59.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:59.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:59.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:19:59.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:59.511+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:00.083 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:19:59.774+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57862 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:20:00.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:19:59.772+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:59.773581+0000) 2024-02-19T20:20:00.084 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:19:59.772+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:59.773694+0000) 2024-02-19T20:20:00.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:59.772+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:59.773107+0000) 2024-02-19T20:20:00.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:59.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:00.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:59.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:00.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:59.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:00.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:59.807+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:00.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:19:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:19:59.944+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:59.945378+0000) 2024-02-19T20:20:00.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:00.248+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:00.249515+0000) 2024-02-19T20:20:00.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:00.165+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:00.166837+0000) 2024-02-19T20:20:00.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:00.173+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:00.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:00.173+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:00.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:00.173+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:00.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:00.173+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:00.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:00.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:00.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:00.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:00.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:00.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:00.548+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:01.083 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:00.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:01.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:00.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:01.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:00.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:01.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:00.820+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:01.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:00.946+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:00.947523+0000) 2024-02-19T20:20:01.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:01.251+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:01.252921+0000) 2024-02-19T20:20:01.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:01.131+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:01.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:01.131+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:01.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:01.131+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:01.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:01.132+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:01.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:01.169+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:01.170328+0000) 2024-02-19T20:20:01.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:01.504+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:01.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:01.504+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:01.504+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:01.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:01.504+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:02.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:01.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:02.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:01.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:02.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:01.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:02.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:01.862+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:02.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:01.948+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:01.949581+0000) 2024-02-19T20:20:02.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:02.254+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:02.255047+0000) 2024-02-19T20:20:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:02.171+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:02.172458+0000) 2024-02-19T20:20:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:02.179+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:02.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:02.179+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:02.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:02.179+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:02.521 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:02.180+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:02.823 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:02.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:02.823 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:02.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:02.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:02.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:02.824 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:02.527+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:03.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:02.821+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:03.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:02.821+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:03.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:02.821+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:03.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:02.822+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:03.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:02.952+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:02.953191+0000) 2024-02-19T20:20:03.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:03.254+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:03.256996+0000) 2024-02-19T20:20:03.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:03.173+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:03.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:03.173+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:03.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:03.173+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:03.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:03.173+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:03.174961+0000) 2024-02-19T20:20:03.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:03.173+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:03.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:03.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:03.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:03.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:03.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:03.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:03.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:03.528+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:04.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:03.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:04.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:03.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:04.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:03.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:04.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:03.856+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:04.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:03.952+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:03.953757+0000) 2024-02-19T20:20:04.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:04.256+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:04.257497+0000) 2024-02-19T20:20:04.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:04.174+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:04.175662+0000) 2024-02-19T20:20:04.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:04.181+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:04.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:04.181+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:04.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:04.181+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:04.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:04.182+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:04.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:04.533+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:04.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:04.533+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:04.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:04.534+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:04.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:04.534+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:05.088 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:20:04.781+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57903 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:20:05.088 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:04.780+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:04.781052+0000) 2024-02-19T20:20:05.089 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:04.779+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:04.780047+0000) 2024-02-19T20:20:05.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:04.778+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:04.779750+0000) 2024-02-19T20:20:05.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:04.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:05.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:04.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:05.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:04.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:05.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:04.907+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:05.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:04.952+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:04.953927+0000) 2024-02-19T20:20:05.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:05.256+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:05.257728+0000) 2024-02-19T20:20:05.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:05.174+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:05.175916+0000) 2024-02-19T20:20:05.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:05.209+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:05.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:05.209+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:05.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:05.209+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:05.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:05.210+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:05.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:05.547+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:05.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:05.547+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:05.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:05.547+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:05.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:05.548+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:06.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:05.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:06.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:05.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:06.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:05.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:06.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:05.889+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:06.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:05.956+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:05.957312+0000) 2024-02-19T20:20:06.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:06.259+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:06.260046+0000) 2024-02-19T20:20:06.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:06.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:06.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:06.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:06.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:06.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:06.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:06.170+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:06.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:06.177+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:06.178533+0000) 2024-02-19T20:20:06.835 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:06.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:06.835 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:06.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:06.835 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:06.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:06.836 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:06.551+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:07.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:06.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:07.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:06.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:07.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:06.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:07.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:06.910+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:07.091 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:06.959+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:06.960660+0000) 2024-02-19T20:20:07.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:07.261+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:07.262218+0000) 2024-02-19T20:20:07.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:07.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:07.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:07.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:07.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:07.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:07.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:07.163+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:07.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:07.179+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:07.180637+0000) 2024-02-19T20:20:07.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:07.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:07.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:07.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:07.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:07.556+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:08.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:08.181+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:08.182804+0000) 2024-02-19T20:20:08.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:08.190+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:08.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:08.190+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:08.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:08.190+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:08.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:08.191+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:08.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:07.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:08.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:07.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:08.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:07.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:08.266 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:07.943+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:08.266 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:07.963+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:07.964198+0000) 2024-02-19T20:20:08.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:08.263+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:08.264396+0000) 2024-02-19T20:20:08.894 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:08.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:08.895 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:08.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:08.895 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:08.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:08.895 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:08.532+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:09.150 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:08.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:09.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:08.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:09.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:08.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:09.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:08.894+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:09.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:08.963+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:08.964724+0000) 2024-02-19T20:20:09.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:09.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:09.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:09.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:09.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:09.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:09.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:09.150+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:09.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:09.182+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:09.183183+0000) 2024-02-19T20:20:09.521 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:09.263+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:09.264893+0000) 2024-02-19T20:20:09.807 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:20:09.789+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57943 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:20:09.808 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:09.786+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:09.787116+0000) 2024-02-19T20:20:09.808 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:09.787+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:09.788084+0000) 2024-02-19T20:20:09.808 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:09.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:09.808 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:09.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:09.809 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:09.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:09.809 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:09.556+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:09.809 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:09.786+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:09.787340+0000) 2024-02-19T20:20:10.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:09.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:10.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:09.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:10.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:09.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:10.085 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:09.919+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:10.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:09.963+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:09.964927+0000) 2024-02-19T20:20:10.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:10.264+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:10.265047+0000) 2024-02-19T20:20:10.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:10.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:10.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:10.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:10.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:10.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:10.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:10.109+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:10.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:10.182+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:10.183346+0000) 2024-02-19T20:20:10.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:10.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:10.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:10.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:10.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:10.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:10.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:10.553+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:11.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:11.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:11.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:11.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:11.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:11.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:11.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:11.094+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:11.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:10.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:11.188 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:10.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:11.188 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:10.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:11.188 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:10.901+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:11.188 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:10.966+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:10.967067+0000) 2024-02-19T20:20:11.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:11.185+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:11.186865+0000) 2024-02-19T20:20:11.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:11.266+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:11.267781+0000) 2024-02-19T20:20:11.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:11.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:11.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:11.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:11.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:11.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:11.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:11.511+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:12.190 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:12.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:12.190 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:12.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:12.190 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:12.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:12.190 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:12.120+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:12.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:11.923+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:12.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:11.923+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:12.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:11.923+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:12.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:11.924+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:12.191 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:11.968+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:11.969408+0000) 2024-02-19T20:20:12.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:12.189+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:12.190281+0000) 2024-02-19T20:20:12.463 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:12.268+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:12.269939+0000) 2024-02-19T20:20:12.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:12.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:12.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:12.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:12.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:12.462+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:12.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:12.462+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:13.192 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:13.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:13.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:13.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:13.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:13.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:13.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:13.096+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:13.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:12.880+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:13.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:12.880+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:13.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:12.880+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:13.193 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:12.880+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:13.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:12.970+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:12.971500+0000) 2024-02-19T20:20:13.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:13.191+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:13.192653+0000) 2024-02-19T20:20:13.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:13.271+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:13.272426+0000) 2024-02-19T20:20:13.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:13.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:13.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:13.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:13.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:13.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:13.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:13.513+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:14.085 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:14.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:14.085 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:14.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:14.085 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:14.052+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:14.085 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:14.052+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:14.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:13.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:14.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:13.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:14.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:13.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:14.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:13.833+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:14.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:13.971+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:13.972001+0000) 2024-02-19T20:20:14.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:14.192+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:14.193171+0000) 2024-02-19T20:20:14.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:14.271+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:14.272802+0000) 2024-02-19T20:20:14.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:14.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:14.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:14.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:14.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:14.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:14.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:14.523+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:15.085 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:20:14.796+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 57984 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:20:15.086 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:14.793+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:14.794719+0000) 2024-02-19T20:20:15.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:14.793+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:14.794837+0000) 2024-02-19T20:20:15.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:15.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:15.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:15.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:15.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:15.031+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:15.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:15.031+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:15.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:14.793+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:14.794645+0000) 2024-02-19T20:20:15.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:14.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:15.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:14.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:15.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:14.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:15.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:14.836+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:15.088 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:14.971+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:14.972172+0000) 2024-02-19T20:20:15.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:15.192+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:15.193366+0000) 2024-02-19T20:20:15.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:15.271+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:15.272955+0000) 2024-02-19T20:20:15.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:15.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:15.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:15.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:15.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:15.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:15.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:15.483+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:16.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:16.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:16.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:16.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:16.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:16.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:16.080 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:16.060+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:16.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:15.817+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:16.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:15.817+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:16.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:15.817+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:16.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:15.818+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:16.081 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:15.973+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:15.974108+0000) 2024-02-19T20:20:16.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:16.195+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:16.196880+0000) 2024-02-19T20:20:16.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:16.275+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:16.276202+0000) 2024-02-19T20:20:16.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:16.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:16.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:16.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:16.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:16.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:16.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:16.531+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:17.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:16.843+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:17.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:16.843+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:17.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:16.843+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:17.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:16.843+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:17.199 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:16.976+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:16.977656+0000) 2024-02-19T20:20:17.199 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:17.027+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:17.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:17.027+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:17.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:17.027+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:17.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:17.028+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:17.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:17.198+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:17.198994+0000) 2024-02-19T20:20:17.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:17.278+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:17.279514+0000) 2024-02-19T20:20:17.979 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:17.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:17.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:17.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:17.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:17.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:17.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:17.817+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:17.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:17.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:17.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:17.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:17.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:17.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:17.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:17.575+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:18.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:17.978+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:17.979688+0000) 2024-02-19T20:20:18.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:18.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:18.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:18.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:18.050+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:18.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:18.200+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:18.201567+0000) 2024-02-19T20:20:18.579 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:18.281+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:18.282190+0000) 2024-02-19T20:20:18.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:18.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:18.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:18.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:18.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:18.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:18.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:18.772+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:18.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:18.578+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:18.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:18.578+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:18.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:18.578+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:18.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:18.579+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:19.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:18.979+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:18.980137+0000) 2024-02-19T20:20:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:19.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:19.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:19.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:19.062+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:19.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:19.201+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:19.202030+0000) 2024-02-19T20:20:19.537 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:19.281+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:19.282753+0000) 2024-02-19T20:20:19.537 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:19.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:19.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:19.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:19.538 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:19.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:19.798 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:19.537+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:19.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:19.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:19.799 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:19.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:19.799 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:19.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:19.799 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:19.786+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:20.086 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:20:19.806+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58024 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:20:20.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:19.802+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:19.803102+0000) 2024-02-19T20:20:20.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:19.979+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:19.980876+0000) 2024-02-19T20:20:20.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:19.801+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:19.802097+0000) 2024-02-19T20:20:20.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:20.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:20.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:20.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:20.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:20.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:20.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:20.029+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:20.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:19.801+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:19.802017+0000) 2024-02-19T20:20:20.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:20.201+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:20.202208+0000) 2024-02-19T20:20:20.358 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:20.281+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:20.282952+0000) 2024-02-19T20:20:20.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:20.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:20.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:20.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:20.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:20.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:20.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:20.537+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:21.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:20.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:21.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:20.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:21.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:20.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:21.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:20.991+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:21.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:20.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:21.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:20.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:21.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:20.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:21.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:20.798+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:21.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:20.982+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:20.983063+0000) 2024-02-19T20:20:21.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:21.203+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:21.204201+0000) 2024-02-19T20:20:21.358 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:21.284+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:21.285073+0000) 2024-02-19T20:20:21.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:21.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:21.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:21.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:21.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:21.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:21.513+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:22.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:21.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:22.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:21.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:22.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:21.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:22.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:21.789+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:22.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:21.985+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:21.986556+0000) 2024-02-19T20:20:22.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:21.962+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:22.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:21.962+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:22.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:21.962+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:22.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:21.962+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:22.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:22.205+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:22.206201+0000) 2024-02-19T20:20:22.358 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:22.286+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:22.287208+0000) 2024-02-19T20:20:22.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:22.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:22.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:22.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:22.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:22.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:22.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:22.511+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:23.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:22.963+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:23.301 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:22.963+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:23.301 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:22.963+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:23.301 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:22.964+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:23.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:22.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:23.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:22.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:23.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:22.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:23.302 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:22.829+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:23.303 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:22.987+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:22.988595+0000) 2024-02-19T20:20:23.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:23.207+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:23.208223+0000) 2024-02-19T20:20:23.358 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:23.288+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:23.289224+0000) 2024-02-19T20:20:23.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:23.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:23.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:23.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:23.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:23.549+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:23.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:23.549+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:24.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:23.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:24.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:23.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:24.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:23.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:24.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:23.983+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:24.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:23.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:24.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:23.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:24.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:23.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:24.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:23.856+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:24.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:23.988+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:23.989106+0000) 2024-02-19T20:20:24.358 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:24.288+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:24.289802+0000) 2024-02-19T20:20:24.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:24.207+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:24.208643+0000) 2024-02-19T20:20:24.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:24.566+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:24.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:24.566+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:24.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:24.566+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:24.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:24.566+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:25.086 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:20:24.813+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58065 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:20:25.087 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:24.810+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:24.811801+0000) 2024-02-19T20:20:25.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:24.810+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:24.811331+0000) 2024-02-19T20:20:25.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:24.943+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:25.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:24.943+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:25.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:24.943+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:25.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:24.944+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:25.088 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:24.811+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:24.812041+0000) 2024-02-19T20:20:25.088 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:24.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:25.088 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:24.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:25.088 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:24.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:25.088 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:24.856+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:25.088 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:24.988+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:24.989282+0000) 2024-02-19T20:20:25.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:25.289+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:25.290019+0000) 2024-02-19T20:20:25.360 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:25.207+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:25.208813+0000) 2024-02-19T20:20:25.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:25.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:25.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:25.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:25.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:25.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:25.770 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:25.573+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:26.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:25.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:26.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:25.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:26.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:25.909+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:26.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:25.910+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:26.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:25.884+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:26.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:25.884+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:26.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:25.884+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:26.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:25.885+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:26.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:25.991+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:25.992524+0000) 2024-02-19T20:20:26.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:26.209+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:26.210894+0000) 2024-02-19T20:20:26.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:26.291+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:26.292018+0000) 2024-02-19T20:20:26.859 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:26.596+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:26.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:26.596+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:26.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:26.596+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:26.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:26.596+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:27.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:26.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:27.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:26.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:27.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:26.858+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:27.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:26.859+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:27.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:26.993+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:26.994914+0000) 2024-02-19T20:20:27.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:26.864+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:27.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:26.864+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:27.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:26.864+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:27.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:26.865+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:27.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:27.211+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:27.212956+0000) 2024-02-19T20:20:27.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:27.293+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:27.294596+0000) 2024-02-19T20:20:27.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:27.634+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:27.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:27.634+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:27.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:27.634+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:27.870 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:27.634+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:27.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:27.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:27.901+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:28.217 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:27.902+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:28.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:27.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:28.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:27.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:28.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:27.868+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:28.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:27.869+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:28.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:27.996+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:27.997423+0000) 2024-02-19T20:20:28.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:28.296+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:28.297231+0000) 2024-02-19T20:20:28.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:28.213+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:28.214971+0000) 2024-02-19T20:20:28.909 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:28.596+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:28.909 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:28.597+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:28.909 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:28.597+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:28.909 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:28.597+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:29.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:28.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:29.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:28.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:29.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:28.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:29.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:28.909+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:29.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:28.997+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:28.998023+0000) 2024-02-19T20:20:29.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:28.951+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:29.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:28.951+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:29.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:28.951+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:29.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:28.951+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:29.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:29.214+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:29.215524+0000) 2024-02-19T20:20:29.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:29.296+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:29.297824+0000) 2024-02-19T20:20:29.938 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:20:29.820+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58105 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:20:29.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:29.818+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:29.819603+0000) 2024-02-19T20:20:29.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:29.817+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:29.818678+0000) 2024-02-19T20:20:29.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:29.643+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:29.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:29.643+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:29.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:29.643+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:29.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:29.643+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:29.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:29.817+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:29.818274+0000) 2024-02-19T20:20:30.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:29.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:29.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:29.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:30.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:29.968+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:30.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:29.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:30.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:29.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:30.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:29.937+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:30.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:29.938+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:30.217 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:29.997+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:29.998195+0000) 2024-02-19T20:20:30.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:30.297+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:30.298028+0000) 2024-02-19T20:20:30.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:30.214+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:30.215745+0000) 2024-02-19T20:20:30.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:30.644+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:30.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:30.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:30.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:30.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:30.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:30.645+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:31.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:31.014+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:31.014+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:31.014+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:31.015+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:31.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:31.217+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:31.217958+0000) 2024-02-19T20:20:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:30.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:31.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:30.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:30.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:30.979+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:31.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:30.999+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:31.000667+0000) 2024-02-19T20:20:31.594 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:31.299+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:31.300775+0000) 2024-02-19T20:20:31.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:31.668+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:31.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:31.668+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:31.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:31.668+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:31.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:31.669+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:32.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:31.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:31.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:31.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:31.971+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:32.002+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:32.003249+0000) 2024-02-19T20:20:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:31.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:31.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:31.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:31.989+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:32.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:32.219+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:32.220345+0000) 2024-02-19T20:20:32.595 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:32.301+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:32.302838+0000) 2024-02-19T20:20:32.995 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:32.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:32.995 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:32.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:32.995 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:32.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:32.995 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:32.680+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:33.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:33.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:33.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:33.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:33.033+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:33.221+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:33.222867+0000) 2024-02-19T20:20:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:32.994+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:32.994+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:32.994+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:32.995+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:33.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:33.004+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:33.005467+0000) 2024-02-19T20:20:33.581 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:33.304+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:33.305068+0000) 2024-02-19T20:20:33.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:33.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:33.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:33.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:33.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:33.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:33.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:33.714+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:34.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:33.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:33.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:33.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:33.983+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:34.005+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:34.006021+0000) 2024-02-19T20:20:34.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:34.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:34.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:34.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:34.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:34.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:34.069+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:34.222+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:34.223372+0000) 2024-02-19T20:20:34.595 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:34.304+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:34.305486+0000) 2024-02-19T20:20:34.845 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:20:34.827+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58146 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:20:34.846 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:34.825+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:34.826576+0000) 2024-02-19T20:20:34.846 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:34.825+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:34.826072+0000) 2024-02-19T20:20:34.846 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:34.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:34.846 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:34.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:34.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:34.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:34.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:34.680+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:34.847 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:34.824+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:34.825841+0000) 2024-02-19T20:20:35.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:35.040+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:35.224 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:35.040+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:35.224 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:35.040+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:35.224 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:35.041+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:35.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:35.005+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:35.006172+0000) 2024-02-19T20:20:35.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:35.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:35.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:35.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:35.225 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:35.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:35.225 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:35.019+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:35.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:35.304+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:35.305668+0000) 2024-02-19T20:20:35.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:35.222+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:35.223618+0000) 2024-02-19T20:20:36.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:35.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:36.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:35.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:36.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:35.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:36.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:35.698+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:36.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:36.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:36.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:36.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:36.065+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:36.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:36.225+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:36.226227+0000) 2024-02-19T20:20:36.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:36.007+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:36.008058+0000) 2024-02-19T20:20:36.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:36.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:36.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:36.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:36.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:36.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:36.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:36.012+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:36.598 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:36.308+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:36.308997+0000) 2024-02-19T20:20:36.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:36.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:36.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:36.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:36.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:36.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:36.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:36.698+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:37.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:37.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:37.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:37.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:37.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:37.029+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:37.227+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:37.228643+0000) 2024-02-19T20:20:37.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:36.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:37.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:36.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:37.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:36.983+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:37.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:36.983+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:37.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:37.009+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:37.010254+0000) 2024-02-19T20:20:37.598 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:37.311+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:37.312587+0000) 2024-02-19T20:20:37.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:37.704+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:37.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:37.704+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:37.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:37.704+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:37.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:37.705+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:38.231 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:38.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:38.231 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:38.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:38.231 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:38.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:38.231 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:38.035+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:38.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:37.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:38.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:37.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:38.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:37.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:38.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:37.963+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:38.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:38.011+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:38.012328+0000) 2024-02-19T20:20:38.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:38.230+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:38.231228+0000) 2024-02-19T20:20:38.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:38.315+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:38.316152+0000) 2024-02-19T20:20:39.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:38.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:39.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:38.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:39.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:38.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:39.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:38.723+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:39.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:39.003+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:39.003+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:39.003+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:39.004+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:39.011+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:39.012923+0000) 2024-02-19T20:20:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:39.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:39.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:39.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:39.052+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:39.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:39.230+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:39.231745+0000) 2024-02-19T20:20:39.598 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:39.315+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:39.316789+0000) 2024-02-19T20:20:39.853 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:20:39.835+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58186 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:20:39.854 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:39.832+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:39.833425+0000) 2024-02-19T20:20:39.854 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:39.832+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:39.833114+0000) 2024-02-19T20:20:39.854 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:39.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:39.855 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:39.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:39.855 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:39.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:39.855 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:39.762+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:39.855 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:39.832+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:39.833490+0000) 2024-02-19T20:20:40.232 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:40.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:40.232 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:40.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:40.232 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:40.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:40.232 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:40.009+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:40.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:40.012+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:40.013108+0000) 2024-02-19T20:20:40.233 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:40.024+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:40.233 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:40.024+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:40.233 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:40.024+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:40.233 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:40.025+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:40.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:40.230+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:40.231947+0000) 2024-02-19T20:20:40.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:40.316+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:40.316978+0000) 2024-02-19T20:20:41.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:40.764+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:41.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:40.764+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:41.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:40.764+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:41.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:40.764+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:41.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:41.023+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:41.023+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:41.023+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:41.024+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:41.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:41.233+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:41.234531+0000) 2024-02-19T20:20:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:41.014+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:41.015710+0000) 2024-02-19T20:20:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:41.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:41.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:41.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:41.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:41.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:41.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:41.020+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:41.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:41.318+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:41.319599+0000) 2024-02-19T20:20:42.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:41.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:42.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:41.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:42.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:41.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:42.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:41.802+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:42.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:42.018+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:42.019200+0000) 2024-02-19T20:20:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:42.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:42.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:42.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:42.021+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:42.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:42.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:42.010+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:42.011+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:42.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:42.236+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:42.237384+0000) 2024-02-19T20:20:42.601 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:42.321+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:42.322211+0000) 2024-02-19T20:20:43.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:42.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:43.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:42.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:43.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:42.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:43.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:42.763+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:43.284 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:43.020+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:43.021561+0000) 2024-02-19T20:20:43.284 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:43.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:43.284 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:43.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:43.284 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:43.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:43.284 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:43.037+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:43.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:43.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:43.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:43.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:43.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:43.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:43.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:43.033+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:43.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:43.239+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:43.240024+0000) 2024-02-19T20:20:43.581 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:43.324+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:43.325834+0000) 2024-02-19T20:20:44.014 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:43.755+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:44.014 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:43.755+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:44.014 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:43.755+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:44.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:43.755+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:44.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:44.021+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:44.022146+0000) 2024-02-19T20:20:44.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:44.046+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:44.046+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:44.046+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:44.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:44.047+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:44.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:44.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:44.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:44.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:44.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:44.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:44.014+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:44.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:44.239+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:44.240451+0000) 2024-02-19T20:20:44.601 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:44.325+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:44.326458+0000) 2024-02-19T20:20:44.859 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:20:44.842+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58228 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:20:44.859 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:44.840+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:44.841305+0000) 2024-02-19T20:20:44.860 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:44.838+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:44.839793+0000) 2024-02-19T20:20:44.860 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:44.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:44.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:44.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:44.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:44.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:44.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:44.758+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:44.861 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:44.840+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:44.841435+0000) 2024-02-19T20:20:45.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:45.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:45.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:45.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:45.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:45.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:45.241 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:45.058+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:45.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:45.021+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:45.022358+0000) 2024-02-19T20:20:45.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:45.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:45.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:45.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:45.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:45.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:45.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:45.050+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:45.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:45.239+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:45.240613+0000) 2024-02-19T20:20:45.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:45.325+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:45.326644+0000) 2024-02-19T20:20:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:45.783+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:45.783+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:45.783+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:45.783+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:46.284 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:46.024+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:46.025874+0000) 2024-02-19T20:20:46.284 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:46.085+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:46.284 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:46.085+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:46.285 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:46.085+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:46.285 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:46.086+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:46.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:46.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:46.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:46.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:46.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:46.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:46.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:46.067+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:46.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:46.242+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:46.243866+0000) 2024-02-19T20:20:46.604 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:46.328+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:46.329144+0000) 2024-02-19T20:20:47.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:46.743+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:47.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:46.743+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:47.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:46.743+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:47.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:46.743+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:47.284 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:47.028+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:47.029344+0000) 2024-02-19T20:20:47.285 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:47.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:47.285 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:47.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:47.285 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:47.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:47.285 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:47.073+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:47.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:47.045+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:47.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:47.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:47.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:47.046+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:47.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:47.047+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:47.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:47.245+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:47.246010+0000) 2024-02-19T20:20:47.605 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:47.330+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:47.331671+0000) 2024-02-19T20:20:48.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:47.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:48.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:47.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:48.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:47.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:48.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:47.763+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:48.284 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:48.031+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:48.032059+0000) 2024-02-19T20:20:48.285 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:48.043+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:48.285 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:48.043+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:48.285 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:48.043+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:48.285 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:48.044+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:48.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:48.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:48.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:48.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:48.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:48.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:48.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:48.045+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:48.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:48.248+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:48.248389+0000) 2024-02-19T20:20:48.606 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:48.333+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:48.334167+0000) 2024-02-19T20:20:49.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:48.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:49.056 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:48.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:49.056 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:48.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:49.056 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:48.776+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:49.284 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:49.031+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:49.032566+0000) 2024-02-19T20:20:49.285 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:49.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:49.285 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:49.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:49.285 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:49.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:49.285 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:49.051+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:49.285 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:49.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:49.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:49.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:49.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:49.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:49.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:49.064+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:49.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:49.248+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:49.249004+0000) 2024-02-19T20:20:49.605 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:49.333+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:49.334609+0000) 2024-02-19T20:20:49.856 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:20:49.850+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58268 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:20:49.857 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:49.846+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:49.847541+0000) 2024-02-19T20:20:49.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:49.846+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:49.847319+0000) 2024-02-19T20:20:49.857 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:49.826+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:49.857 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:49.826+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:49.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:49.826+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:49.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:49.826+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:49.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:49.846+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:49.847776+0000) 2024-02-19T20:20:50.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:50.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:50.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:50.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:50.143 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:50.037+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:50.143 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:50.038+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:50.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:50.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:50.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:50.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:50.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:50.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:50.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:50.021+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:50.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:50.031+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:50.032736+0000) 2024-02-19T20:20:50.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:50.332+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:50.334807+0000) 2024-02-19T20:20:50.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:50.247+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:50.249192+0000) 2024-02-19T20:20:51.252 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:51.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:51.252 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:51.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:51.253 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:51.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:51.253 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:51.049+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:51.253 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:50.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:51.253 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:50.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:51.253 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:50.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:51.253 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:50.857+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:51.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:51.033+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:51.034762+0000) 2024-02-19T20:20:51.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:51.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:51.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:51.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:51.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:51.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:51.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:51.041+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:51.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:51.251+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:51.252443+0000) 2024-02-19T20:20:51.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:51.336+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:51.337253+0000) 2024-02-19T20:20:52.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:51.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:52.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:51.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:52.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:51.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:52.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:51.814+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:52.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:52.036+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:52.036989+0000) 2024-02-19T20:20:52.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:52.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:52.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:52.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:52.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:52.072+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:52.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:52.074+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:52.341 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:52.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:52.341 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:52.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:52.342 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:52.044+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:52.342 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:52.046+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:52.342 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:52.254+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:52.254972+0000) 2024-02-19T20:20:52.608 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:52.338+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:52.339688+0000) 2024-02-19T20:20:52.915 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:52.770+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:52.915 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:52.770+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:52.915 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:52.770+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:52.915 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:52.771+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:53.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:53.038+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:53.039389+0000) 2024-02-19T20:20:53.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:53.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:53.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:53.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:53.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:53.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:53.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:53.040+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:53.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:53.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:53.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:53.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:53.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:53.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:53.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:53.056+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:53.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:53.256+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:53.257093+0000) 2024-02-19T20:20:53.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:53.341+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:53.342321+0000) 2024-02-19T20:20:53.998 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:53.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:53.998 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:53.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:53.998 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:53.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:53.998 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:53.765+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:54.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:53.996+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:54.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:53.996+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:54.265 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:53.996+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:54.266 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:53.997+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:54.266 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:54.038+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:54.039879+0000) 2024-02-19T20:20:54.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:54.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:54.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:54.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:54.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:54.077+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:54.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:54.077+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:54.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:54.256+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:54.257554+0000) 2024-02-19T20:20:54.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:54.342+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:54.342974+0000) 2024-02-19T20:20:54.864 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:20:54.857+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58309 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:20:54.864 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:54.853+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:54.854825+0000) 2024-02-19T20:20:54.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:54.855+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:54.856516+0000) 2024-02-19T20:20:54.865 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:54.733+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:54.865 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:54.733+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:54.865 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:54.733+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:54.866 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:54.734+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:54.866 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:54.854+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:54.855319+0000) 2024-02-19T20:20:55.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:55.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:55.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:55.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:55.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:55.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:55.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:55.029+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:55.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:55.039+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:55.040062+0000) 2024-02-19T20:20:55.259 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:55.080+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:55.259 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:55.080+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:55.259 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:55.080+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:55.259 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:55.081+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:55.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:55.256+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:55.257812+0000) 2024-02-19T20:20:55.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:55.342+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:55.343169+0000) 2024-02-19T20:20:55.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:55.725+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:55.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:55.725+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:55.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:55.725+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:55.988 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:55.726+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:56.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:56.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:56.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:56.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:56.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:56.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:56.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:56.036+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:56.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:56.041+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:56.042453+0000) 2024-02-19T20:20:56.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:56.058+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:56.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:56.058+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:56.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:56.058+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:56.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:56.059+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:56.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:56.259+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:56.260351+0000) 2024-02-19T20:20:56.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:56.344+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:56.345149+0000) 2024-02-19T20:20:57.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:56.732+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:57.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:56.732+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:57.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:56.732+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:57.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:56.733+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:57.347 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:57.043+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:57.044506+0000) 2024-02-19T20:20:57.348 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:57.095+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:57.348 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:57.095+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:57.349 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:57.095+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:57.349 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:57.097+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:57.349 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:57.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:57.350 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:57.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:57.350 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:57.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:57.350 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:57.073+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:57.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:57.262+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:57.263435+0000) 2024-02-19T20:20:57.612 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:57.346+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:57.347325+0000) 2024-02-19T20:20:58.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:57.685+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:58.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:57.685+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:58.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:57.685+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:58.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:57.685+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:58.350 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:58.046+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:58.047087+0000) 2024-02-19T20:20:58.350 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:58.131+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:58.350 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:58.131+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:58.350 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:58.131+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:58.350 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:58.132+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:58.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:58.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:58.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:58.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:58.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:58.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:58.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:58.091+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:58.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:58.264+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:58.265655+0000) 2024-02-19T20:20:58.612 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:58.349+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:58.350097+0000) 2024-02-19T20:20:59.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:58.730+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:59.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:58.730+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:59.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:58.730+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:59.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:58.730+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:59.350 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:59.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:59.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:59.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:59.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:59.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:20:59.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:59.117+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:20:59.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:59.265+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:59.266126+0000) 2024-02-19T20:20:59.351 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:59.046+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:59.047609+0000) 2024-02-19T20:20:59.351 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:59.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:59.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:59.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:59.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:59.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:20:59.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:59.092+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:20:59.612 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:59.349+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:59.350750+0000) 2024-02-19T20:20:59.862 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:59.734+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:59.862 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:59.734+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:59.863 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:59.734+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:20:59.863 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:59.735+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:00.141 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:20:59.864+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58349 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:21:00.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:20:59.862+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:59.863036+0000) 2024-02-19T20:21:00.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:20:59.861+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:59.862475+0000) 2024-02-19T20:21:00.143 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:00.140+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:00.143 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:00.140+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:00.143 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:00.140+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:00.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:20:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:20:59.863+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:59.864022+0000) 2024-02-19T20:21:00.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:00.046+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:00.047860+0000) 2024-02-19T20:21:00.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:00.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:00.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:00.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:00.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:00.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:00.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:00.049+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:00.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:00.141+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:00.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:00.265+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:00.266323+0000) 2024-02-19T20:21:00.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:00.350+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:00.350957+0000) 2024-02-19T20:21:01.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:00.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:01.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:00.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:01.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:00.771+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:01.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:00.771+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:01.353 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:01.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:01.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:01.037+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:01.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:01.037+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:01.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:01.038+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:01.354 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:01.049+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:01.050230+0000) 2024-02-19T20:21:01.354 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:01.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:01.354 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:01.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:01.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:01.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:01.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:01.129+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:01.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:01.267+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:01.268854+0000) 2024-02-19T20:21:01.614 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:01.352+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:01.353690+0000) 2024-02-19T20:21:02.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:01.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:02.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:01.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:02.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:01.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:02.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:01.732+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:02.286 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:02.032+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:02.286 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:02.032+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:02.286 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:02.032+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:02.286 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:02.033+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:02.287 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:02.051+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:02.052601+0000) 2024-02-19T20:21:02.287 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:02.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:02.287 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:02.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:02.287 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:02.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:02.287 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:02.117+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:02.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:02.270+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:02.271409+0000) 2024-02-19T20:21:02.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:02.356+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:02.357169+0000) 2024-02-19T20:21:03.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:02.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:03.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:02.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:03.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:02.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:03.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:02.727+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:03.360 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:03.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:03.360 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:03.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:03.360 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:03.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:03.360 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:03.054+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:03.360 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:03.055+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:03.056050+0000) 2024-02-19T20:21:03.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:03.120+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:03.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:03.120+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:03.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:03.120+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:03.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:03.121+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:03.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:03.272+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:03.273599+0000) 2024-02-19T20:21:03.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:03.358+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:03.359729+0000) 2024-02-19T20:21:04.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:03.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:04.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:03.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:04.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:03.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:04.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:03.756+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:04.360 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:04.055+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:04.056746+0000) 2024-02-19T20:21:04.360 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:04.071+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:04.360 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:04.071+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:04.361 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:04.071+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:04.361 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:04.071+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:04.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:04.107+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:04.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:04.107+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:04.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:04.107+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:04.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:04.108+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:04.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:04.273+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:04.274190+0000) 2024-02-19T20:21:04.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:04.359+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:04.360357+0000) 2024-02-19T20:21:04.866 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:04.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:04.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:04.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:04.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:04.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:04.867 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:04.724+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:05.120 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:21:04.871+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58390 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:21:05.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:04.868+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:04.869832+0000) 2024-02-19T20:21:05.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:04.869+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:04.870693+0000) 2024-02-19T20:21:05.121 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:04.868+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:04.869780+0000) 2024-02-19T20:21:05.121 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:05.055+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:05.056864+0000) 2024-02-19T20:21:05.121 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:05.063+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:05.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:05.063+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:05.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:05.063+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:05.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:05.065+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:05.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:05.359+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:05.360553+0000) 2024-02-19T20:21:05.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:05.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:05.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:05.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:05.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:05.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:05.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:05.120+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:05.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:05.273+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:05.274382+0000) 2024-02-19T20:21:06.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:05.713+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:06.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:05.713+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:06.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:05.713+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:06.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:05.713+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:06.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:06.046+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:06.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:06.046+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:06.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:06.046+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:06.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:06.047+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:06.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:06.058+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:06.059014+0000) 2024-02-19T20:21:06.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:06.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:06.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:06.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:06.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:06.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:06.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:06.084+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:06.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:06.275+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:06.276380+0000) 2024-02-19T20:21:06.619 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:06.362+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:06.363096+0000) 2024-02-19T20:21:07.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:06.713+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:07.065 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:06.713+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:07.066 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:06.713+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:07.066 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:06.713+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:07.363 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:07.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:07.364 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:07.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:07.364 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:07.117+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:07.364 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:07.118+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:07.364 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:07.277+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:07.278908+0000) 2024-02-19T20:21:07.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:07.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:07.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:07.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:07.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:07.051+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:07.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:07.052+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:07.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:07.061+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:07.062514+0000) 2024-02-19T20:21:07.619 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:07.365+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:07.366003+0000) 2024-02-19T20:21:08.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:07.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:08.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:07.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:08.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:07.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:08.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:07.761+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:08.282 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:08.135+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:08.282 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:08.135+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:08.282 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:08.135+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:08.282 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:08.136+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:08.283 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:08.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:08.283 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:08.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:08.283 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:08.029+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:08.283 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:08.029+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:08.283 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:08.064+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:08.065261+0000) 2024-02-19T20:21:08.620 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:08.281+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:08.282198+0000) 2024-02-19T20:21:08.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:08.368+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:08.369577+0000) 2024-02-19T20:21:09.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:08.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:09.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:08.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:09.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:08.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:09.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:08.722+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:09.364 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:09.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:09.364 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:09.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:09.364 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:09.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:09.364 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:09.173+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:09.364 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:09.281+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:09.282814+0000) 2024-02-19T20:21:09.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:09.055+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:09.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:09.055+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:09.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:09.055+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:09.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:09.056+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:09.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:09.065+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:09.065993+0000) 2024-02-19T20:21:09.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:09.369+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:09.370031+0000) 2024-02-19T20:21:09.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:09.725+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:09.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:09.725+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:09.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:09.725+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:09.874 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:09.725+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:10.143 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:21:09.879+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58430 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:21:10.144 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:09.875+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:09.876323+0000) 2024-02-19T20:21:10.144 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:09.877+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:09.878017+0000) 2024-02-19T20:21:10.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:09.875+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:09.876820+0000) 2024-02-19T20:21:10.144 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:10.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:10.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:10.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:10.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:10.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:10.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:10.041+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:10.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:10.065+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:10.066161+0000) 2024-02-19T20:21:10.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:10.369+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:10.370205+0000) 2024-02-19T20:21:10.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:10.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:10.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:10.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:10.164+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:10.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:10.282+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:10.282985+0000) 2024-02-19T20:21:11.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:10.705+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:11.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:10.705+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:11.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:10.705+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:11.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:10.705+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:11.349 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:11.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:11.350 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:11.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:11.350 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:11.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:11.350 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:11.150+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:11.351 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:11.285+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:11.286120+0000) 2024-02-19T20:21:11.351 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:11.067+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:11.068418+0000) 2024-02-19T20:21:11.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:11.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:11.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:11.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:11.352 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:11.081+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:11.353 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:11.081+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:11.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:11.371+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:11.372362+0000) 2024-02-19T20:21:12.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:11.712+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:12.026 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:11.712+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:12.026 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:11.712+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:12.026 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:11.713+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:12.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:12.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:12.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:12.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:12.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:12.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:12.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:12.048+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:12.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:12.069+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:12.070638+0000) 2024-02-19T20:21:12.366 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:12.190+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:12.366 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:12.190+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:12.366 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:12.190+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:12.366 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:12.191+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:12.366 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:12.287+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:12.288316+0000) 2024-02-19T20:21:12.623 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:12.374+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:12.375927+0000) 2024-02-19T20:21:13.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:12.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:13.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:12.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:13.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:12.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:13.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:12.762+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:13.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:13.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:13.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:13.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:13.007+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:13.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:13.072+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:13.073249+0000) 2024-02-19T20:21:13.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:13.180+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:13.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:13.180+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:13.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:13.180+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:13.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:13.180+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:13.588 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:13.290+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:13.291756+0000) 2024-02-19T20:21:13.588 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:13.377+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:13.378747+0000) 2024-02-19T20:21:14.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:13.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:14.014 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:13.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:14.014 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:13.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:14.014 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:13.804+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:14.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:14.173+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:14.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:14.173+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:14.173+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:14.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:14.173+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:14.012+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:14.012+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:14.012+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:14.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:14.013+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:14.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:14.072+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:14.073903+0000) 2024-02-19T20:21:14.623 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:14.291+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:14.292260+0000) 2024-02-19T20:21:14.623 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:14.378+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:14.379241+0000) 2024-02-19T20:21:14.881 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:14.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:14.881 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:14.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:14.881 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:14.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:14.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:14.801+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:15.140 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:21:14.886+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58471 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:21:15.140 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:14.883+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:14.884702+0000) 2024-02-19T20:21:15.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:14.883+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:14.884023+0000) 2024-02-19T20:21:15.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:15.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:15.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:15.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:15.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:15.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:15.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:14.883+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:14.884647+0000) 2024-02-19T20:21:15.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:15.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:15.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:15.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:15.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:15.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:15.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:15.035+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:15.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:15.073+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:15.074099+0000) 2024-02-19T20:21:15.519 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:15.378+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:15.379392+0000) 2024-02-19T20:21:15.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:15.140+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:15.520 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:15.291+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:15.292446+0000) 2024-02-19T20:21:16.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:15.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:16.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:15.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:16.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:15.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:16.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:15.839+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:16.287 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:16.160+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:16.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:16.160+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:16.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:16.160+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:16.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:16.161+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:16.288 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:16.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:16.288 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:16.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:16.288 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:16.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:16.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:16.019+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:16.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:16.075+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:16.076164+0000) 2024-02-19T20:21:16.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:16.294+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:16.295026+0000) 2024-02-19T20:21:16.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:16.380+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:16.381475+0000) 2024-02-19T20:21:17.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:16.804+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:17.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:16.804+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:16.804+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:16.805+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:17.287 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:17.136+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:17.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:17.136+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:17.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:17.136+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:17.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:17.137+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:17.288 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:17.023+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:17.288 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:17.023+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:17.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:17.023+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:17.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:17.024+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:17.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:17.078+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:17.079561+0000) 2024-02-19T20:21:17.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:17.296+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:17.297680+0000) 2024-02-19T20:21:17.628 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:17.383+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:17.384062+0000) 2024-02-19T20:21:18.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:17.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:18.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:17.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:18.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:17.799+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:18.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:17.799+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:18.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:18.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:18.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:18.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:18.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:18.066+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:18.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:18.067+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:18.365 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:18.080+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:18.081616+0000) 2024-02-19T20:21:18.366 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:18.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:18.366 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:18.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:18.366 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:18.091+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:18.366 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:18.091+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:18.366 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:18.298+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:18.299805+0000) 2024-02-19T20:21:18.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:18.385+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:18.386462+0000) 2024-02-19T20:21:19.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:18.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:19.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:18.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:19.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:18.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:19.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:18.813+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:19.300 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:19.121+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:19.300 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:19.121+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:19.300 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:19.121+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:19.301 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:19.122+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:19.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:19.037+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:19.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:19.037+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:19.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:19.037+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:19.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:19.038+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:19.301 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:19.081+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:19.082384+0000) 2024-02-19T20:21:19.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:19.299+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:19.300424+0000) 2024-02-19T20:21:19.628 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:19.386+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:19.387072+0000) 2024-02-19T20:21:19.888 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:19.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:19.889 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:19.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:19.889 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:19.825+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:19.889 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:19.826+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:20.144 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:21:19.893+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58511 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:21:20.147 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:19.890+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:19.891833+0000) 2024-02-19T20:21:20.147 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:20.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:20.147 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:20.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:20.147 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:20.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:20.148 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:20.135+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:20.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:19.890+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:19.891907+0000) 2024-02-19T20:21:20.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:20.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:20.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:20.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:20.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:20.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:20.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:20.008+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:20.149 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:20.081+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:20.082549+0000) 2024-02-19T20:21:20.149 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:19.891+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:19.892034+0000) 2024-02-19T20:21:20.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:20.299+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:20.300640+0000) 2024-02-19T20:21:20.520 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:20.386+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:20.387252+0000) 2024-02-19T20:21:21.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:20.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:21.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:20.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:21.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:20.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:21.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:20.792+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:21.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:21.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:21.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:21.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:21.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:21.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:21.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:21.097+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:21.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:21.030+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:21.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:21.030+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:21.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:21.030+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:21.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:21.030+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:21.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:21.084+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:21.085080+0000) 2024-02-19T20:21:21.632 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:21.302+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:21.303845+0000) 2024-02-19T20:21:21.632 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:21.388+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:21.389139+0000) 2024-02-19T20:21:22.000 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:21.811+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:22.000 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:21.811+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:22.000 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:21.811+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:22.000 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:21.812+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:22.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:22.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:22.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:22.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:22.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:22.063+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:21.999+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:21.999+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:21.999+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:21.999+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:22.086+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:22.087180+0000) 2024-02-19T20:21:22.632 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:22.305+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:22.306154+0000) 2024-02-19T20:21:22.633 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:22.391+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:22.392577+0000) 2024-02-19T20:21:22.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:22.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:22.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:22.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:22.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:22.858+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:22.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:22.859+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:23.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:22.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:23.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:22.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:22.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:22.970+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:23.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:23.088+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:23.089947+0000) 2024-02-19T20:21:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:23.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:23.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:23.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:23.068+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:23.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:23.069+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:23.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:23.308+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:23.309723+0000) 2024-02-19T20:21:23.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:23.394+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:23.395138+0000) 2024-02-19T20:21:24.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:23.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:24.093 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:23.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:24.093 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:23.933+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:24.093 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:23.934+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:24.093 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:24.089+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:24.090719+0000) 2024-02-19T20:21:24.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:23.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:24.094 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:23.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:24.094 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:23.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:24.094 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:23.841+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:24.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:24.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:24.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:24.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:24.366 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:24.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:24.366 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:24.097+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:24.366 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:24.309+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:24.310233+0000) 2024-02-19T20:21:24.633 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:24.394+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:24.395631+0000) 2024-02-19T20:21:24.896 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:24.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:24.896 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:24.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:24.896 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:24.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:24.896 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:24.875+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:25.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:21:24.901+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58552 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:21:25.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:24.897+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:24.898847+0000) 2024-02-19T20:21:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:24.897+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:24.898390+0000) 2024-02-19T20:21:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:25.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:25.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:25.058+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:25.059+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:25.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:24.897+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:24.898410+0000) 2024-02-19T20:21:25.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:24.921+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:25.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:24.921+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:25.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:24.921+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:25.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:24.922+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:25.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:25.089+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:25.090914+0000) 2024-02-19T20:21:25.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:25.309+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:25.310441+0000) 2024-02-19T20:21:25.633 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:25.394+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:25.395797+0000) 2024-02-19T20:21:25.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:25.836+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:25.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:25.836+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:25.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:25.836+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:25.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:25.837+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:26.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:26.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:26.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:26.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:26.019+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:26.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:26.019+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:26.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:25.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:26.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:25.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:26.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:25.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:26.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:25.901+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:26.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:26.092+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:26.092996+0000) 2024-02-19T20:21:26.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:26.312+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:26.313708+0000) 2024-02-19T20:21:26.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:26.397+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:26.398336+0000) 2024-02-19T20:21:26.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:26.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:26.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:26.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:26.887 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:26.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:26.887 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:26.852+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:27.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:26.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:27.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:26.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:26.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:27.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:27.000+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:26.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:26.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:26.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:27.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:26.886+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:27.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:27.094+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:27.095066+0000) 2024-02-19T20:21:27.636 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:27.316+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:27.317057+0000) 2024-02-19T20:21:27.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:27.400+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:27.401079+0000) 2024-02-19T20:21:27.903 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:27.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:27.903 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:27.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:27.903 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:27.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:27.903 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:27.808+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:28.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:27.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:28.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:27.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:28.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:27.902+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:28.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:27.903+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:28.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:28.097+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:28.098616+0000) 2024-02-19T20:21:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:28.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:28.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:28.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:28.028+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:28.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:28.028+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:28.636 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:28.319+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:28.320583+0000) 2024-02-19T20:21:28.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:28.402+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:28.403765+0000) 2024-02-19T20:21:29.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:28.866+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:29.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:28.866+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:29.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:28.866+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:29.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:28.866+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:28.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:28.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:28.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:28.810+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:29.288 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:29.098+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:29.099373+0000) 2024-02-19T20:21:29.289 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:29.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:29.289 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:29.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:29.289 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:29.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:29.289 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:29.023+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:29.636 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:29.320+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:29.321115+0000) 2024-02-19T20:21:29.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:29.403+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:29.404405+0000) 2024-02-19T20:21:29.904 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:29.839+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:29.904 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:29.839+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:29.904 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:29.839+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:29.904 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:29.839+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:29.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:29.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:29.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:29.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:29.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:29.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:29.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:29.848+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:30.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:21:29.909+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58592 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:21:30.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:29.906+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:29.907871+0000) 2024-02-19T20:21:30.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:29.905+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:29.906534+0000) 2024-02-19T20:21:30.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:30.098+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:30.099547+0000) 2024-02-19T20:21:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:29.906+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:29.907049+0000) 2024-02-19T20:21:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:30.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:30.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:30.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:30.007+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:30.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:30.320+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:30.321320+0000) 2024-02-19T20:21:30.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:30.403+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:30.404592+0000) 2024-02-19T20:21:31.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:30.882+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:31.019 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:30.882+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:31.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:30.882+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:31.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:30.883+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:31.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:30.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:31.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:30.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:31.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:30.819+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:31.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:30.819+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:31.289 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:31.101+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:31.102094+0000) 2024-02-19T20:21:31.289 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:31.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:31.289 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:31.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:31.289 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:31.035+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:31.290 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:31.036+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:31.639 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:31.322+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:31.323954+0000) 2024-02-19T20:21:31.640 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:31.406+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:31.407009+0000) 2024-02-19T20:21:31.914 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:31.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:31.914 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:31.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:31.914 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:31.856+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:31.914 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:31.856+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:32.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:32.075+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:32.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:32.075+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:32.075+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:32.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:32.075+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:31.913+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:31.913+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:31.913+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:32.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:31.913+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:32.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:32.103+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:32.104470+0000) 2024-02-19T20:21:32.640 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:32.326+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:32.327442+0000) 2024-02-19T20:21:32.640 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:32.408+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:32.409834+0000) 2024-02-19T20:21:32.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:32.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:32.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:32.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:32.934 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:32.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:32.934 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:32.882+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:33.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:32.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:33.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:32.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:33.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:32.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:33.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:32.933+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:33.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:33.105+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:33.106635+0000) 2024-02-19T20:21:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:33.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:33.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:33.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:33.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:33.050+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:33.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:33.050+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:33.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:33.328+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:33.329489+0000) 2024-02-19T20:21:33.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:33.411+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:33.412286+0000) 2024-02-19T20:21:34.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:33.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:33.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:33.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:33.981+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:34.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:34.106+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:34.107175+0000) 2024-02-19T20:21:34.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:34.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:34.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:34.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:34.079+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:34.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:34.079+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:34.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:33.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:34.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:33.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:34.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:33.860+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:34.271 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:33.861+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:34.640 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:34.329+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:34.330306+0000) 2024-02-19T20:21:34.640 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:34.411+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:34.412861+0000) 2024-02-19T20:21:34.911 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:34.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:34.911 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:34.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:34.911 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:34.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:34.911 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:34.849+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:35.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:21:34.916+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58633 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:21:35.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:34.914+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:34.915429+0000) 2024-02-19T20:21:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:34.913+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:34.914168+0000) 2024-02-19T20:21:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:35.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:35.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:35.090+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:35.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:35.091+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:35.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:34.914+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:34.915271+0000) 2024-02-19T20:21:35.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:35.027+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:35.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:35.027+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:35.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:35.027+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:35.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:35.028+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:35.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:35.106+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:35.107357+0000) 2024-02-19T20:21:35.640 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:35.329+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:35.330501+0000) 2024-02-19T20:21:35.640 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:35.412+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:35.413021+0000) 2024-02-19T20:21:35.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:35.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:35.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:35.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:35.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:35.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:35.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:35.801+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:36.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:36.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:36.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:36.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:36.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:36.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:36.065+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:36.008+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:36.008+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:36.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:36.008+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:36.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:36.009+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:36.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:36.108+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:36.109766+0000) 2024-02-19T20:21:36.643 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:36.332+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:36.333678+0000) 2024-02-19T20:21:36.644 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:36.414+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:36.415024+0000) 2024-02-19T20:21:36.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:36.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:36.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:36.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:36.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:36.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:36.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:36.759+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:37.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:37.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:37.281 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:37.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:37.281 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:37.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:37.281 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:37.094+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:37.282 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:36.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:37.282 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:36.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:37.282 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:36.978+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:37.282 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:36.979+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:37.282 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:37.110+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:37.111763+0000) 2024-02-19T20:21:37.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:37.335+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:37.336182+0000) 2024-02-19T20:21:37.645 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:37.416+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:37.417067+0000) 2024-02-19T20:21:37.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:37.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:37.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:37.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:37.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:37.809+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:37.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:37.809+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:38.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:38.126+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:38.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:38.126+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:38.126+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:38.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:38.128+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:37.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:37.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:37.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:37.972+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:38.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:38.113+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:38.114214+0000) 2024-02-19T20:21:38.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:38.336+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:38.338155+0000) 2024-02-19T20:21:38.645 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:38.418+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:38.420731+0000) 2024-02-19T20:21:39.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:38.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:39.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:38.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:39.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:38.838+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:39.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:38.838+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:39.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:39.120+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:39.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:39.120+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:39.120+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:39.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:39.121+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:39.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:39.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:39.005+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:39.006+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:39.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:39.114+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:39.115003+0000) 2024-02-19T20:21:39.645 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:39.337+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:39.338688+0000) 2024-02-19T20:21:39.645 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:39.420+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:39.421413+0000) 2024-02-19T20:21:39.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:39.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:39.919 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:39.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:39.919 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:39.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:39.919 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:39.820+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:40.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:21:39.924+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58673 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:21:40.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:39.921+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:39.922182+0000) 2024-02-19T20:21:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:39.920+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:39.921293+0000) 2024-02-19T20:21:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:40.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:40.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:40.076+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:40.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:40.077+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:39.921+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:39.922409+0000) 2024-02-19T20:21:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:40.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:40.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:40.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:40.050+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:40.114+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:40.115163+0000) 2024-02-19T20:21:40.645 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:40.337+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:40.338869+0000) 2024-02-19T20:21:40.645 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:40.420+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:40.421632+0000) 2024-02-19T20:21:41.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:40.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:41.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:40.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:41.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:40.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:41.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:40.823+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:41.341 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:41.107+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:41.341 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:41.107+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:41.341 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:41.107+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:41.341 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:41.108+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:41.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:41.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:41.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:41.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:41.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:41.077+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:41.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:41.078+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:41.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:41.116+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:41.117499+0000) 2024-02-19T20:21:41.647 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:41.340+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:41.341301+0000) 2024-02-19T20:21:41.648 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:41.422+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:41.423784+0000) 2024-02-19T20:21:41.931 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:41.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:41.931 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:41.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:41.931 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:41.835+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:41.932 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:41.835+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:42.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:42.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:42.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:42.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:42.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:42.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:42.156+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:42.076+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:42.076+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:42.076+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:42.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:42.077+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:42.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:42.118+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:42.119942+0000) 2024-02-19T20:21:42.649 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:42.342+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:42.343558+0000) 2024-02-19T20:21:42.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:42.426+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:42.427393+0000) 2024-02-19T20:21:43.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:42.869+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:43.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:42.869+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:43.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:42.869+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:43.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:42.869+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:43.345 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:43.153+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:43.346 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:43.153+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:43.346 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:43.153+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:43.346 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:43.154+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:43.346 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:43.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:43.346 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:43.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:43.346 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:43.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:43.347 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:43.122+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:43.347 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:43.122+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:43.123501+0000) 2024-02-19T20:21:43.648 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:43.344+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:43.345696+0000) 2024-02-19T20:21:43.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:43.428+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:43.429799+0000) 2024-02-19T20:21:44.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:43.849+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:44.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:43.849+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:44.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:43.849+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:44.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:43.849+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:44.346 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:44.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:44.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:44.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:44.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:44.106+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:44.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:44.106+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:44.347 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:44.123+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:44.124220+0000) 2024-02-19T20:21:44.347 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:44.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:44.348 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:44.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:44.348 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:44.125+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:44.348 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:44.125+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:44.648 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:44.345+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:44.346383+0000) 2024-02-19T20:21:44.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:44.429+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:44.430466+0000) 2024-02-19T20:21:44.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:44.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:44.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:44.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:44.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:44.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:44.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:44.824+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:45.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:21:44.931+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58714 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:21:45.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:44.929+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:44.930553+0000) 2024-02-19T20:21:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:44.930+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:44.931190+0000) 2024-02-19T20:21:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:45.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:45.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:45.091+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:45.092+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:45.123+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:45.124308+0000) 2024-02-19T20:21:45.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:44.929+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:44.930416+0000) 2024-02-19T20:21:45.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:45.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:45.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:45.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:45.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:45.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:45.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:45.096+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:45.649 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:45.345+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:45.346610+0000) 2024-02-19T20:21:45.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:45.429+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:45.430658+0000) 2024-02-19T20:21:46.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:45.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:45.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:45.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:46.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:45.866+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:46.349 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:46.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:46.349 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:46.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:46.349 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:46.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:46.349 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:46.058+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:46.350 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:46.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:46.350 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:46.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:46.350 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:46.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:46.350 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:46.089+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:46.350 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:46.125+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:46.126385+0000) 2024-02-19T20:21:46.651 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:46.348+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:46.349106+0000) 2024-02-19T20:21:46.651 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:46.432+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:46.433294+0000) 2024-02-19T20:21:46.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:46.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:46.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:46.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:46.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:46.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:46.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:46.824+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:47.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:47.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:47.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:47.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:47.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:47.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:47.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:47.049+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:47.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:47.128+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:47.128988+0000) 2024-02-19T20:21:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:47.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:47.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:47.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:47.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:47.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:47.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:47.039+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:47.652 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:47.350+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:47.351649+0000) 2024-02-19T20:21:47.652 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:47.436+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:47.437129+0000) 2024-02-19T20:21:47.996 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:47.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:47.996 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:47.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:47.996 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:47.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:47.997 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:47.846+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:48.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:47.995+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:47.995+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:47.995+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:48.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:47.996+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:48.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:48.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:48.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:48.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:48.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:48.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:48.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:48.061+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:48.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:48.130+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:48.131405+0000) 2024-02-19T20:21:48.652 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:48.352+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:48.353952+0000) 2024-02-19T20:21:48.652 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:48.438+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:48.439641+0000) 2024-02-19T20:21:48.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:48.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:48.975 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:48.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:48.975 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:48.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:48.975 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:48.867+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:49.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:48.973+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:49.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:48.974+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:49.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:48.974+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:49.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:48.974+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:49.107+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:49.107+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:49.107+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:49.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:49.108+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:49.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:49.131+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:49.132041+0000) 2024-02-19T20:21:49.652 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:49.353+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:49.354726+0000) 2024-02-19T20:21:49.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:49.439+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:49.440216+0000) 2024-02-19T20:21:49.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:49.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:49.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:49.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:49.934 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:49.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:49.934 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:49.899+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:50.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:21:49.938+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58754 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:21:50.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:49.936+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:49.937176+0000) 2024-02-19T20:21:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:49.937+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:49.938718+0000) 2024-02-19T20:21:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:50.131+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:50.132210+0000) 2024-02-19T20:21:50.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:50.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:50.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:50.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:50.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:50.157+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:50.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:50.158+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:50.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:49.935+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:49.936109+0000) 2024-02-19T20:21:50.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:49.997+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:50.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:49.997+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:50.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:49.997+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:50.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:49.997+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:50.652 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:50.353+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:50.354912+0000) 2024-02-19T20:21:50.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:50.439+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:50.440391+0000) 2024-02-19T20:21:50.951 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:50.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:50.951 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:50.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:50.952 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:50.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:50.952 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:50.880+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:51.269 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:51.133+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:51.134133+0000) 2024-02-19T20:21:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:51.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:51.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:51.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:51.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:51.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:51.163+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:50.950+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:51.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:50.950+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:51.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:50.950+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:51.272 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:50.951+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:51.655 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:51.356+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:51.357402+0000) 2024-02-19T20:21:51.655 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:51.442+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:51.443970+0000) 2024-02-19T20:21:51.993 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:51.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:51.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:51.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:51.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:51.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:51.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:51.873+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:52.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:51.992+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:51.992+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:51.992+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:52.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:51.993+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:52.136+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:52.137661+0000) 2024-02-19T20:21:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:52.167+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:52.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:52.167+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:52.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:52.167+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:52.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:52.168+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:52.656 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:52.358+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:52.359400+0000) 2024-02-19T20:21:52.657 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:52.445+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:52.446012+0000) 2024-02-19T20:21:53.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:52.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:53.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:52.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:53.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:52.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:53.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:52.890+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:53.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:53.011+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:53.011+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:53.011+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:53.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:53.012+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:53.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:53.139+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:53.140304+0000) 2024-02-19T20:21:53.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:53.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:53.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:53.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:53.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:53.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:53.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:53.201+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:53.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:53.360+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:53.361824+0000) 2024-02-19T20:21:53.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:53.448+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:53.449561+0000) 2024-02-19T20:21:54.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:53.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:54.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:53.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:54.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:53.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:54.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:53.992+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:54.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:53.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:54.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:53.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:54.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:53.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:54.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:53.874+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:54.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:54.361+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:54.362587+0000) 2024-02-19T20:21:54.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:54.139+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:54.140870+0000) 2024-02-19T20:21:54.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:54.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:54.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:54.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:54.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:54.161+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:54.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:54.162+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:54.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:54.449+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:54.450449+0000) 2024-02-19T20:21:55.140 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:21:54.946+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58795 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:21:55.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:54.943+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:54.944130+0000) 2024-02-19T20:21:55.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:54.943+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:54.944567+0000) 2024-02-19T20:21:55.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:54.973+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:55.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:54.973+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:55.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:54.973+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:55.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:54.974+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:55.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:54.887+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:55.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:54.887+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:55.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:54.887+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:55.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:54.888+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:55.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:54.942+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:54.943554+0000) 2024-02-19T20:21:55.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:55.361+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:55.362787+0000) 2024-02-19T20:21:55.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:55.140+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:55.141018+0000) 2024-02-19T20:21:55.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:55.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:55.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:55.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:55.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:55.172+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:55.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:55.173+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:55.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:55.449+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:55.450636+0000) 2024-02-19T20:21:56.144 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:55.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:56.144 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:55.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:56.144 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:55.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:56.145 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:55.983+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:56.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:55.853+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:56.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:55.853+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:56.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:55.853+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:56.145 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:55.854+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:56.453 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:56.363+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:56.364747+0000) 2024-02-19T20:21:56.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:56.143+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:56.144277+0000) 2024-02-19T20:21:56.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:56.171+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:56.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:56.171+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:56.454 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:56.171+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:56.454 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:56.172+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:56.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:56.452+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:56.453260+0000) 2024-02-19T20:21:57.092 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:56.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:57.093 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:56.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:57.093 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:56.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:57.093 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:56.936+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:57.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:56.826+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:57.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:56.826+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:57.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:56.826+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:57.094 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:56.827+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:57.367 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:57.146+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:57.147630+0000) 2024-02-19T20:21:57.367 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:57.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:57.367 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:57.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:57.367 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:57.208+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:57.367 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:57.209+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:57.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:57.365+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:57.366907+0000) 2024-02-19T20:21:57.659 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:57.454+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:57.455316+0000) 2024-02-19T20:21:57.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:57.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:57.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:57.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:57.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:57.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:57.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:57.810+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:58.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:57.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:58.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:57.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:57.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:58.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:57.957+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:58.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:58.149+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:58.150809+0000) 2024-02-19T20:21:58.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:58.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:58.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:58.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:58.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:58.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:58.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:58.163+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:58.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:58.367+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:58.368910+0000) 2024-02-19T20:21:58.659 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:58.457+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:58.458672+0000) 2024-02-19T20:21:58.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:58.783+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:58.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:58.783+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:58.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:58.783+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:58.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:58.783+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:59.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:58.956+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:59.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:58.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:59.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:58.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:21:59.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:58.957+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:21:59.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:59.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:59.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:59.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:59.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:59.133+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:21:59.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:59.134+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:21:59.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:59.150+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:59.151602+0000) 2024-02-19T20:21:59.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:59.368+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:59.369530+0000) 2024-02-19T20:21:59.659 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:59.458+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:59.459385+0000) 2024-02-19T20:21:59.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:59.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:59.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:59.766+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:59.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:59.766+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:21:59.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:59.766+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:00.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:21:59.953+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58834 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:22:00.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:21:59.950+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:59.951611+0000) 2024-02-19T20:22:00.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:21:59.951+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:59.952447+0000) 2024-02-19T20:22:00.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:00.090+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:00.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:00.090+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:00.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:00.090+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:00.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:00.091+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:00.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:00.150+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:00.151816+0000) 2024-02-19T20:22:00.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:59.951+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:59.952535+0000) 2024-02-19T20:22:00.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:59.996+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:00.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:59.996+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:00.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:59.996+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:00.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:21:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:21:59.996+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:00.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:00.368+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:00.369724+0000) 2024-02-19T20:22:00.660 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:00.458+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:00.459559+0000) 2024-02-19T20:22:01.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:00.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:01.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:00.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:01.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:00.801+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:01.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:00.802+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:01.292 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:01.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:01.292 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:01.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:01.292 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:01.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:01.292 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:01.033+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:01.293 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:01.089+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:01.293 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:01.089+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:01.293 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:01.089+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:01.293 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:01.090+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:01.293 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:01.154+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:01.155328+0000) 2024-02-19T20:22:01.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:01.370+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:01.371835+0000) 2024-02-19T20:22:01.662 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:01.460+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:01.461603+0000) 2024-02-19T20:22:02.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:01.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:02.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:01.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:02.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:01.848+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:02.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:01.848+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:02.370 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:02.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:02.370 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:02.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:02.370 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:02.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:02.370 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:02.074+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:02.370 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:02.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:02.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:02.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:02.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:02.049+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:02.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:02.050+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:02.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:02.156+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:02.157289+0000) 2024-02-19T20:22:02.662 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:02.372+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:02.373943+0000) 2024-02-19T20:22:02.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:02.463+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:02.464914+0000) 2024-02-19T20:22:03.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:02.869+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:03.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:02.869+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:03.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:02.869+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:03.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:02.870+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:03.370 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:03.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:03.370 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:03.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:03.370 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:03.119+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:03.370 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:03.120+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:03.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:03.069+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:03.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:03.069+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:03.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:03.069+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:03.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:03.070+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:03.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:03.158+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:03.159024+0000) 2024-02-19T20:22:03.662 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:03.375+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:03.376565+0000) 2024-02-19T20:22:03.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:03.466+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:03.467026+0000) 2024-02-19T20:22:04.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:03.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:04.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:03.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:04.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:03.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:04.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:03.873+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:04.370 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:04.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:04.370 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:04.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:04.370 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:04.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:04.370 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:04.150+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:04.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:04.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:04.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:04.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:04.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:04.109+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:04.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:04.110+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:04.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:04.158+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:04.159943+0000) 2024-02-19T20:22:04.663 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:04.376+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:04.377162+0000) 2024-02-19T20:22:04.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:04.466+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:04.467613+0000) 2024-02-19T20:22:04.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:04.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:04.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:04.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:04.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:04.891+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:04.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:04.892+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:05.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:22:04.961+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58875 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:22:05.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:04.957+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:04.958955+0000) 2024-02-19T20:22:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:04.957+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:04.958973+0000) 2024-02-19T20:22:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:05.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:05.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:05.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:05.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:05.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:05.173+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:04.958+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:04.959460+0000) 2024-02-19T20:22:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:05.064+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:05.064+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:05.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:05.064+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:05.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:05.065+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:05.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:05.159+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:05.160105+0000) 2024-02-19T20:22:05.663 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:05.376+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:05.377331+0000) 2024-02-19T20:22:05.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:05.466+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:05.467798+0000) 2024-02-19T20:22:06.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:05.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:06.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:05.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:06.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:05.883+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:06.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:05.883+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:06.370 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:06.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:06.370 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:06.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:06.370 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:06.061+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:06.370 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:06.061+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:06.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:06.161+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:06.162270+0000) 2024-02-19T20:22:06.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:06.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:06.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:06.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:06.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:06.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:06.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:06.188+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:06.643 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:06.378+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:06.379934+0000) 2024-02-19T20:22:06.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:06.468+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:06.469839+0000) 2024-02-19T20:22:07.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:07.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:07.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:07.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:07.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:07.152+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:07.208 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:07.152+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:07.209 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:07.098+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:07.209 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:07.098+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:07.209 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:07.098+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:07.209 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:07.099+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:07.209 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:07.163+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:07.164187+0000) 2024-02-19T20:22:07.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:06.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:07.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:06.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:07.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:06.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:07.210 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:06.926+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:07.471 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:07.381+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:07.382404+0000) 2024-02-19T20:22:07.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:07.470+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:07.471919+0000) 2024-02-19T20:22:08.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:08.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:08.411 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:08.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:08.411 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:08.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:08.411 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:08.138+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:08.412 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:08.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:08.412 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:08.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:08.412 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:08.117+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:08.412 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:08.118+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:08.412 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:08.165+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:08.166177+0000) 2024-02-19T20:22:08.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:07.967+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:08.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:07.967+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:08.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:07.967+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:08.413 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:07.968+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:08.666 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:08.383+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:08.384843+0000) 2024-02-19T20:22:08.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:08.474+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:08.475544+0000) 2024-02-19T20:22:09.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:09.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:09.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:09.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:09.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:09.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:09.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:09.131+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:08.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:08.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:08.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:09.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:08.928+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:09.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:09.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:09.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:09.089+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:09.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:09.165+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:09.166777+0000) 2024-02-19T20:22:09.666 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:09.384+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:09.385701+0000) 2024-02-19T20:22:09.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:09.475+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:09.476158+0000) 2024-02-19T20:22:09.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:09.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:09.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:09.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:09.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:09.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:09.963 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:09.895+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:10.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:22:09.968+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58915 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:22:10.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:09.965+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:09.966766+0000) 2024-02-19T20:22:10.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:09.964+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:09.965592+0000) 2024-02-19T20:22:10.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:10.120+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:10.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:10.120+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:10.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:10.120+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:10.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:10.121+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:10.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:10.166+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:10.166988+0000) 2024-02-19T20:22:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:09.965+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:09.966400+0000) 2024-02-19T20:22:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:10.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:10.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:10.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:10.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:10.159+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:10.666 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:10.384+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:10.385863+0000) 2024-02-19T20:22:10.667 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:10.475+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:10.476351+0000) 2024-02-19T20:22:11.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:10.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:11.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:10.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:11.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:10.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:11.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:10.905+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:11.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:11.193+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:11.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:11.193+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:11.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:11.193+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:11.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:11.193+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:11.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:11.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:11.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:11.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:11.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:11.075+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:11.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:11.076+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:11.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:11.168+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:11.169083+0000) 2024-02-19T20:22:11.669 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:11.388+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:11.389083+0000) 2024-02-19T20:22:11.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:11.477+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:11.478363+0000) 2024-02-19T20:22:12.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:11.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:12.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:11.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:12.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:11.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:12.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:11.861+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:12.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:12.159+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:12.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:12.159+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:12.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:12.159+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:12.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:12.159+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:12.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:12.108+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:12.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:12.108+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:12.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:12.108+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:12.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:12.108+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:12.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:12.171+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:12.172626+0000) 2024-02-19T20:22:12.670 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:12.390+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:12.391621+0000) 2024-02-19T20:22:12.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:12.479+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:12.480470+0000) 2024-02-19T20:22:13.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:12.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:13.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:12.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:13.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:12.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:13.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:12.907+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:13.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:13.114+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:13.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:13.114+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:13.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:13.114+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:13.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:13.115+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:13.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:13.124+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:13.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:13.124+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:13.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:13.124+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:13.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:13.125+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:13.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:13.173+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:13.174637+0000) 2024-02-19T20:22:13.670 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:13.393+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:13.394824+0000) 2024-02-19T20:22:13.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:13.481+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:13.482801+0000) 2024-02-19T20:22:14.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:13.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:14.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:13.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:14.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:13.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:14.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:13.907+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:14.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:14.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:14.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:14.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:14.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:14.103+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:14.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:14.104+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:14.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:14.116+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:14.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:14.116+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:14.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:14.116+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:14.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:14.116+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:14.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:14.174+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:14.175392+0000) 2024-02-19T20:22:14.670 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:14.394+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:14.395468+0000) 2024-02-19T20:22:14.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:14.482+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:14.483731+0000) 2024-02-19T20:22:14.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:14.884+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:14.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:14.884+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:14.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:14.884+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:14.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:14.885+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:15.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:14.972+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:14.973656+0000) 2024-02-19T20:22:15.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:22:14.976+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58956 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:22:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:14.972+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:14.973576+0000) 2024-02-19T20:22:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:15.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:15.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:15.065+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:15.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:15.066+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:15.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:14.972+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:14.973330+0000) 2024-02-19T20:22:15.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:15.084+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:15.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:15.084+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:15.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:15.084+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:15.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:15.084+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:15.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:15.174+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:15.175551+0000) 2024-02-19T20:22:15.670 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:15.394+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:15.395676+0000) 2024-02-19T20:22:15.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:15.482+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:15.483907+0000) 2024-02-19T20:22:16.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:15.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:16.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:15.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:16.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:15.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:16.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:15.839+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:16.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:16.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:16.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:16.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:16.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:16.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:16.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:16.052+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:16.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:16.050+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:16.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:16.050+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:16.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:16.050+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:16.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:16.051+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:16.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:16.176+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:16.177912+0000) 2024-02-19T20:22:16.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:16.396+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:16.397709+0000) 2024-02-19T20:22:16.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:16.485+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:16.485974+0000) 2024-02-19T20:22:17.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:16.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:17.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:16.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:16.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:17.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:16.823+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:17.294 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:17.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:17.294 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:17.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:17.294 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:17.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:17.294 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:17.057+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:17.294 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:17.039+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:17.295 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:17.039+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:17.295 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:17.039+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:17.295 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:17.040+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:17.295 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:17.180+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:17.181439+0000) 2024-02-19T20:22:17.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:17.399+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:17.400361+0000) 2024-02-19T20:22:17.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:17.487+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:17.488518+0000) 2024-02-19T20:22:18.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:17.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:18.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:17.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:18.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:17.839+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:18.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:17.839+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:18.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:18.099+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:18.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:18.099+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:18.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:18.099+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:18.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:18.099+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:18.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:18.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:18.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:18.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:18.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:18.053+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:18.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:18.054+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:18.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:18.183+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:18.183974+0000) 2024-02-19T20:22:18.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:18.403+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:18.404011+0000) 2024-02-19T20:22:18.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:18.489+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:18.490580+0000) 2024-02-19T20:22:19.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:18.827+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:19.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:18.827+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:19.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:18.827+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:19.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:18.827+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:19.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:19.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:19.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:19.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:19.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:19.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:19.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:19.071+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:19.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:19.092+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:19.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:19.092+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:19.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:19.092+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:19.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:19.093+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:19.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:19.183+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:19.184805+0000) 2024-02-19T20:22:19.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:19.403+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:19.404659+0000) 2024-02-19T20:22:19.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:19.490+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:19.491484+0000) 2024-02-19T20:22:19.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:19.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:19.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:19.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:19.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:19.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:19.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:19.804+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:20.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:22:19.984+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 58996 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:22:20.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:19.980+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:19.981158+0000) 2024-02-19T20:22:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:19.981+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:19.982043+0000) 2024-02-19T20:22:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:20.104+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:20.104+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:20.104+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:20.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:20.105+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:20.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:19.982+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:19.983918+0000) 2024-02-19T20:22:20.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:20.100+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:20.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:20.100+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:20.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:20.100+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:20.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:20.101+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:20.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:20.184+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:20.185025+0000) 2024-02-19T20:22:20.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:20.403+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:20.404909+0000) 2024-02-19T20:22:20.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:20.490+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:20.491690+0000) 2024-02-19T20:22:21.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:20.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:21.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:20.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:21.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:20.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:21.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:20.776+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:21.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:21.108+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:21.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:21.108+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:21.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:21.108+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:21.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:21.108+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:21.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:21.186+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:21.187146+0000) 2024-02-19T20:22:21.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:21.137+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:21.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:21.137+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:21.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:21.137+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:21.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:21.138+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:21.676 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:21.406+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:21.407023+0000) 2024-02-19T20:22:21.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:21.492+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:21.493693+0000) 2024-02-19T20:22:22.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:21.773+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:22.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:21.773+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:22.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:21.773+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:22.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:21.774+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:22.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:22.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:22.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:22.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:22.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:22.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:22.151+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:22.105+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:22.105+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:22.105+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:22.106+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:22.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:22.189+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:22.190669+0000) 2024-02-19T20:22:22.676 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:22.409+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:22.410522+0000) 2024-02-19T20:22:22.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:22.496+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:22.497028+0000) 2024-02-19T20:22:23.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:22.785+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:23.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:22.785+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:23.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:22.785+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:23.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:22.785+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:23.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:23.167+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:23.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:23.167+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:23.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:23.167+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:23.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:23.168+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:23.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:23.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:23.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:23.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:23.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:23.121+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:23.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:23.121+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:23.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:23.191+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:23.192829+0000) 2024-02-19T20:22:23.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:23.411+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:23.412631+0000) 2024-02-19T20:22:23.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:23.498+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:23.499124+0000) 2024-02-19T20:22:24.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:23.777+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:24.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:23.777+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:24.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:23.777+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:24.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:23.777+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:24.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:24.193+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:24.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:24.193+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:24.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:24.193+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:24.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:24.193+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:24.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:24.080+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:24.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:24.080+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:24.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:24.080+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:24.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:24.081+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:24.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:24.192+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:24.193720+0000) 2024-02-19T20:22:24.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:24.412+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:24.413349+0000) 2024-02-19T20:22:24.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:24.498+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:24.499794+0000) 2024-02-19T20:22:24.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:24.770+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:24.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:24.770+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:24.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:24.770+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:24.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:24.770+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:25.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:24.988+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:24.989951+0000) 2024-02-19T20:22:25.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:22:24.992+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59037 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:22:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:24.988+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:24.989657+0000) 2024-02-19T20:22:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:25.201+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:25.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:25.201+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:25.201+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:25.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:25.202+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:25.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:24.988+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:24.989201+0000) 2024-02-19T20:22:25.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:25.099+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:25.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:25.099+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:25.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:25.099+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:25.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:25.100+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:25.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:25.192+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:25.193913+0000) 2024-02-19T20:22:25.677 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:25.412+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:25.413521+0000) 2024-02-19T20:22:25.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:25.498+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:25.499961+0000) 2024-02-19T20:22:26.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:25.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:26.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:25.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:26.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:25.808+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:26.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:25.808+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:26.294 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:26.074+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:26.295 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:26.074+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:26.295 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:26.074+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:26.295 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:26.074+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:26.295 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:26.194+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:26.196010+0000) 2024-02-19T20:22:26.296 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:26.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:26.296 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:26.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:26.296 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:26.178+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:26.297 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:26.179+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:26.679 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:26.414+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:26.416821+0000) 2024-02-19T20:22:26.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:26.502+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:26.503501+0000) 2024-02-19T20:22:27.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:26.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:27.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:26.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:27.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:26.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:27.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:26.822+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:27.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:27.104+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:27.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:27.104+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:27.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:27.104+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:27.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:27.105+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:27.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:27.197+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:27.198412+0000) 2024-02-19T20:22:27.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:27.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:27.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:27.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:27.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:27.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:27.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:27.177+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:27.681 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:27.418+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:27.419303+0000) 2024-02-19T20:22:27.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:27.504+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:27.505720+0000) 2024-02-19T20:22:28.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:27.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:28.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:27.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:28.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:27.807+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:28.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:27.808+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:28.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:28.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:28.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:28.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:28.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:28.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:28.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:28.169+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:28.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:28.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:28.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:28.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:28.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:28.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:28.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:28.145+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:28.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:28.199+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:28.200448+0000) 2024-02-19T20:22:28.681 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:28.420+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:28.421513+0000) 2024-02-19T20:22:28.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:28.508+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:28.509357+0000) 2024-02-19T20:22:29.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:28.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:29.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:28.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:28.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:29.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:28.822+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:29.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:29.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:29.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:29.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:29.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:29.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:29.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:29.170+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:29.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:29.138+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:29.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:29.138+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:29.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:29.138+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:29.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:29.139+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:29.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:29.200+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:29.201102+0000) 2024-02-19T20:22:29.681 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:29.421+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:29.422165+0000) 2024-02-19T20:22:29.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:29.509+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:29.510015+0000) 2024-02-19T20:22:29.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:29.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:29.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:29.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:29.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:29.776+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:29.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:29.776+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:30.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:29.996+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:29.997357+0000) 2024-02-19T20:22:30.270 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:22:29.999+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59077 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:22:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:29.995+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:29.996837+0000) 2024-02-19T20:22:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:30.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:30.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:30.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:30.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:30.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:30.174+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:30.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:29.996+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:29.996987+0000) 2024-02-19T20:22:30.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:30.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:30.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:30.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:30.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:30.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:30.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:30.171+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:30.272 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:30.200+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:30.201348+0000) 2024-02-19T20:22:30.681 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:30.421+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:30.422341+0000) 2024-02-19T20:22:30.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:30.509+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:30.510183+0000) 2024-02-19T20:22:31.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:30.740+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:31.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:30.740+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:31.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:30.740+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:31.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:30.740+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:31.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:31.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:31.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:31.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:31.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:31.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:31.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:31.169+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:31.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:31.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:31.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:31.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:31.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:31.191+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:31.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:31.192+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:31.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:31.203+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:31.204926+0000) 2024-02-19T20:22:31.684 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:31.423+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:31.424847+0000) 2024-02-19T20:22:31.684 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:31.511+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:31.512888+0000) 2024-02-19T20:22:32.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:31.753+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:32.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:31.753+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:31.753+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:32.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:31.754+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:32.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:32.182+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:32.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:32.182+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:32.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:32.182+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:32.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:32.183+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:32.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:32.207+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:32.208416+0000) 2024-02-19T20:22:32.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:32.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:32.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:32.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:32.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:32.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:32.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:32.164+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:32.685 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:32.426+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:32.426997+0000) 2024-02-19T20:22:32.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:32.514+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:32.514988+0000) 2024-02-19T20:22:33.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:32.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:33.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:32.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:33.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:32.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:33.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:32.796+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:33.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:33.145+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:33.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:33.145+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:33.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:33.145+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:33.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:33.146+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:33.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:33.209+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:33.210604+0000) 2024-02-19T20:22:33.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:33.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:33.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:33.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:33.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:33.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:33.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:33.211+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:33.685 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:33.428+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:33.429447+0000) 2024-02-19T20:22:33.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:33.516+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:33.517684+0000) 2024-02-19T20:22:34.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:33.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:34.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:33.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:34.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:33.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:34.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:33.757+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:34.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:34.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:34.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:34.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:34.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:34.164+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:34.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:34.165+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:34.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:34.210+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:34.211216+0000) 2024-02-19T20:22:34.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:34.220+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:34.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:34.220+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:34.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:34.220+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:34.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:34.221+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:34.685 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:34.429+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:34.430236+0000) 2024-02-19T20:22:34.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:34.517+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:34.518806+0000) 2024-02-19T20:22:35.000 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:34.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:35.001 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:34.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:35.001 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:34.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:11:56.549740+0000 (oldest deadline 2024-02-19T20:12:16.549740+0000) 2024-02-19T20:22:35.001 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:34.736+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:35.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:22:35.006+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59118 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:22:35.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:35.003+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:35.004571+0000) 2024-02-19T20:22:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:35.004+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:35.005212+0000) 2024-02-19T20:22:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:35.185+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:35.185+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:35.185+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:35.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:35.186+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:35.003+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:35.004108+0000) 2024-02-19T20:22:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:35.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:35.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:35.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:35.204+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:35.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:35.210+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:35.211366+0000) 2024-02-19T20:22:35.686 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:35.429+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:35.430426+0000) 2024-02-19T20:22:35.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:35.518+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:35.519013+0000) 2024-02-19T20:22:36.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:35.692+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:36.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:36.143+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:36.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:36.143+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:36.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:36.143+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:36.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:36.144+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:36.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:36.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:36.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:36.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:36.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:36.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:36.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:36.203+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:36.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:36.212+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:36.213517+0000) 2024-02-19T20:22:36.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:36.431+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:36.432617+0000) 2024-02-19T20:22:36.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:36.520+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:36.521119+0000) 2024-02-19T20:22:36.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:36.673+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:37.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:37.107+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:37.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:37.107+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:37.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:37.107+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:37.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:37.108+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:37.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:37.214+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:37.215726+0000) 2024-02-19T20:22:37.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:37.238+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:37.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:37.238+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:37.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:37.238+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:37.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:37.238+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:37.686 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:37.433+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:37.434777+0000) 2024-02-19T20:22:37.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:37.522+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:37.523308+0000) 2024-02-19T20:22:37.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:37.685+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:38.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:38.217+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:38.218322+0000) 2024-02-19T20:22:38.528 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:38.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:38.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:38.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:38.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:38.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:38.529 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:38.238+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:38.529 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:38.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:38.529 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:38.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:38.530 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:38.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:02.881519+0000 (oldest deadline 2024-02-19T20:12:22.881519+0000) 2024-02-19T20:22:38.530 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:38.064+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:38.689 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:38.437+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:38.438305+0000) 2024-02-19T20:22:38.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:38.525+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:38.526915+0000) 2024-02-19T20:22:39.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:38.709+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:39.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:39.062+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:39.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:39.217+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:39.218941+0000) 2024-02-19T20:22:39.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:39.242+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:39.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:39.242+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:39.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:39.242+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:39.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:39.243+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:39.689 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:39.438+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:39.438958+0000) 2024-02-19T20:22:39.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:39.526+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:39.527559+0000) 2024-02-19T20:22:40.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:39.701+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:40.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:22:40.013+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59158 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:22:40.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:40.011+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:40.012302+0000) 2024-02-19T20:22:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:40.011+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:40.012123+0000) 2024-02-19T20:22:40.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:40.105+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:40.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:40.012+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:40.013096+0000) 2024-02-19T20:22:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:40.218+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:40.219134+0000) 2024-02-19T20:22:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:40.246+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:40.246+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:40.246+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:12:07.597097+0000 (oldest deadline 2024-02-19T20:12:27.597097+0000) 2024-02-19T20:22:40.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:40.247+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:40.689 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:40.438+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:40.439155+0000) 2024-02-19T20:22:40.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:40.526+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:40.527716+0000) 2024-02-19T20:22:41.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:40.707+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:41.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:41.153+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:41.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:41.220+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:41.221521+0000) 2024-02-19T20:22:41.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:41.292+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:41.692 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:41.440+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:41.441215+0000) 2024-02-19T20:22:41.693 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:41.530+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:41.531050+0000) 2024-02-19T20:22:42.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:41.694+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:42.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:42.175+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:42.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:42.222+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:42.223738+0000) 2024-02-19T20:22:42.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:42.259+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:42.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:42.442+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:42.443788+0000) 2024-02-19T20:22:42.693 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:42.532+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:42.533844+0000) 2024-02-19T20:22:43.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:42.705+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:43.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:43.128+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:43.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:43.224+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:43.225872+0000) 2024-02-19T20:22:43.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:43.275+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:43.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:43.444+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:43.445787+0000) 2024-02-19T20:22:43.693 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:43.536+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:43.537342+0000) 2024-02-19T20:22:44.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:43.733+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:44.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:44.225+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:44.226644+0000) 2024-02-19T20:22:44.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:44.293+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:44.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:44.126+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:44.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:44.445+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:44.446824+0000) 2024-02-19T20:22:44.693 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:44.537+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:44.538248+0000) 2024-02-19T20:22:45.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:44.706+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:45.269 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:22:45.021+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59199 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:22:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:45.017+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:45.018004+0000) 2024-02-19T20:22:45.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:45.122+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:45.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:45.019+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:45.020263+0000) 2024-02-19T20:22:45.270 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:45.018+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:45.019338+0000) 2024-02-19T20:22:45.271 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:45.225+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:45.226829+0000) 2024-02-19T20:22:45.538 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:45.277+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:45.538 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:45.446+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:45.447025+0000) 2024-02-19T20:22:46.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:45.537+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:45.538416+0000) 2024-02-19T20:22:46.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:45.731+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:46.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:46.227+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:46.228933+0000) 2024-02-19T20:22:46.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:46.290+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:46.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:46.122+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:46.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:46.448+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:46.449056+0000) 2024-02-19T20:22:46.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:46.539+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:46.540959+0000) 2024-02-19T20:22:47.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:46.774+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:47.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:47.126+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:47.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:47.229+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:47.230862+0000) 2024-02-19T20:22:47.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:47.277+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:47.697 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:47.450+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:47.451545+0000) 2024-02-19T20:22:47.698 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:47.541+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:47.542951+0000) 2024-02-19T20:22:48.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:47.728+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:48.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:48.122+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:48.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:48.232+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:48.233315+0000) 2024-02-19T20:22:48.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:48.260+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:48.697 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:48.452+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:48.453697+0000) 2024-02-19T20:22:48.698 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:48.544+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:48.545030+0000) 2024-02-19T20:22:49.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:48.718+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:49.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:49.155+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:49.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:49.233+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:49.234013+0000) 2024-02-19T20:22:49.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:49.299+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:49.697 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:49.453+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:49.454366+0000) 2024-02-19T20:22:49.698 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:49.545+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:49.546066+0000) 2024-02-19T20:22:50.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:49.754+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:50.280 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:22:50.028+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59239 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:22:50.280 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:50.025+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:50.026264+0000) 2024-02-19T20:22:50.281 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:50.026+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:50.027644+0000) 2024-02-19T20:22:50.281 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:50.199+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:50.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:50.025+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:50.026035+0000) 2024-02-19T20:22:50.281 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:50.233+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:50.234195+0000) 2024-02-19T20:22:50.282 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:50.279+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:50.546 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:50.453+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:50.454535+0000) 2024-02-19T20:22:51.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:50.545+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:50.546245+0000) 2024-02-19T20:22:51.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:50.746+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:51.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:51.211+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:51.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:51.236+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:51.237736+0000) 2024-02-19T20:22:51.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:51.244+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:51.698 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:51.456+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:51.456959+0000) 2024-02-19T20:22:51.698 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:51.548+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:51.549042+0000) 2024-02-19T20:22:51.698 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:51.697+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:52.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:52.195+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:52.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:52.215+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:52.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:52.238+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:52.239705+0000) 2024-02-19T20:22:52.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:52.458+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:52.458960+0000) 2024-02-19T20:22:52.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:52.550+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:52.551141+0000) 2024-02-19T20:22:53.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:52.724+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:53.375 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:53.181+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:53.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:53.240+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:53.241753+0000) 2024-02-19T20:22:53.376 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:53.196+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:53.700 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:53.460+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:53.461049+0000) 2024-02-19T20:22:53.701 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:53.552+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:53.553167+0000) 2024-02-19T20:22:53.701 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:53.699+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:54.376 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:54.213+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:54.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:54.153+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:54.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:54.241+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:54.242804+0000) 2024-02-19T20:22:54.701 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:54.460+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:54.461714+0000) 2024-02-19T20:22:54.701 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:54.553+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:54.554167+0000) 2024-02-19T20:22:55.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:54.750+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:55.297 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:22:55.036+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59280 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:22:55.298 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:55.034+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:55.035409+0000) 2024-02-19T20:22:55.298 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:55.176+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:55.298 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:55.033+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:55.034096+0000) 2024-02-19T20:22:55.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:55.032+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:55.033405+0000) 2024-02-19T20:22:55.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:55.163+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:55.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:55.242+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:55.243024+0000) 2024-02-19T20:22:55.553 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:55.460+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:55.461903+0000) 2024-02-19T20:22:56.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:55.553+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:55.554369+0000) 2024-02-19T20:22:56.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:55.746+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:56.298 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:56.197+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:56.298 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:56.197+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:56.298 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:56.245+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:56.246607+0000) 2024-02-19T20:22:56.556 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:56.463+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:56.464019+0000) 2024-02-19T20:22:57.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:56.555+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:56.556659+0000) 2024-02-19T20:22:57.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:56.759+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:57.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:57.243+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:57.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:57.197+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:57.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:57.249+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:57.250141+0000) 2024-02-19T20:22:57.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:57.465+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:57.466652+0000) 2024-02-19T20:22:57.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:57.559+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:57.560080+0000) 2024-02-19T20:22:58.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:57.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:22:58.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:57.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:22:58.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:57.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:22:58.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:57.727+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:58.376 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:58.234+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:58.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:58.186+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:58.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:58.251+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:58.252572+0000) 2024-02-19T20:22:58.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:58.467+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:58.468900+0000) 2024-02-19T20:22:58.705 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:58.561+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:58.562502+0000) 2024-02-19T20:22:59.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:58.716+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:22:59.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:58.716+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:22:59.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:58.716+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:22:59.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:58.716+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:59.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:59.172+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:22:59.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:22:59.252+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:59.253695+0000) 2024-02-19T20:22:59.376 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:59.237+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:22:59.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:22:59.468+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:59.469916+0000) 2024-02-19T20:22:59.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:59.562+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:59.563190+0000) 2024-02-19T20:23:00.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:59.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:00.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:59.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:00.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:59.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:00.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:22:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:22:59.757+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:00.298 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:23:00.044+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59320 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:23:00.298 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:00.041+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:00.042515+0000) 2024-02-19T20:23:00.298 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:00.239+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:00.299 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:00.040+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:00.041409+0000) 2024-02-19T20:23:00.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:00.040+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:00.041564+0000) 2024-02-19T20:23:00.299 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:00.221+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:00.300 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:00.252+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:00.253902+0000) 2024-02-19T20:23:00.562 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:00.469+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:00.470104+0000) 2024-02-19T20:23:01.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:00.562+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:00.563376+0000) 2024-02-19T20:23:01.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:00.732+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:01.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:00.732+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:01.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:00.732+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:01.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:00.732+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:01.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:01.218+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:01.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:01.256+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:01.257503+0000) 2024-02-19T20:23:01.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:01.205+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:01.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:01.205+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:01.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:01.205+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:01.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:01.206+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:01.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:01.471+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:01.472203+0000) 2024-02-19T20:23:01.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:01.565+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:01.566009+0000) 2024-02-19T20:23:02.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:01.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:02.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:01.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:02.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:01.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:02.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:01.764+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:02.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:02.239+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:02.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:02.260+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:02.261016+0000) 2024-02-19T20:23:02.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:02.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:02.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:02.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:02.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:02.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:02.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:02.207+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:02.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:02.474+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:02.475451+0000) 2024-02-19T20:23:02.708 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:02.567+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:02.568182+0000) 2024-02-19T20:23:03.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:02.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:03.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:02.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:03.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:02.797+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:03.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:02.797+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:03.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:03.246+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:03.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:03.262+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:03.263389+0000) 2024-02-19T20:23:03.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:03.186+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:03.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:03.186+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:03.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:03.186+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:03.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:03.187+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:03.708 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:03.476+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:03.477571+0000) 2024-02-19T20:23:03.708 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:03.570+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:03.571636+0000) 2024-02-19T20:23:04.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:03.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:04.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:03.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:04.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:03.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:04.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:03.842+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:04.376 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:04.192+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:04.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:04.192+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:04.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:04.192+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:04.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:04.193+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:04.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:04.245+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:04.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:04.245+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:04.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:04.245+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:04.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:04.246+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:04.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:04.263+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:04.264101+0000) 2024-02-19T20:23:04.708 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:04.477+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:04.478293+0000) 2024-02-19T20:23:04.708 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:04.571+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:04.572730+0000) 2024-02-19T20:23:05.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:04.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:05.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:04.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:05.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:04.855+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:05.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:04.855+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:05.263 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:23:05.051+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59361 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:23:05.264 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:05.050+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:05.051808+0000) 2024-02-19T20:23:05.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:05.047+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:05.048773+0000) 2024-02-19T20:23:05.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:05.050+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:05.051639+0000) 2024-02-19T20:23:05.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:05.151+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:05.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:05.151+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:05.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:05.151+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:05.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:05.151+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:05.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:05.477+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:05.478501+0000) 2024-02-19T20:23:05.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:05.263+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:05.264289+0000) 2024-02-19T20:23:05.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:05.288+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:05.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:05.288+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:05.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:05.288+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:05.520 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:05.289+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:05.851 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:05.571+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:05.572941+0000) 2024-02-19T20:23:06.102 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:05.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:06.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:05.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:06.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:05.851+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:06.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:05.852+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:06.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:06.265+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:06.266236+0000) 2024-02-19T20:23:06.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:06.311+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:06.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:06.311+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:06.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:06.311+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:06.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:06.312+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:06.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:06.107+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:06.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:06.107+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:06.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:06.107+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:06.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:06.108+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:06.710 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:06.479+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:06.480441+0000) 2024-02-19T20:23:06.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:06.575+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:06.576471+0000) 2024-02-19T20:23:07.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:06.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:07.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:06.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:07.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:06.859+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:07.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:06.859+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:07.314 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:07.268+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:07.269776+0000) 2024-02-19T20:23:07.314 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:07.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:07.314 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:07.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:07.314 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:07.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:07.315 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:07.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:07.315 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:07.060+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:07.315 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:07.061+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:07.577 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:07.481+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:07.482835+0000) 2024-02-19T20:23:07.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:07.313+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:07.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:07.314+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:08.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:07.577+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:07.578637+0000) 2024-02-19T20:23:08.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:07.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:08.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:07.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:08.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:07.814+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:08.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:07.814+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:08.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:08.270+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:08.271770+0000) 2024-02-19T20:23:08.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:08.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:08.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:08.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:08.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:08.057+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:08.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:08.057+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:08.711 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:08.483+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:08.484905+0000) 2024-02-19T20:23:08.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:08.580+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:08.581094+0000) 2024-02-19T20:23:08.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:08.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:08.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:08.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:08.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:08.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:08.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:08.336+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:09.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:08.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:09.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:08.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:09.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:08.847+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:09.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:08.848+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:09.332 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:09.271+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:09.272536+0000) 2024-02-19T20:23:09.332 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:09.332+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:09.332 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:09.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:09.332 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:09.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:09.332 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:09.073+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:09.333 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:09.073+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:09.711 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:09.484+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:09.485768+0000) 2024-02-19T20:23:09.712 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:09.581+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:09.582194+0000) 2024-02-19T20:23:09.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:09.332+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:09.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:09.332+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:09.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:09.333+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:10.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:09.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:10.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:09.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:10.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:09.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:10.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:09.822+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:10.377 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:23:10.058+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59401 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:23:10.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:10.055+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:10.056384+0000) 2024-02-19T20:23:10.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:10.055+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:10.056078+0000) 2024-02-19T20:23:10.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:10.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:10.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:10.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:10.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:10.094+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:10.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:10.095+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:10.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:10.056+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:10.057619+0000) 2024-02-19T20:23:10.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:10.271+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:10.272760+0000) 2024-02-19T20:23:10.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:10.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:10.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:10.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:10.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:10.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:10.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:10.298+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:10.711 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:10.485+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:10.485970+0000) 2024-02-19T20:23:10.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:10.581+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:10.582356+0000) 2024-02-19T20:23:11.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:10.827+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:11.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:10.827+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:11.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:10.827+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:11.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:10.827+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:11.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:11.093+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:11.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:11.093+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:11.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:11.093+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:11.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:11.093+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:11.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:11.274+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:11.274986+0000) 2024-02-19T20:23:11.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:11.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:11.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:11.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:11.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:11.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:11.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:11.311+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:11.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:11.487+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:11.488476+0000) 2024-02-19T20:23:11.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:11.584+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:11.585935+0000) 2024-02-19T20:23:12.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:11.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:12.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:11.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:12.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:11.861+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:12.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:11.862+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:12.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:12.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:12.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:12.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:12.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:12.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:12.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:12.055+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:12.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:12.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:12.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:12.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:12.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:12.262+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:12.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:12.263+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:12.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:12.276+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:12.277080+0000) 2024-02-19T20:23:12.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:12.489+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:12.490971+0000) 2024-02-19T20:23:12.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:12.587+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:12.588021+0000) 2024-02-19T20:23:13.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:12.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:13.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:12.841+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:13.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:12.842+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:13.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:12.842+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:13.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:13.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:13.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:13.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:13.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:13.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:13.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:13.064+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:13.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:13.278+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:13.279822+0000) 2024-02-19T20:23:13.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:13.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:13.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:13.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:13.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:13.282+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:13.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:13.283+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:13.715 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:13.492+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:13.493507+0000) 2024-02-19T20:23:13.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:13.588+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:13.590679+0000) 2024-02-19T20:23:14.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:13.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:14.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:13.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:14.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:13.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:14.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:13.823+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:14.377 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:14.279+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:14.280592+0000) 2024-02-19T20:23:14.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:14.318+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:14.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:14.318+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:14.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:14.318+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:14.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:14.319+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:14.379 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:14.097+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:14.379 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:14.097+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:14.379 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:14.097+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:14.379 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:14.097+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:14.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:14.493+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:14.494712+0000) 2024-02-19T20:23:14.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:14.590+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:14.591533+0000) 2024-02-19T20:23:15.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:14.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:15.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:14.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:15.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:14.796+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:15.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:14.797+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:15.315 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:23:15.065+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59442 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:23:15.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:15.062+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:15.063974+0000) 2024-02-19T20:23:15.316 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:15.062+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:15.063167+0000) 2024-02-19T20:23:15.316 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:15.279+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:15.280806+0000) 2024-02-19T20:23:15.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:15.063+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:15.064324+0000) 2024-02-19T20:23:15.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:15.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:15.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:15.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:15.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:15.130+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:15.317 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:15.131+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:15.590 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:15.493+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:15.494919+0000) 2024-02-19T20:23:15.591 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:15.315+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:15.591 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:15.315+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:15.591 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:15.315+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:15.591 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:15.316+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:16.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:15.590+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:15.591741+0000) 2024-02-19T20:23:16.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:15.794+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:16.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:15.794+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:16.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:15.794+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:16.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:15.794+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:16.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:16.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:16.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:16.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:16.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:16.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:16.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:16.129+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:16.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:16.283+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:16.284343+0000) 2024-02-19T20:23:16.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:16.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:16.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:16.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:16.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:16.324+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:16.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:16.325+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:16.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:16.496+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:16.497104+0000) 2024-02-19T20:23:16.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:16.593+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:16.594150+0000) 2024-02-19T20:23:17.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:16.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:17.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:16.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:17.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:16.820+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:17.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:16.821+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:17.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:17.160+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:17.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:17.160+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:17.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:17.160+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:17.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:17.161+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:17.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:17.285+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:17.286306+0000) 2024-02-19T20:23:17.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:17.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:17.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:17.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:17.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:17.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:17.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:17.298+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:17.720 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:17.498+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:17.499593+0000) 2024-02-19T20:23:17.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:17.595+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:17.596723+0000) 2024-02-19T20:23:18.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:17.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:18.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:17.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:18.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:17.863+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:18.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:17.863+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:18.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:18.147+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:18.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:18.147+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:18.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:18.147+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:18.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:18.147+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:18.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:18.287+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:18.288927+0000) 2024-02-19T20:23:18.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:18.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:18.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:18.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:18.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:18.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:18.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:18.310+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:18.719 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:18.502+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:18.503145+0000) 2024-02-19T20:23:18.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:18.598+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:18.599212+0000) 2024-02-19T20:23:19.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:18.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:19.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:18.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:19.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:18.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:19.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:18.907+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:19.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:19.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:19.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:19.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:19.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:19.177+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:19.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:19.178+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:19.378 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:19.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:19.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:19.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:19.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:19.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:19.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:19.274+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:19.379 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:19.288+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:19.289653+0000) 2024-02-19T20:23:19.719 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:19.503+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:19.504290+0000) 2024-02-19T20:23:19.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:19.598+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:19.599939+0000) 2024-02-19T20:23:20.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:19.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:20.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:19.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:20.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:19.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:20.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:19.912+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:20.321 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:23:20.073+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59482 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:23:20.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:20.069+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:20.070625+0000) 2024-02-19T20:23:20.322 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:20.069+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:20.070689+0000) 2024-02-19T20:23:20.322 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:20.129+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:20.322 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:20.129+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:20.322 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:20.129+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:20.322 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:20.130+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:20.322 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:20.071+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:20.072086+0000) 2024-02-19T20:23:20.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:20.288+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:20.289838+0000) 2024-02-19T20:23:20.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:20.321+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:20.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:20.321+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:20.323 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:20.321+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:20.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:20.322+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:20.599 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:20.503+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:20.504475+0000) 2024-02-19T20:23:20.930 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:20.599+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:20.600123+0000) 2024-02-19T20:23:21.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:21.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:21.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:21.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:21.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:21.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:21.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:21.099+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:21.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:20.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:21.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:20.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:21.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:20.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:21.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:20.931+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:21.603 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:21.505+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:21.506641+0000) 2024-02-19T20:23:21.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:21.291+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:21.292220+0000) 2024-02-19T20:23:21.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:21.361+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:21.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:21.361+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:21.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:21.361+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:21.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:21.362+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:21.912 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:21.601+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:21.602546+0000) 2024-02-19T20:23:21.912 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:21.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:22.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:21.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:22.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:21.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:22.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:21.913+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:22.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:22.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:22.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:22.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:22.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:22.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:22.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:22.114+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:22.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:22.293+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:22.294583+0000) 2024-02-19T20:23:22.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:22.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:22.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:22.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:22.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:22.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:22.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:22.348+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:22.605 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:22.507+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:22.508765+0000) 2024-02-19T20:23:22.606 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:22.604+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:22.605045+0000) 2024-02-19T20:23:23.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:23.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:23.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:23.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:23.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:23.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:23.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:23.163+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:23.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:22.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:23.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:22.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:23.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:22.931+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:23.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:22.931+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:23.579 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:23.510+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:23.511213+0000) 2024-02-19T20:23:23.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:23.295+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:23.296751+0000) 2024-02-19T20:23:23.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:23.321+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:23.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:23.321+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:23.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:23.321+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:23.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:23.322+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:23.924 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:23.606+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:23.607858+0000) 2024-02-19T20:23:23.924 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:23.924+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:23.924 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:23.924+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:23.924 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:23.924+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:24.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:23.924+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:24.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:24.185+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:24.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:24.185+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:24.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:24.185+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:24.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:24.186+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:24.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:24.296+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:24.297628+0000) 2024-02-19T20:23:24.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:24.320+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:24.512 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:24.320+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:24.512 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:24.320+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:24.512 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:24.321+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:24.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:24.511+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:24.512235+0000) 2024-02-19T20:23:24.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:24.607+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:24.608640+0000) 2024-02-19T20:23:25.199 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:23:25.080+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59523 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:23:25.200 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:25.078+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:25.079364+0000) 2024-02-19T20:23:25.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:24.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:25.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:24.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:25.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:24.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:25.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:24.949+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:25.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:25.077+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:25.078812+0000) 2024-02-19T20:23:25.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:25.077+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:25.078423+0000) 2024-02-19T20:23:25.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:25.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:25.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:25.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:25.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:25.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:25.202 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:25.163+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:25.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:25.296+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:25.297815+0000) 2024-02-19T20:23:25.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:25.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:25.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:25.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:25.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:25.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:25.511 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:25.310+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:25.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:25.511+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:25.512438+0000) 2024-02-19T20:23:25.769 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:25.607+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:25.608856+0000) 2024-02-19T20:23:26.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:26.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:26.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:26.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:26.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:26.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:26.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:26.164+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:26.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:25.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:26.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:25.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:26.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:25.935+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:26.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:25.935+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:26.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:26.514+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:26.515170+0000) 2024-02-19T20:23:26.610 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:26.300+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:26.301081+0000) 2024-02-19T20:23:26.610 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:26.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:26.610 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:26.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:26.610 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:26.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:26.610 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:26.310+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:26.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:26.610+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:26.610999+0000) 2024-02-19T20:23:26.893 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:26.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:26.893 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:26.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:26.893 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:26.893+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:27.144 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:26.893+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:27.516 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:27.145+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:27.516 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:27.145+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:27.516 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:27.145+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:27.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:27.145+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:27.517 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:27.299+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:27.517 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:27.299+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:27.517 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:27.299+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:27.517 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:27.299+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:27.517 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:27.303+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:27.304562+0000) 2024-02-19T20:23:27.768 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:27.516+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:27.517469+0000) 2024-02-19T20:23:27.768 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:27.613+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:27.614585+0000) 2024-02-19T20:23:28.149 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:27.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:28.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:27.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:28.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:27.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:28.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:27.890+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:28.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:28.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:28.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:28.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:28.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:28.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:28.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:28.150+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:28.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:28.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:28.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:28.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:28.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:28.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:28.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:28.271+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:28.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:28.306+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:28.307941+0000) 2024-02-19T20:23:28.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:28.519+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:28.520929+0000) 2024-02-19T20:23:28.919 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:28.616+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:28.616993+0000) 2024-02-19T20:23:29.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:28.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:29.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:28.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:29.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:28.919+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:29.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:28.920+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:29.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:29.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:29.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:29.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:29.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:29.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:29.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:29.200+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:29.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:29.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:29.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:29.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:29.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:29.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:29.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:29.297+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:29.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:29.308+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:29.309072+0000) 2024-02-19T20:23:29.902 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:29.520+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:29.521668+0000) 2024-02-19T20:23:29.903 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:29.617+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:29.618190+0000) 2024-02-19T20:23:30.203 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:23:30.087+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59563 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:23:30.204 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:30.085+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:30.086694+0000) 2024-02-19T20:23:30.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:30.084+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:30.085653+0000) 2024-02-19T20:23:30.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:29.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:30.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:29.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:30.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:29.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:30.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:29.903+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:30.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:30.084+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:30.085974+0000) 2024-02-19T20:23:30.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:30.213+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:30.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:30.213+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:30.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:30.213+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:30.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:30.214+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:30.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:30.308+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:30.309243+0000) 2024-02-19T20:23:30.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:30.325+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:30.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:30.325+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:30.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:30.325+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:30.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:30.326+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:30.881 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:30.520+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:30.521875+0000) 2024-02-19T20:23:30.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:30.617+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:30.618356+0000) 2024-02-19T20:23:30.882 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:30.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:31.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:30.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:31.232 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:30.882+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:31.232 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:30.882+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:31.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:31.302+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:31.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:31.302+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:31.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:31.302+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:31.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:31.302+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:31.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:31.310+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:31.311503+0000) 2024-02-19T20:23:31.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:31.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:31.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:31.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:31.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:31.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:31.519 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:31.233+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:31.898 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:31.523+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:31.524388+0000) 2024-02-19T20:23:31.899 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:31.619+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:31.620498+0000) 2024-02-19T20:23:31.899 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:31.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:31.899 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:31.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:32.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:31.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:32.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:31.899+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:32.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:32.236+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:32.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:32.237+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:32.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:32.237+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:32.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:32.238+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:32.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:32.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:32.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:32.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:32.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:32.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:32.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:32.292+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:32.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:32.313+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:32.314073+0000) 2024-02-19T20:23:32.907 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:32.525+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:32.526942+0000) 2024-02-19T20:23:32.907 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:32.621+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:32.622639+0000) 2024-02-19T20:23:33.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:32.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:33.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:32.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:33.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:32.907+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:33.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:32.908+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:33.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:33.304+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:33.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:33.304+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:33.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:33.304+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:33.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:33.305+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:33.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:33.315+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:33.316249+0000) 2024-02-19T20:23:33.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:33.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:33.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:33.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:33.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:33.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:33.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:33.284+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:33.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:33.529+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:33.530271+0000) 2024-02-19T20:23:33.897 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:33.623+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:33.624792+0000) 2024-02-19T20:23:33.898 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:33.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:33.898 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:33.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:33.898 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:33.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:34.259 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:33.898+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:34.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:34.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:34.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:34.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:34.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:34.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:34.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:34.260+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:34.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:34.316+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:34.317008+0000) 2024-02-19T20:23:34.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:34.329+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:34.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:34.329+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:34.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:34.329+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:34.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:34.330+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:34.867 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:34.530+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:34.531038+0000) 2024-02-19T20:23:34.868 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:34.624+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:34.625658+0000) 2024-02-19T20:23:35.211 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:23:35.094+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59604 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:23:35.212 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:35.091+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:35.092796+0000) 2024-02-19T20:23:35.213 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:35.091+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:35.092460+0000) 2024-02-19T20:23:35.213 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:34.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:35.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:34.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:35.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:34.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:35.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:34.868+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:35.214 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:35.092+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:35.093142+0000) 2024-02-19T20:23:35.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:35.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:35.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:35.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:35.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:35.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:35.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:35.299+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:35.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:35.303+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:35.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:35.303+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:35.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:35.303+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:35.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:35.303+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:35.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:35.316+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:35.317102+0000) 2024-02-19T20:23:35.851 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:35.530+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:35.531212+0000) 2024-02-19T20:23:35.852 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:35.624+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:35.625846+0000) 2024-02-19T20:23:36.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:35.852+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:36.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:35.852+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:36.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:35.852+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:36.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:35.852+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:36.379 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:36.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:36.379 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:36.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:36.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:36.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:36.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:36.283+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:36.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:36.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:36.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:36.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:36.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:36.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:36.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:36.261+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:36.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:36.318+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:36.319316+0000) 2024-02-19T20:23:36.734 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:36.532+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:36.533765+0000) 2024-02-19T20:23:36.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:36.627+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:36.628308+0000) 2024-02-19T20:23:37.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:36.817+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:37.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:36.817+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:37.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:36.817+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:37.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:36.817+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:37.379 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:37.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:37.379 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:37.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:37.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:37.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:37.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:37.259+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:37.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:37.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:37.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:37.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:37.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:37.293+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:37.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:37.293+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:37.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:37.320+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:37.321912+0000) 2024-02-19T20:23:37.735 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:37.535+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:37.536060+0000) 2024-02-19T20:23:37.736 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:37.630+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:37.631067+0000) 2024-02-19T20:23:38.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:37.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:38.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:37.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:38.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:37.791+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:38.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:37.791+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:38.379 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:38.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:38.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:38.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:38.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:38.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:38.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:38.246+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:38.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:38.324+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:38.325497+0000) 2024-02-19T20:23:38.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:38.325+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:38.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:38.325+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:38.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:38.325+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:38.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:38.326+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:38.632 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:38.537+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:38.538425+0000) 2024-02-19T20:23:39.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:38.632+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:38.633679+0000) 2024-02-19T20:23:39.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:38.795+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:39.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:38.795+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:39.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:38.795+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:39.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:38.795+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:39.379 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:39.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:39.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:39.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:39.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:39.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:39.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:39.229+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:39.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:39.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:39.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:39.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:39.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:39.323+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:39.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:39.323+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:39.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:39.325+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:39.326441+0000) 2024-02-19T20:23:39.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:39.538+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:39.539489+0000) 2024-02-19T20:23:40.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:39.633+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:39.634504+0000) 2024-02-19T20:23:40.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:39.817+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:40.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:39.817+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:40.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:39.817+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:40.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:39.817+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:40.379 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:23:40.102+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59644 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:23:40.380 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:40.098+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:40.099895+0000) 2024-02-19T20:23:40.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:40.099+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:40.100439+0000) 2024-02-19T20:23:40.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:40.181+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:40.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:40.181+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:40.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:40.181+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:40.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:40.182+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:40.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:40.099+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:40.100229+0000) 2024-02-19T20:23:40.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:40.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:40.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:40.309+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:40.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:40.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:40.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:40.310+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:40.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:40.325+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:40.326719+0000) 2024-02-19T20:23:40.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:40.538+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:40.539699+0000) 2024-02-19T20:23:41.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:40.633+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:40.634731+0000) 2024-02-19T20:23:41.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:40.781+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:41.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:40.781+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:41.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:40.781+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:41.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:40.781+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:41.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:41.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:41.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:41.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:41.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:41.297+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:41.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:41.297+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:41.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:41.327+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:41.328900+0000) 2024-02-19T20:23:41.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:41.198+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:41.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:41.198+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:41.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:41.198+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:41.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:41.198+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:41.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:41.540+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:41.541703+0000) 2024-02-19T20:23:42.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:41.635+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:41.636838+0000) 2024-02-19T20:23:42.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:41.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:42.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:41.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:42.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:41.800+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:42.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:41.800+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:42.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:42.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:42.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:42.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:42.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:42.168+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:42.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:42.168+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:42.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:42.330+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:42.331080+0000) 2024-02-19T20:23:42.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:42.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:42.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:42.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:42.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:42.337+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:42.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:42.338+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:42.638 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:42.542+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:42.543791+0000) 2024-02-19T20:23:43.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:42.639+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:42.640394+0000) 2024-02-19T20:23:43.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:42.767+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:43.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:42.767+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:43.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:42.767+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:43.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:42.767+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:43.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:43.205+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:43.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:43.205+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:43.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:43.205+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:43.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:43.205+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:43.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:43.333+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:43.334374+0000) 2024-02-19T20:23:43.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:43.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:43.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:43.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:43.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:43.349+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:43.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:43.349+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:43.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:43.545+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:43.546011+0000) 2024-02-19T20:23:44.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:43.642+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:43.643392+0000) 2024-02-19T20:23:44.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:43.741+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:44.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:43.741+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:44.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:43.741+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:44.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:43.742+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:44.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:44.252+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:44.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:44.252+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:44.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:44.252+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:44.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:44.252+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:44.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:44.334+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:44.335373+0000) 2024-02-19T20:23:44.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:44.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:44.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:44.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:44.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:44.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:44.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:44.354+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:44.642 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:44.545+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:44.546950+0000) 2024-02-19T20:23:44.945 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:44.643+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:44.644655+0000) 2024-02-19T20:23:44.945 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:44.737+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:44.945 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:44.737+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:44.945 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:44.737+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:44.946 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:44.738+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:45.226 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:23:45.109+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59685 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:23:45.226 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:45.106+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:45.107535+0000) 2024-02-19T20:23:45.226 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:45.106+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:45.107936+0000) 2024-02-19T20:23:45.227 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:45.106+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:45.107474+0000) 2024-02-19T20:23:45.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:45.259+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:45.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:45.259+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:45.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:45.259+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:45.518 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:45.260+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:45.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:45.334+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:45.335533+0000) 2024-02-19T20:23:45.518 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:45.399+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:45.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:45.399+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:45.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:45.399+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:45.519 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:45.400+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:46.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:45.546+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:45.547134+0000) 2024-02-19T20:23:46.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:45.643+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:45.644839+0000) 2024-02-19T20:23:46.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:45.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:46.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:45.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:46.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:45.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:46.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:45.727+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:46.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:46.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:46.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:46.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:46.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:46.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:46.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:46.216+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:46.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:46.336+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:46.337759+0000) 2024-02-19T20:23:46.645 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:46.548+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:46.549780+0000) 2024-02-19T20:23:46.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:46.383+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:46.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:46.383+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:46.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:46.383+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:46.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:46.384+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:47.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:46.645+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:46.646889+0000) 2024-02-19T20:23:47.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:46.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:47.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:46.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:47.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:46.756+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:47.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:46.757+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:47.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:47.340+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:47.341359+0000) 2024-02-19T20:23:47.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:47.200+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:47.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:47.200+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:47.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:47.200+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:47.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:47.201+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:47.647 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:47.551+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:47.552262+0000) 2024-02-19T20:23:47.648 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:47.400+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:47.648 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:47.400+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:47.648 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:47.400+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:47.648 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:47.400+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:48.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:47.648+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:47.649486+0000) 2024-02-19T20:23:48.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:47.750+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:48.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:47.750+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:48.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:47.750+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:48.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:47.750+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:48.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:48.342+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:48.343837+0000) 2024-02-19T20:23:48.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:48.249+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:48.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:48.249+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:48.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:48.249+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:48.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:48.250+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:48.650 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:48.554+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:48.555882+0000) 2024-02-19T20:23:48.650 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:48.413+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:48.650 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:48.413+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:48.650 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:48.413+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:48.651 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:48.414+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:49.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:48.650+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:48.651906+0000) 2024-02-19T20:23:49.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:48.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:49.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:48.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:49.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:48.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:49.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:48.751+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:49.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:49.344+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:49.345174+0000) 2024-02-19T20:23:49.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:49.207+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:49.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:49.207+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:49.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:49.207+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:49.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:49.208+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:49.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:49.556+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:49.557166+0000) 2024-02-19T20:23:49.653 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:49.415+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:49.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:49.415+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:49.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:49.415+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:49.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:49.416+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:50.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:49.651+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:49.652928+0000) 2024-02-19T20:23:50.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:49.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:50.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:49.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:50.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:49.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:50.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:49.761+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:50.380 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:23:50.116+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59725 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:23:50.381 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:50.113+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:50.114548+0000) 2024-02-19T20:23:50.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:50.113+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:50.114021+0000) 2024-02-19T20:23:50.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:50.344+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:50.345368+0000) 2024-02-19T20:23:50.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:50.114+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:50.115156+0000) 2024-02-19T20:23:50.382 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:50.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:50.382 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:50.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:50.382 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:50.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:50.382 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:50.224+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:50.651 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:50.556+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:50.557318+0000) 2024-02-19T20:23:50.651 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:50.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:50.651 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:50.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:50.651 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:50.458+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:50.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:50.459+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:51.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:50.652+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:50.653118+0000) 2024-02-19T20:23:51.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:50.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:51.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:50.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:51.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:50.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:51.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:50.751+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:51.345 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:51.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:51.345 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:51.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:51.346 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:51.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:51.346 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:51.235+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:51.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:51.558+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:51.559664+0000) 2024-02-19T20:23:51.653 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:51.346+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:51.347572+0000) 2024-02-19T20:23:51.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:51.459+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:51.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:51.459+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:51.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:51.459+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:51.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:51.460+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:52.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:51.654+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:51.655430+0000) 2024-02-19T20:23:52.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:51.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:52.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:51.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:52.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:51.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:52.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:51.776+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:52.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:52.348+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:52.349735+0000) 2024-02-19T20:23:52.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:52.266+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:52.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:52.266+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:52.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:52.266+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:52.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:52.267+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:52.656 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:52.561+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:52.562195+0000) 2024-02-19T20:23:52.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:52.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:52.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:52.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:52.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:52.506+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:52.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:52.507+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:53.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:52.657+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:52.658647+0000) 2024-02-19T20:23:53.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:52.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:53.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:52.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:53.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:52.778+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:53.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:52.778+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:53.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:53.351+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:53.352172+0000) 2024-02-19T20:23:53.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:53.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:53.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:53.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:53.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:53.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:53.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:53.317+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:53.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:53.563+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:53.564580+0000) 2024-02-19T20:23:53.659 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:53.469+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:53.659 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:53.469+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:53.659 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:53.469+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:53.660 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:53.470+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:54.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:53.660+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:53.660976+0000) 2024-02-19T20:23:54.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:53.750+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:54.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:53.750+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:54.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:53.750+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:54.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:53.750+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:54.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:54.356+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:54.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:54.356+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:54.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:54.356+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:54.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:54.357+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:54.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:54.352+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:54.352977+0000) 2024-02-19T20:23:54.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:54.564+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:54.565427+0000) 2024-02-19T20:23:54.660 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:54.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:54.660 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:54.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:54.660 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:54.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:54.660 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:54.432+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:55.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:54.660+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:54.661831+0000) 2024-02-19T20:23:55.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:54.719+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:55.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:54.719+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:55.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:54.719+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:55.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:54.719+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:55.381 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:23:55.123+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59766 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:23:55.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:55.120+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:55.121344+0000) 2024-02-19T20:23:55.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:55.120+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:55.121755+0000) 2024-02-19T20:23:55.382 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:55.352+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:55.353175+0000) 2024-02-19T20:23:55.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:55.119+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:55.120881+0000) 2024-02-19T20:23:55.660 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:55.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:55.660 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:55.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:55.660 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:55.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:55.660 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:55.461+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:55.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:55.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:55.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:55.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:55.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:55.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:55.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:55.383+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:55.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:55.564+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:55.565655+0000) 2024-02-19T20:23:56.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:55.661+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:55.662034+0000) 2024-02-19T20:23:56.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:55.758+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:56.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:55.758+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:56.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:55.758+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:56.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:55.758+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:56.565 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:56.361+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:56.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:56.361+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:56.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:56.361+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:56.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:56.362+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:56.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:56.355+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:56.356323+0000) 2024-02-19T20:23:56.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:56.424+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:56.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:56.424+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:56.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:56.424+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:56.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:56.425+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:57.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:56.566+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:56.567685+0000) 2024-02-19T20:23:57.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:56.664+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:56.665516+0000) 2024-02-19T20:23:57.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:56.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:57.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:56.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:57.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:56.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:57.019 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:56.726+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:57.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:57.358+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:57.359823+0000) 2024-02-19T20:23:57.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:57.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:57.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:57.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:57.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:57.460+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:57.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:57.460+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:57.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:57.360+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:57.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:57.360+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:57.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:57.360+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:57.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:57.360+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:57.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:57.569+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:57.570188+0000) 2024-02-19T20:23:58.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:57.668+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:57.669069+0000) 2024-02-19T20:23:58.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:57.709+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:58.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:57.709+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:58.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:57.709+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:58.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:57.710+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:58.360 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:58.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:58.360 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:58.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:58.360 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:58.316+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:58.360 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:58.317+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:58.669 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:58.572+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:58.573464+0000) 2024-02-19T20:23:58.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:58.361+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:58.362388+0000) 2024-02-19T20:23:58.670 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:58.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:58.670 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:58.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:58.670 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:58.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:58.670 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:58.462+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:59.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:58.670+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:58.671573+0000) 2024-02-19T20:23:59.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:58.733+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:59.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:58.734+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:59.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:58.734+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:23:59.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:58.734+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:23:59.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:59.276+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:59.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:59.276+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:59.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:59.276+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:23:59.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:59.276+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:23:59.670 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:23:59.573+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:59.574504+0000) 2024-02-19T20:23:59.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:59.362+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:59.363729+0000) 2024-02-19T20:23:59.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:59.490+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:59.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:59.490+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:59.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:59.490+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:23:59.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:23:59.491+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:00.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:59.671+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:59.672473+0000) 2024-02-19T20:24:00.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:59.739+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:00.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:59.739+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:00.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:59.739+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:00.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:23:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:23:59.739+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:00.361 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:24:00.130+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59806 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:24:00.362 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:00.128+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:00.129472+0000) 2024-02-19T20:24:00.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:00.127+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:00.128428+0000) 2024-02-19T20:24:00.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:00.272+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:00.363 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:00.272+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:00.363 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:00.272+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:00.363 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:00.273+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:00.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:00.127+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:00.128721+0000) 2024-02-19T20:24:00.670 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:00.573+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:00.574706+0000) 2024-02-19T20:24:00.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:00.363+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:00.363957+0000) 2024-02-19T20:24:00.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:00.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:00.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:00.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:00.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:00.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:00.671 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:00.536+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:01.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:00.671+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:00.672660+0000) 2024-02-19T20:24:01.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:00.729+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:01.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:00.729+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:01.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:00.729+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:01.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:00.730+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:01.364 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:01.291+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:01.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:01.291+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:01.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:01.291+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:01.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:01.291+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:01.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:01.574+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:01.576725+0000) 2024-02-19T20:24:01.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:01.366+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:01.367161+0000) 2024-02-19T20:24:01.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:01.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:01.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:01.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:01.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:01.545+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:01.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:01.545+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:02.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:01.673+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:01.675289+0000) 2024-02-19T20:24:02.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:01.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:02.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:01.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:02.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:01.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:02.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:01.763+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:02.367 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:02.280+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:02.368 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:02.280+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:02.368 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:02.280+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:02.368 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:02.281+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:02.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:02.578+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:02.579219+0000) 2024-02-19T20:24:02.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:02.368+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:02.369870+0000) 2024-02-19T20:24:02.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:02.541+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:02.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:02.541+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:02.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:02.541+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:02.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:02.542+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:03.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:02.676+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:02.677573+0000) 2024-02-19T20:24:03.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:02.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:03.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:02.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:03.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:02.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:03.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:02.813+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:03.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:03.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:03.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:03.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:03.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:03.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:03.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:03.289+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:03.678 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:03.580+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:03.581801+0000) 2024-02-19T20:24:03.678 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:03.372+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:03.373273+0000) 2024-02-19T20:24:03.678 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:03.515+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:03.678 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:03.515+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:03.678 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:03.515+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:03.678 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:03.517+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:04.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:03.679+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:03.680305+0000) 2024-02-19T20:24:04.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:03.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:04.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:03.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:04.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:03.845+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:04.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:03.845+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:04.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:04.240+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:04.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:04.240+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:04.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:04.240+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:04.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:04.241+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:04.679 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:04.581+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:04.582726+0000) 2024-02-19T20:24:04.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:04.373+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:04.374290+0000) 2024-02-19T20:24:04.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:04.508+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:04.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:04.508+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:04.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:04.508+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:04.680 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:04.509+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:05.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:04.680+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:04.681375+0000) 2024-02-19T20:24:05.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:04.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:05.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:04.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:05.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:04.824+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:05.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:04.824+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:05.372 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:24:05.138+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59847 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:24:05.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:05.135+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:05.136614+0000) 2024-02-19T20:24:05.372 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:05.135+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:05.136730+0000) 2024-02-19T20:24:05.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:05.135+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:05.136338+0000) 2024-02-19T20:24:05.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:05.210+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:05.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:05.210+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:05.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:05.210+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:05.373 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:05.211+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:05.679 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:05.581+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:05.582938+0000) 2024-02-19T20:24:05.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:05.373+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:05.374505+0000) 2024-02-19T20:24:05.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:05.504+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:05.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:05.504+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:05.680 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:05.504+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:05.680 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:05.505+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:06.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:05.680+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:05.681559+0000) 2024-02-19T20:24:06.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:05.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:06.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:05.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:06.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:05.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:06.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:05.810+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:06.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:06.197+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:06.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:06.197+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:06.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:06.197+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:06.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:06.197+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:06.681 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:06.585+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:06.586468+0000) 2024-02-19T20:24:06.681 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:06.376+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:06.377600+0000) 2024-02-19T20:24:06.681 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:06.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:06.681 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:06.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:06.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:06.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:06.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:06.463+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:07.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:06.682+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:06.683632+0000) 2024-02-19T20:24:07.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:06.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:07.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:06.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:07.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:06.850+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:07.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:06.850+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:07.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:07.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:07.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:07.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:07.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:07.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:07.378 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:07.150+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:07.683 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:07.587+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:07.588602+0000) 2024-02-19T20:24:07.683 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:07.378+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:07.379912+0000) 2024-02-19T20:24:07.684 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:07.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:07.684 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:07.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:07.684 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:07.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:07.684 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:07.492+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:08.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:07.684+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:07.685808+0000) 2024-02-19T20:24:08.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:07.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:08.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:07.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:08.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:07.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:08.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:07.872+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:08.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:08.197+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:08.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:08.197+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:08.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:08.197+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:08.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:08.198+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:08.687 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:08.590+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:08.591185+0000) 2024-02-19T20:24:08.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:08.381+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:08.382472+0000) 2024-02-19T20:24:08.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:08.486+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:08.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:08.486+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:08.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:08.486+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:08.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:08.487+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:09.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:08.688+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:08.689409+0000) 2024-02-19T20:24:09.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:08.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:09.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:08.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:09.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:08.899+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:09.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:08.899+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:09.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:09.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:09.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:09.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:09.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:09.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:09.382 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:09.157+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:09.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:09.591+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:09.592030+0000) 2024-02-19T20:24:09.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:09.382+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:09.383809+0000) 2024-02-19T20:24:09.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:09.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:09.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:09.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:09.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:09.462+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:09.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:09.462+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:10.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:09.689+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:09.690742+0000) 2024-02-19T20:24:10.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:09.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:10.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:09.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:10.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:09.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:10.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:09.911+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:10.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:10.141+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:10.142854+0000) 2024-02-19T20:24:10.382 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:24:10.145+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59887 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:24:10.382 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:10.142+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:10.143500+0000) 2024-02-19T20:24:10.382 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:10.141+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:10.142874+0000) 2024-02-19T20:24:10.382 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:10.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:10.383 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:10.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:10.383 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:10.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:10.383 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:10.164+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:10.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:10.591+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:10.592231+0000) 2024-02-19T20:24:10.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:10.383+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:10.383997+0000) 2024-02-19T20:24:10.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:10.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:10.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:10.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:10.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:10.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:10.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:10.432+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:11.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:10.690+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:10.691061+0000) 2024-02-19T20:24:11.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:10.924+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:11.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:10.924+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:11.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:10.924+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:11.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:10.924+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:11.382 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:11.118+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:11.382 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:11.118+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:11.382 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:11.118+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:11.383 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:11.119+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:11.691 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:11.593+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:11.594459+0000) 2024-02-19T20:24:11.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:11.385+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:11.386478+0000) 2024-02-19T20:24:11.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:11.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:11.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:11.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:11.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:11.425+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:11.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:11.426+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:11.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:11.692+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:11.693789+0000) 2024-02-19T20:24:11.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:11.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:11.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:11.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:11.962 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:11.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:12.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:11.963+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:12.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:12.121+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:12.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:12.121+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:12.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:12.121+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:12.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:12.122+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:12.594 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:12.388+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:12.389196+0000) 2024-02-19T20:24:12.594 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:12.426+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:12.594 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:12.426+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:12.595 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:12.426+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:12.595 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:12.427+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:12.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:12.595+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:12.596509+0000) 2024-02-19T20:24:12.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:12.696+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:12.697239+0000) 2024-02-19T20:24:12.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:12.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:12.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:12.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:12.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:12.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:13.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:12.982+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:13.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:13.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:13.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:13.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:13.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:13.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:13.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:13.112+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:13.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:13.390+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:13.391793+0000) 2024-02-19T20:24:13.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:13.413+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:13.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:13.413+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:13.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:13.413+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:13.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:13.414+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:13.994 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:13.598+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:13.599037+0000) 2024-02-19T20:24:13.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:13.698+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:13.699882+0000) 2024-02-19T20:24:13.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:13.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:13.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:13.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:14.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:13.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:14.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:13.996+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:14.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:14.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:14.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:14.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:14.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:14.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:14.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:14.098+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:14.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:14.391+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:14.392682+0000) 2024-02-19T20:24:14.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:14.409+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:14.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:14.409+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:14.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:14.409+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:14.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:14.409+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:14.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:14.599+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:14.600572+0000) 2024-02-19T20:24:14.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:14.699+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:14.700738+0000) 2024-02-19T20:24:15.267 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:24:15.152+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59928 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:24:15.267 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:15.149+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:15.150191+0000) 2024-02-19T20:24:15.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:14.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:15.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:14.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:15.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:14.984+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:15.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:14.984+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:15.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:15.149+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:15.150563+0000) 2024-02-19T20:24:15.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:15.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:15.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:15.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:15.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:15.139+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:15.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:15.140+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:15.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:15.149+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:15.150658+0000) 2024-02-19T20:24:15.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:15.391+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:15.392875+0000) 2024-02-19T20:24:15.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:15.432+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:15.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:15.432+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:15.598 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:15.432+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:15.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:15.432+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:16.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:15.599+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:15.600835+0000) 2024-02-19T20:24:16.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:15.699+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:15.700964+0000) 2024-02-19T20:24:16.304 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:16.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:16.304 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:16.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:16.304 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:16.169+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:16.304 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:16.169+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:16.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:16.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:16.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:16.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:16.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:16.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:16.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:16.032+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:16.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:16.393+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:16.394884+0000) 2024-02-19T20:24:16.601 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:16.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:16.601 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:16.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:16.601 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:16.440+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:16.601 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:16.441+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:17.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:16.702+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:16.702988+0000) 2024-02-19T20:24:17.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:16.602+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:16.603380+0000) 2024-02-19T20:24:17.304 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:17.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:17.304 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:17.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:17.304 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:17.150+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:17.305 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:17.151+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:17.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:17.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:17.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:17.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:17.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:17.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:17.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:17.053+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:17.603 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:17.396+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:17.397017+0000) 2024-02-19T20:24:17.603 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:17.444+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:17.603 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:17.444+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:17.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:17.444+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:17.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:17.445+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:18.016 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:17.604+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:17.605819+0000) 2024-02-19T20:24:18.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:17.704+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:17.705416+0000) 2024-02-19T20:24:18.304 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:18.201+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:18.304 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:18.201+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:18.304 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:18.201+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:18.305 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:18.201+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:18.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:18.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:18.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:18.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:18.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:18.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:18.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:18.029+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:18.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:18.398+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:18.399386+0000) 2024-02-19T20:24:18.606 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:18.399+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:18.606 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:18.399+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:18.606 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:18.399+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:18.606 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:18.400+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:19.016 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:18.607+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:18.608395+0000) 2024-02-19T20:24:19.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:18.706+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:18.707427+0000) 2024-02-19T20:24:19.304 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:19.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:19.304 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:19.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:19.305 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:19.245+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:19.305 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:19.246+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:19.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:19.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:19.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:19.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:19.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:19.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:19.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:19.024+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:19.606 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:19.397+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:19.607 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:19.397+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:19.607 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:19.397+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:19.607 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:19.398+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:19.607 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:19.399+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:19.400712+0000) 2024-02-19T20:24:19.997 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:19.608+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:19.609516+0000) 2024-02-19T20:24:19.997 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:19.707+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:19.708470+0000) 2024-02-19T20:24:20.266 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:20.158+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:20.159482+0000) 2024-02-19T20:24:20.267 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:24:20.160+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 59968 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:24:20.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:20.157+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:20.158380+0000) 2024-02-19T20:24:20.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:19.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:20.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:19.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:20.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:19.998+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:20.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:19.999+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:20.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:20.156+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:20.157433+0000) 2024-02-19T20:24:20.607 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:20.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:20.607 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:20.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:20.607 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:20.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:20.607 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:20.379+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:20.607 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:20.399+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:20.400919+0000) 2024-02-19T20:24:20.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:20.269+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:20.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:20.270+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:20.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:20.270+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:20.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:20.270+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:20.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:20.608+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:20.609730+0000) 2024-02-19T20:24:20.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:20.707+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:20.708668+0000) 2024-02-19T20:24:21.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:20.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:21.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:20.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:21.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:20.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:21.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:20.972+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:21.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:21.300+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:21.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:21.300+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:21.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:21.300+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:21.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:21.301+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:21.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:21.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:21.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:21.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:21.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:21.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:21.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:21.364+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:21.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:21.402+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:21.402999+0000) 2024-02-19T20:24:21.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:21.611+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:21.612822+0000) 2024-02-19T20:24:21.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:21.711+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:21.712114+0000) 2024-02-19T20:24:22.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:21.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:22.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:21.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:22.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:21.957+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:22.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:21.958+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:22.613 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:22.390+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:22.613 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:22.390+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:22.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:22.390+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:22.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:22.391+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:22.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:22.405+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:22.406451+0000) 2024-02-19T20:24:22.614 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:22.322+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:22.614 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:22.322+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:22.614 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:22.322+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:22.615 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:22.322+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:22.615 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:22.615+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:22.616185+0000) 2024-02-19T20:24:22.928 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:22.713+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:22.714577+0000) 2024-02-19T20:24:22.928 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:22.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:22.928 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:22.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:22.929 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:22.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:23.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:22.930+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:23.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:23.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:23.579 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:23.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:23.579 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:23.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:23.580 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:23.337+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:23.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:23.385+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:23.580 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:23.385+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:23.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:23.385+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:23.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:23.386+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:23.581 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:23.408+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:23.409809+0000) 2024-02-19T20:24:23.907 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:23.618+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:23.619612+0000) 2024-02-19T20:24:23.908 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:23.717+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:23.718203+0000) 2024-02-19T20:24:23.908 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:23.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:23.908 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:23.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:23.908 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:23.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:24.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:23.909+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:24.618 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:24.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:24.618 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:24.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:24.618 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:24.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:24.619 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:24.320+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:24.619 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:24.409+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:24.410715+0000) 2024-02-19T20:24:24.619 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:24.430+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:24.619 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:24.430+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:24.619 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:24.430+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:24.619 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:24.431+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:24.880 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:24.620+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:24.621026+0000) 2024-02-19T20:24:24.881 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:24.718+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:24.719044+0000) 2024-02-19T20:24:25.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:24.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:25.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:24.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:25.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:24.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:25.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:24.896+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:25.475 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:24:25.167+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60009 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:24:25.475 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:25.165+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:25.166383+0000) 2024-02-19T20:24:25.476 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:25.165+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:25.166765+0000) 2024-02-19T20:24:25.476 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:25.409+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:25.410842+0000) 2024-02-19T20:24:25.476 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:25.164+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:25.165316+0000) 2024-02-19T20:24:25.476 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:25.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:25.476 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:25.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:25.476 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:25.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:25.477 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:25.278+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:25.766 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:25.620+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:25.621226+0000) 2024-02-19T20:24:25.767 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:25.718+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:25.719231+0000) 2024-02-19T20:24:25.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:25.477+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:25.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:25.477+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:25.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:25.477+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:25.768 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:25.477+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:26.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:25.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:26.110 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:25.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:26.110 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:25.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:26.110 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:25.938+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:26.383 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:26.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:26.383 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:26.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:26.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:26.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:26.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:26.285+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:26.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:26.622+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:26.623131+0000) 2024-02-19T20:24:26.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:26.411+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:26.412894+0000) 2024-02-19T20:24:26.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:26.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:26.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:26.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:26.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:26.475+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:26.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:26.477+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:27.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:26.720+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:26.721196+0000) 2024-02-19T20:24:27.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:26.939+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:27.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:26.939+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:27.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:26.939+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:27.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:26.939+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:27.383 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:27.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:27.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:27.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:27.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:27.299+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:27.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:27.299+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:27.720 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:27.624+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:27.625718+0000) 2024-02-19T20:24:27.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:27.414+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:27.415445+0000) 2024-02-19T20:24:27.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:27.485+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:27.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:27.485+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:27.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:27.485+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:27.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:27.486+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:28.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:27.722+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:27.723465+0000) 2024-02-19T20:24:28.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:27.953+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:28.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:27.953+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:28.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:27.953+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:28.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:27.953+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:28.383 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:28.302+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:28.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:28.302+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:28.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:28.302+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:28.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:28.303+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:28.645 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:28.628+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:28.629241+0000) 2024-02-19T20:24:28.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:28.417+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:28.418776+0000) 2024-02-19T20:24:28.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:28.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:28.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:28.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:28.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:28.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:28.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:28.515+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:28.919 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:28.724+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:28.725882+0000) 2024-02-19T20:24:28.920 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:28.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:28.920 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:28.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:28.920 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:28.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:29.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:28.922+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:29.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:29.419+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:29.420152+0000) 2024-02-19T20:24:29.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:29.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:29.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:29.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:29.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:29.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:29.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:29.524+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:29.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:29.321+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:29.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:29.321+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:29.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:29.321+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:29.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:29.322+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:29.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:29.629+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:29.630253+0000) 2024-02-19T20:24:29.962 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:29.725+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:29.726941+0000) 2024-02-19T20:24:29.962 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:29.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:30.266 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:24:30.174+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60049 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:24:30.267 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:30.172+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:30.173166+0000) 2024-02-19T20:24:30.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:30.171+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:30.172360+0000) 2024-02-19T20:24:30.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:29.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:30.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:29.963+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:30.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:29.964+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:30.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:30.171+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:30.172770+0000) 2024-02-19T20:24:30.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:30.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:30.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:30.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:30.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:30.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:30.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:30.367+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:30.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:30.419+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:30.420366+0000) 2024-02-19T20:24:30.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:30.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:30.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:30.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:30.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:30.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:30.629 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:30.506+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:30.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:30.629+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:30.630424+0000) 2024-02-19T20:24:30.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:30.726+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:30.727106+0000) 2024-02-19T20:24:30.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:30.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:30.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:30.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:30.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:30.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:31.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:30.976+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:31.630 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:31.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:31.630 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:31.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:31.630 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:31.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:31.631 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:31.349+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:31.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:31.421+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:31.422390+0000) 2024-02-19T20:24:31.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:31.549+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:31.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:31.549+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:31.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:31.549+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:31.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:31.550+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:32.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:31.728+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:31.729276+0000) 2024-02-19T20:24:32.014 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:31.632+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:31.633284+0000) 2024-02-19T20:24:32.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:32.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:32.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:32.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:32.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:32.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:32.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:32.016+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:32.632 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:32.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:32.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:32.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:32.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:32.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:32.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:32.360+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:32.633 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:32.423+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:32.424463+0000) 2024-02-19T20:24:32.633 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:32.540+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:32.633 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:32.540+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:32.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:32.540+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:32.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:32.541+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:33.002 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:32.634+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:32.635796+0000) 2024-02-19T20:24:33.002 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:32.730+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:32.731384+0000) 2024-02-19T20:24:33.002 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:33.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:33.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:33.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:33.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:33.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:33.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:33.005+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:33.577 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:33.426+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:33.427837+0000) 2024-02-19T20:24:33.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:33.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:33.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:33.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:33.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:33.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:33.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:33.313+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:33.995 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:33.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:33.995 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:33.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:33.995 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:33.586+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:33.995 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:33.587+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:33.996 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:33.636+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:33.637914+0000) 2024-02-19T20:24:33.996 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:33.732+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:33.733802+0000) 2024-02-19T20:24:33.996 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:33.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:33.996 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:33.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:33.996 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:33.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:34.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:33.997+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:34.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:34.428+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:34.429260+0000) 2024-02-19T20:24:34.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:34.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:34.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:34.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:34.567 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:34.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:34.567 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:34.333+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:34.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:34.638+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:34.639307+0000) 2024-02-19T20:24:34.966 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:34.734+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:34.735048+0000) 2024-02-19T20:24:34.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:34.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:34.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:34.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:34.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:34.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:34.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:34.569+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:35.266 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:24:35.181+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60090 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:24:35.267 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:35.179+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:35.180044+0000) 2024-02-19T20:24:35.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:35.179+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:35.180134+0000) 2024-02-19T20:24:35.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:34.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:35.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:34.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:35.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:34.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:35.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:34.969+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:35.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:35.177+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:35.178833+0000) 2024-02-19T20:24:35.608 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:35.428+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:35.429430+0000) 2024-02-19T20:24:35.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:35.362+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:35.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:35.362+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:35.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:35.362+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:35.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:35.363+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:35.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:35.638+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:35.639486+0000) 2024-02-19T20:24:35.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:35.734+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:35.735218+0000) 2024-02-19T20:24:35.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:35.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:35.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:35.610+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:35.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:35.610+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:35.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:35.610+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:35.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:35.611+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:36.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:35.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:36.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:35.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:36.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:35.959+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:36.609 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:36.431+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:36.432668+0000) 2024-02-19T20:24:36.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:36.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:36.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:36.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:36.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:36.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:36.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:36.346+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:36.948 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:36.640+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:36.641475+0000) 2024-02-19T20:24:36.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:36.737+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:36.738784+0000) 2024-02-19T20:24:36.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:36.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:36.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:36.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:36.949 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:36.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:36.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:36.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:36.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:36.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:36.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:36.613+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:37.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:36.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:37.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:36.951+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:37.641 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:37.433+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:37.434748+0000) 2024-02-19T20:24:37.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:37.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:37.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:37.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:37.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:37.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:37.642 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:37.321+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:37.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:37.643+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:37.644077+0000) 2024-02-19T20:24:37.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:37.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:37.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:37.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:37.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:37.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:37.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:37.659+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:37.925 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:37.740+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:37.741000+0000) 2024-02-19T20:24:37.925 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:37.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:37.925 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:37.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:37.925 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:37.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:38.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:37.926+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:38.643 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:38.435+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:38.436837+0000) 2024-02-19T20:24:38.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:38.321+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:38.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:38.321+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:38.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:38.321+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:38.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:38.322+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:38.934 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:38.645+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:38.646294+0000) 2024-02-19T20:24:38.935 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:38.742+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:38.743149+0000) 2024-02-19T20:24:38.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:38.687+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:38.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:38.687+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:38.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:38.687+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:38.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:38.688+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:39.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:38.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:39.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:38.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:39.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:38.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:39.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:38.937+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:39.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:39.436+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:39.437781+0000) 2024-02-19T20:24:39.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:39.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:39.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:39.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:39.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:39.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:39.645 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:39.277+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:40.016 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:39.646+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:39.647261+0000) 2024-02-19T20:24:40.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:39.647+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:40.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:39.647+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:40.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:39.647+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:40.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:39.648+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:40.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:39.743+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:39.744186+0000) 2024-02-19T20:24:40.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:39.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:40.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:39.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:40.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:39.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:40.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:39.895+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:40.384 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:24:40.188+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60130 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:24:40.385 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:40.186+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:40.187321+0000) 2024-02-19T20:24:40.385 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:40.186+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:40.187699+0000) 2024-02-19T20:24:40.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:40.185+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:40.186760+0000) 2024-02-19T20:24:40.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:40.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:40.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:40.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:40.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:40.297+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:40.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:40.298+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:40.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:40.437+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:40.438008+0000) 2024-02-19T20:24:40.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:40.601+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:40.644 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:40.601+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:40.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:40.601+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:40.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:40.602+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:40.902 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:40.646+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:40.647451+0000) 2024-02-19T20:24:40.902 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:40.743+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:40.744400+0000) 2024-02-19T20:24:40.902 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:40.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:40.902 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:40.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:41.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:40.904+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:41.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:40.905+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:41.647 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:41.440+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:41.441274+0000) 2024-02-19T20:24:41.647 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:41.278+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:41.648 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:41.278+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:41.648 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:41.278+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:41.648 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:41.278+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:42.016 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:41.649+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:41.650718+0000) 2024-02-19T20:24:42.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:41.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:42.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:41.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:42.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:41.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:42.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:41.651+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:42.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:41.746+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:41.747931+0000) 2024-02-19T20:24:42.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:41.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:42.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:41.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:42.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:41.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:42.018 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:41.872+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:42.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:42.302+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:42.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:42.302+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:42.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:42.302+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:42.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:42.303+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:42.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:42.442+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:42.443429+0000) 2024-02-19T20:24:42.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:42.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:42.646 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:42.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:43.019 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:42.653+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:42.654002+0000) 2024-02-19T20:24:43.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:42.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:43.020 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:42.649+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:43.020 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:42.749+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:42.750633+0000) 2024-02-19T20:24:43.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:42.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:43.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:42.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:43.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:42.872+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:43.021 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:42.873+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:43.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:43.273+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:43.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:43.273+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:43.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:43.273+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:43.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:43.274+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:43.647 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:43.444+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:43.445447+0000) 2024-02-19T20:24:43.647 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:43.649+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:43.647 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:43.649+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:43.647 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:43.649+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:43.899 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:43.656+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:43.657575+0000) 2024-02-19T20:24:43.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:43.650+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:43.899 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:43.751+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:43.752687+0000) 2024-02-19T20:24:43.899 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:43.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:44.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:43.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:44.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:43.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:44.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:43.901+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:44.516 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:44.445+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:44.446301+0000) 2024-02-19T20:24:44.516 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:44.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:44.516 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:44.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:44.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:44.260+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:44.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:44.261+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:44.878 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:44.657+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:44.658616+0000) 2024-02-19T20:24:44.879 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:44.753+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:44.754156+0000) 2024-02-19T20:24:44.879 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:44.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:44.879 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:44.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:44.879 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:44.880+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:44.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:44.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:44.879 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:44.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:44.880 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:44.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:44.880 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:44.620+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:45.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:44.881+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:45.443 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:24:45.195+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60171 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:24:45.443 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:45.192+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:45.193908+0000) 2024-02-19T20:24:45.444 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:45.193+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:45.194466+0000) 2024-02-19T20:24:45.444 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:45.192+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:45.193666+0000) 2024-02-19T20:24:45.444 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:45.236+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:45.444 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:45.236+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:45.444 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:45.236+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:45.445 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:45.236+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:45.751 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:45.657+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:45.658834+0000) 2024-02-19T20:24:45.751 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:45.445+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:45.446468+0000) 2024-02-19T20:24:45.751 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:45.585+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:45.751 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:45.585+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:45.751 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:45.585+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:45.751 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:45.586+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:46.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:45.753+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:45.754308+0000) 2024-02-19T20:24:46.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:45.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:46.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:45.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:46.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:45.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:46.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:45.909+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:46.383 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:46.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:46.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:46.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:46.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:46.206+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:46.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:46.207+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:46.659 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:46.448+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:46.449011+0000) 2024-02-19T20:24:46.659 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:46.566+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:46.659 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:46.566+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:46.660 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:46.566+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:46.660 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:46.566+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:47.016 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:46.661+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:46.662361+0000) 2024-02-19T20:24:47.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:46.756+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:46.757688+0000) 2024-02-19T20:24:47.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:46.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:47.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:46.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:47.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:46.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:47.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:46.894+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:47.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:47.256+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:47.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:47.256+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:47.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:47.256+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:47.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:47.256+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:47.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:47.450+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:47.451026+0000) 2024-02-19T20:24:47.662 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:47.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:47.662 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:47.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:47.662 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:47.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:47.662 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:47.524+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:47.914 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:47.759+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:47.760156+0000) 2024-02-19T20:24:47.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:47.663+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:47.664924+0000) 2024-02-19T20:24:48.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:47.916+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:48.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:47.916+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:48.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:47.916+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:48.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:47.917+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:48.553 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:48.452+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:48.453590+0000) 2024-02-19T20:24:48.553 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:48.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:48.553 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:48.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:48.553 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:48.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:48.553 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:48.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:48.554 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:48.292+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:48.554 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:48.293+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:48.854 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:48.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:48.854 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:48.556+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:48.855 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:48.666+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:48.667210+0000) 2024-02-19T20:24:48.855 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:48.761+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:48.762652+0000) 2024-02-19T20:24:49.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:48.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:49.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:48.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:49.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:48.941+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:49.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:48.942+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:49.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:49.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:49.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:49.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:49.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:49.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:49.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:49.252+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:49.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:49.453+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:49.455028+0000) 2024-02-19T20:24:49.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:49.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:49.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:49.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:49.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:49.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:49.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:49.590+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:49.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:49.666+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:49.668643+0000) 2024-02-19T20:24:49.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:49.762+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:49.763780+0000) 2024-02-19T20:24:49.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:49.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:50.194 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:49.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:50.194 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:49.942+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:50.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:49.942+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:50.451 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:24:50.203+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60211 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:24:50.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:50.200+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:50.201291+0000) 2024-02-19T20:24:50.452 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:50.199+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:50.200853+0000) 2024-02-19T20:24:50.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:50.200+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:50.201232+0000) 2024-02-19T20:24:50.453 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:50.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:50.453 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:50.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:50.453 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:50.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:50.453 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:50.224+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:50.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:50.667+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:50.668846+0000) 2024-02-19T20:24:50.760 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:50.454+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:50.455207+0000) 2024-02-19T20:24:50.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:50.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:50.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:50.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:50.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:50.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:50.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:50.590+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:51.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:50.763+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:50.764044+0000) 2024-02-19T20:24:51.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:50.911+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:51.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:50.911+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:51.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:50.911+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:51.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:50.911+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:51.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:51.227+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:51.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:51.227+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:51.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:51.227+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:51.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:51.228+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:51.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:51.456+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:51.457715+0000) 2024-02-19T20:24:51.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:51.548+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:51.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:51.548+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:51.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:51.548+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:51.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:51.549+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:51.923 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:51.671+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:51.672134+0000) 2024-02-19T20:24:51.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:51.765+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:51.766081+0000) 2024-02-19T20:24:52.221 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:51.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:52.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:51.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:52.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:51.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:52.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:51.925+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:52.516 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:52.459+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:52.460237+0000) 2024-02-19T20:24:52.516 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:52.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:52.516 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:52.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:52.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:52.224+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:52.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:52.224+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:52.793 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:52.674+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:52.675399+0000) 2024-02-19T20:24:52.794 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:52.767+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:52.768162+0000) 2024-02-19T20:24:52.794 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:52.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:52.794 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:52.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:52.794 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:52.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:52.794 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:52.573+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:53.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:52.914+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:53.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:52.914+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:53.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:52.914+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:53.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:52.915+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:53.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:53.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:53.385 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:53.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:53.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:53.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:53.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:53.235+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:53.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:53.461+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:53.462202+0000) 2024-02-19T20:24:53.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:53.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:53.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:53.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:53.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:53.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:53.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:53.623+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:53.930 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:53.677+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:53.678022+0000) 2024-02-19T20:24:53.930 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:53.769+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:53.770241+0000) 2024-02-19T20:24:54.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:53.932+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:54.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:53.932+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:54.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:53.932+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:54.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:53.933+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:54.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:54.462+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:54.463077+0000) 2024-02-19T20:24:54.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:54.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:54.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:54.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:54.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:54.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:54.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:54.277+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:55.016 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:54.678+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:54.678999+0000) 2024-02-19T20:24:55.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:54.638+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:55.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:54.638+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:55.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:54.638+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:55.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:54.639+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:55.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:54.770+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:54.771469+0000) 2024-02-19T20:24:55.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:54.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:55.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:54.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:55.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:54.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:55.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:54.885+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:55.385 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:24:55.211+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60252 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:24:55.386 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:55.208+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:55.209365+0000) 2024-02-19T20:24:55.386 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:55.207+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:55.208581+0000) 2024-02-19T20:24:55.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:55.209+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:55.209977+0000) 2024-02-19T20:24:55.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:55.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:55.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:55.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:55.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:55.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:55.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:55.318+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:55.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:55.462+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:55.463239+0000) 2024-02-19T20:24:55.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:55.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:55.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:55.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:55.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:55.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:55.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:55.618+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:56.016 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:55.678+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:55.679198+0000) 2024-02-19T20:24:56.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:55.770+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:55.771636+0000) 2024-02-19T20:24:56.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:55.902+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:56.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:55.902+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:56.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:55.902+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:56.017 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:55.903+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:56.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:56.464+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:56.465731+0000) 2024-02-19T20:24:56.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:56.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:56.567 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:56.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:56.568 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:56.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:56.568 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:56.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:56.568 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:56.347+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:56.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:56.680+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:56.681686+0000) 2024-02-19T20:24:56.936 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:56.773+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:56.774008+0000) 2024-02-19T20:24:56.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:56.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:56.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:56.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:56.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:56.570+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:57.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:56.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:57.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:56.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:57.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:56.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:57.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:56.938+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:57.586 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:57.466+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:57.467786+0000) 2024-02-19T20:24:57.586 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:57.324+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:57.586 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:57.324+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:57.587 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:57.324+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:57.587 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:57.324+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:57.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:57.682+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:57.683813+0000) 2024-02-19T20:24:57.891 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:57.775+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:57.776432+0000) 2024-02-19T20:24:57.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:57.588+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:57.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:57.588+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:57.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:57.588+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:57.892 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:57.589+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:58.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:57.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:58.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:57.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:58.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:57.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:58.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:57.894+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:58.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:58.469+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:58.469992+0000) 2024-02-19T20:24:58.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:58.339+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:58.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:58.339+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:58.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:58.339+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:58.583 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:58.340+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:58.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:58.685+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:58.685997+0000) 2024-02-19T20:24:58.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:58.584+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:58.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:58.584+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:58.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:58.584+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:58.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:58.585+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:58.926 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:58.777+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:58.778631+0000) 2024-02-19T20:24:58.927 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:58.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:58.927 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:58.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:58.927 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:58.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:59.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:58.929+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:59.576 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:59.469+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:59.470869+0000) 2024-02-19T20:24:59.577 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:59.310+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:59.577 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:59.310+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:59.577 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:59.310+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:24:59.577 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:59.311+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:24:59.884 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:24:59.685+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:59.686905+0000) 2024-02-19T20:24:59.885 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:59.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:59.885 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:59.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:59.885 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:59.579+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:24:59.885 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:24:59.580+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:24:59.885 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:59.779+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:59.780125+0000) 2024-02-19T20:24:59.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:59.887+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:59.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:59.887+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:24:59.886 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:59.887+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:00.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:24:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:24:59.887+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:00.467 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:25:00.218+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60292 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:25:00.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:00.216+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:00.217514+0000) 2024-02-19T20:25:00.468 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:00.215+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:00.216819+0000) 2024-02-19T20:25:00.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:00.216+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:00.217751+0000) 2024-02-19T20:25:00.468 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:00.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:00.469 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:00.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:00.469 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:00.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:00.469 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:00.309+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:00.766 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:00.686+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:00.687076+0000) 2024-02-19T20:25:00.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:00.470+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:00.471020+0000) 2024-02-19T20:25:00.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:00.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:00.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:00.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:00.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:00.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:00.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:00.592+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:01.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:00.779+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:00.780276+0000) 2024-02-19T20:25:01.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:00.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:01.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:00.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:01.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:00.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:01.113 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:00.927+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:01.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:01.287+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:01.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:01.287+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:01.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:01.287+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:01.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:01.287+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:01.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:01.471+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:01.472885+0000) 2024-02-19T20:25:01.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:01.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:01.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:01.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:01.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:01.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:01.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:01.637+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:01.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:01.688+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:01.689720+0000) 2024-02-19T20:25:01.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:01.781+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:01.782436+0000) 2024-02-19T20:25:01.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:01.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:01.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:01.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:01.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:01.958+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:02.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:01.958+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:02.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:02.474+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:02.475012+0000) 2024-02-19T20:25:02.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:02.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:02.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:02.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:02.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:02.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:02.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:02.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:02.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:02.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:02.669 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:02.305+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:02.669 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:02.306+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:02.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:02.671+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:02.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:02.690+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:02.691794+0000) 2024-02-19T20:25:02.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:02.783+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:02.784606+0000) 2024-02-19T20:25:02.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:02.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:02.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:02.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:02.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:02.972+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:03.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:02.973+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:03.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:03.476+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:03.477533+0000) 2024-02-19T20:25:03.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:03.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:03.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:03.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:03.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:03.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:03.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:03.313+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:04.001 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:03.693+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:03.694275+0000) 2024-02-19T20:25:04.002 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:03.787+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:03.788198+0000) 2024-02-19T20:25:04.002 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:03.684+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:04.002 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:03.684+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:04.002 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:03.684+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:04.002 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:03.685+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:04.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:04.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:04.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:04.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:04.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:04.017+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:04.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:04.018+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:04.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:04.477+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:04.478498+0000) 2024-02-19T20:25:04.691 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:04.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:04.692 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:04.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:04.692 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:04.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:04.692 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:04.347+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:04.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:04.694+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:04.695191+0000) 2024-02-19T20:25:04.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:04.788+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:04.789106+0000) 2024-02-19T20:25:04.981 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:04.698+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:04.981 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:04.698+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:04.981 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:04.698+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:04.981 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:04.699+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:05.230 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:25:05.226+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60333 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:25:05.231 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:05.222+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:05.223916+0000) 2024-02-19T20:25:05.231 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:05.223+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:05.224117+0000) 2024-02-19T20:25:05.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:04.983+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:05.231 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:04.983+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:05.232 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:04.983+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:05.232 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:04.983+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:05.232 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:05.223+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:05.224409+0000) 2024-02-19T20:25:05.516 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:05.477+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:05.478683+0000) 2024-02-19T20:25:05.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:05.373+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:05.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:05.373+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:05.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:05.373+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:05.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:05.374+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:06.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:05.694+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:05.695415+0000) 2024-02-19T20:25:06.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:05.788+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:05.789309+0000) 2024-02-19T20:25:06.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:05.736+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:06.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:05.736+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:06.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:05.737+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:06.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:05.738+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:06.308 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:06.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:06.308 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:06.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:06.308 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:06.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:06.308 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:06.026+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:06.694 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:06.479+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:06.480744+0000) 2024-02-19T20:25:06.694 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:06.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:06.694 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:06.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:06.694 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:06.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:06.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:06.326+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:06.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:06.697+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:06.697977+0000) 2024-02-19T20:25:06.999 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:06.791+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:06.792582+0000) 2024-02-19T20:25:06.999 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:07.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:06.999 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:07.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:06.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:06.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:06.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:06.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:07.000 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:06.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:07.000 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:06.706+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:07.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:07.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:07.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:07.001+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:07.685 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:07.482+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:07.482982+0000) 2024-02-19T20:25:07.685 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:07.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:07.686 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:07.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:07.686 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:07.298+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:07.686 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:07.298+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:08.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:07.699+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:07.700554+0000) 2024-02-19T20:25:08.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:07.794+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:07.795120+0000) 2024-02-19T20:25:08.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:07.688+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:08.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:07.688+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:08.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:07.688+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:08.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:07.689+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:08.308 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:08.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:08.308 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:08.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:08.308 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:08.037+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:08.308 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:08.037+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:08.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:08.485+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:08.486224+0000) 2024-02-19T20:25:08.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:08.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:08.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:08.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:08.694 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:08.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:08.694 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:08.333+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:08.993 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:08.701+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:08.702553+0000) 2024-02-19T20:25:08.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:08.796+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:08.797029+0000) 2024-02-19T20:25:08.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:08.696+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:08.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:08.696+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:08.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:08.696+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:08.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:08.697+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:09.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:08.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:09.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:08.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:09.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:08.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:09.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:08.997+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:09.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:09.486+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:09.487725+0000) 2024-02-19T20:25:09.645 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:09.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:09.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:09.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:09.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:09.340+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:09.646 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:09.341+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:10.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:09.702+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:09.703629+0000) 2024-02-19T20:25:10.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:09.797+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:09.798334+0000) 2024-02-19T20:25:10.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:09.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:10.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:09.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:10.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:09.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:10.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:09.648+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:10.308 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:10.230+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:10.231160+0000) 2024-02-19T20:25:10.308 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:25:10.233+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60373 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:25:10.309 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:10.230+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:10.231740+0000) 2024-02-19T20:25:10.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:10.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:10.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:10.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:10.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:10.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:10.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:10.028+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:10.310 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:10.230+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:10.231249+0000) 2024-02-19T20:25:10.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:10.486+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:10.487910+0000) 2024-02-19T20:25:10.606 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:10.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:10.606 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:10.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:10.606 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:10.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:10.606 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:10.328+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:10.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:10.702+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:10.703825+0000) 2024-02-19T20:25:10.959 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:10.608+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:10.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:10.608+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:10.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:10.608+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:10.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:10.609+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:10.960 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:10.797+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:10.798538+0000) 2024-02-19T20:25:11.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:10.985+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:11.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:10.985+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:11.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:10.985+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:11.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:10.985+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:11.590 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:11.490+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:11.491125+0000) 2024-02-19T20:25:11.590 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:11.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:11.590 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:11.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:11.590 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:11.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:11.590 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:11.325+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:12.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:11.704+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:11.705888+0000) 2024-02-19T20:25:12.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:11.800+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:11.800970+0000) 2024-02-19T20:25:12.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:11.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:12.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:11.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:12.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:11.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:12.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:11.593+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:12.308 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:12.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:12.308 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:12.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:12.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:12.025+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:12.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:12.025+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:12.606 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:12.492+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:12.493636+0000) 2024-02-19T20:25:12.606 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:12.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:12.607 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:12.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:12.607 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:12.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:12.607 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:12.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:12.607 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:12.346+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:12.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:12.707+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:12.708006+0000) 2024-02-19T20:25:12.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:12.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:12.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:12.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:12.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:12.610+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:12.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:12.802+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:12.803027+0000) 2024-02-19T20:25:13.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:12.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:13.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:12.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:13.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:12.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:13.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:12.988+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:13.577 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:13.494+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:13.495801+0000) 2024-02-19T20:25:13.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:13.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:13.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:13.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:13.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:13.355+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:13.578 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:13.356+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:14.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:13.709+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:13.710129+0000) 2024-02-19T20:25:14.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:13.804+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:13.805645+0000) 2024-02-19T20:25:14.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:13.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:14.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:13.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:14.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:13.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:14.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:13.612+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:14.308 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:14.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:14.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:14.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:14.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:14.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:14.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:14.030+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:14.565 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:14.495+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:14.496759+0000) 2024-02-19T20:25:14.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:14.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:14.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:14.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:14.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:14.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:14.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:14.336+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:15.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:14.710+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:14.711023+0000) 2024-02-19T20:25:15.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:14.805+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:14.806590+0000) 2024-02-19T20:25:15.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:14.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:15.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:14.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:15.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:14.568+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:15.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:14.570+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:15.290 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:25:15.240+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60414 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:25:15.291 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:15.237+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:15.238044+0000) 2024-02-19T20:25:15.291 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:15.237+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:15.238770+0000) 2024-02-19T20:25:15.292 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:15.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:15.292 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:15.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:15.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:15.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:15.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:15.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:15.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:15.032+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:15.293 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:15.032+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:15.293 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:15.237+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:15.238714+0000) 2024-02-19T20:25:15.575 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:15.293+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:15.575 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:15.294+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:15.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:15.495+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:15.496939+0000) 2024-02-19T20:25:15.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:15.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:15.576 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:15.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:15.576 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:15.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:16.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:15.805+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:15.806812+0000) 2024-02-19T20:25:16.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:15.578+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:16.016 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:15.710+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:15.711219+0000) 2024-02-19T20:25:16.308 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:16.272+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:16.309 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:16.272+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:16.309 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:16.272+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:16.309 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:16.273+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:16.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:16.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:16.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:16.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:16.310 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:16.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:16.310 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:16.037+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:16.616 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:16.498+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:16.499420+0000) 2024-02-19T20:25:16.617 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:16.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:16.617 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:16.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:17.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:16.712+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:16.713224+0000) 2024-02-19T20:25:17.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:16.807+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:16.808810+0000) 2024-02-19T20:25:17.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:16.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:17.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:16.620+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:17.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:17.225+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:17.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:17.225+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:17.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:17.225+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:17.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:17.226+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:17.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:17.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:17.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:17.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:17.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:17.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:17.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:17.077+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:17.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:17.502+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:17.502963+0000) 2024-02-19T20:25:17.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:17.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:17.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:17.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:17.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:17.616+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:17.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:17.617+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:18.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:17.715+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:17.716554+0000) 2024-02-19T20:25:18.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:17.809+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:17.810862+0000) 2024-02-19T20:25:18.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:18.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:18.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:18.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:18.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:18.274+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:18.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:18.274+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:18.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:18.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:18.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:18.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:18.388 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:18.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:18.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:18.071+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:18.656 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:18.504+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:18.505312+0000) 2024-02-19T20:25:18.656 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:18.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:18.656 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:18.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:18.656 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:18.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:19.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:18.660+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:19.016 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:18.718+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:18.719020+0000) 2024-02-19T20:25:19.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:18.811+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:18.812930+0000) 2024-02-19T20:25:19.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:19.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:19.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:19.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:19.293 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:19.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:19.293 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:19.029+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:19.657 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:19.505+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:19.506381+0000) 2024-02-19T20:25:19.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:19.295+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:19.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:19.295+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:19.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:19.295+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:19.658 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:19.296+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:19.993 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:19.719+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:19.720231+0000) 2024-02-19T20:25:19.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:19.660+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:19.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:19.660+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:19.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:19.660+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:19.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:19.661+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:19.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:19.812+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:19.813972+0000) 2024-02-19T20:25:19.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:19.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:19.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:19.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:19.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:19.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:20.245 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:20.244+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:20.245699+0000) 2024-02-19T20:25:20.245 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:20.244+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:20.245924+0000) 2024-02-19T20:25:20.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:19.996+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:20.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:20.244+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:20.245739+0000) 2024-02-19T20:25:20.502 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:25:20.248+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60454 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:25:20.502 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:20.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:20.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:20.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:20.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:20.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:20.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:20.278+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:20.765 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:20.719+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:20.720417+0000) 2024-02-19T20:25:20.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:20.505+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:20.506556+0000) 2024-02-19T20:25:20.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:20.657+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:20.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:20.657+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:20.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:20.657+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:20.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:20.658+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:21.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:20.813+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:20.814136+0000) 2024-02-19T20:25:21.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:20.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:21.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:20.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:21.115 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:20.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:21.115 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:20.988+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:21.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:21.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:21.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:21.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:21.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:21.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:21.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:21.236+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:21.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:21.508+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:21.509167+0000) 2024-02-19T20:25:21.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:21.723+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:21.724139+0000) 2024-02-19T20:25:21.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:21.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:21.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:21.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:21.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:21.706+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:21.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:21.707+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:21.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:21.815+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:21.816560+0000) 2024-02-19T20:25:21.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:21.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:21.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:21.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:21.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:21.987+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:22.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:21.988+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:22.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:22.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:22.508 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:22.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:22.508 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:22.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:22.508 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:22.259+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:22.765 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:22.726+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:22.727620+0000) 2024-02-19T20:25:22.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:22.510+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:22.511782+0000) 2024-02-19T20:25:22.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:22.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:22.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:22.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:22.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:22.742+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:22.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:22.743+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:23.114 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:22.818+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:22.819051+0000) 2024-02-19T20:25:23.115 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:23.034+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:23.115 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:23.034+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:23.115 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:23.034+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:23.115 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:23.034+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:23.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:23.231+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:23.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:23.231+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:23.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:23.231+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:23.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:23.232+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:23.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:23.513+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:23.514227+0000) 2024-02-19T20:25:24.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:23.728+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:23.729832+0000) 2024-02-19T20:25:24.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:23.820+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:23.821854+0000) 2024-02-19T20:25:24.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:23.768+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:24.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:23.768+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:24.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:23.768+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:24.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:23.769+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:24.309 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:24.196+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:24.309 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:24.196+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:24.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:24.196+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:24.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:24.197+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:24.310 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:24.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:24.310 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:24.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:24.310 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:24.048+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:24.310 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:24.049+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:24.726 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:24.514+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:24.515482+0000) 2024-02-19T20:25:25.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:24.729+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:24.730893+0000) 2024-02-19T20:25:25.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:24.821+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:24.822829+0000) 2024-02-19T20:25:25.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:24.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:25.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:24.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:25.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:24.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:25.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:24.803+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:25.258 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:25:25.255+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60495 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:25:25.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:25.251+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:25.252935+0000) 2024-02-19T20:25:25.259 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:25.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:25.259 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:25.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:25.259 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:25.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:25.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:25.189+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:25.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:25.252+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:25.253105+0000) 2024-02-19T20:25:25.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:25.007+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:25.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:25.007+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:25.260 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:25.007+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:25.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:25.008+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:25.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:25.252+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:25.253965+0000) 2024-02-19T20:25:25.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:25.514+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:25.515686+0000) 2024-02-19T20:25:25.765 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:25.730+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:25.731104+0000) 2024-02-19T20:25:26.115 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:25.821+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:26.115 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:25.821+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:26.115 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:25.821+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:26.115 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:25.822+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:26.116 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:25.822+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:25.823002+0000) 2024-02-19T20:25:26.116 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:26.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:26.116 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:26.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:26.116 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:26.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:26.116 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:26.024+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:26.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:26.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:26.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:26.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:26.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:26.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:26.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:26.189+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:26.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:26.517+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:26.518087+0000) 2024-02-19T20:25:26.998 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:26.733+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:26.734381+0000) 2024-02-19T20:25:26.998 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:26.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:26.998 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:26.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:26.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:26.847+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:26.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:26.848+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:26.999 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:26.824+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:26.825804+0000) 2024-02-19T20:25:26.999 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:27.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:26.999 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:27.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:27.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:27.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:27.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:27.002+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:27.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:27.231+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:27.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:27.231+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:27.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:27.231+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:27.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:27.232+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:27.732 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:27.519+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:27.520506+0000) 2024-02-19T20:25:28.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:27.735+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:27.736542+0000) 2024-02-19T20:25:28.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:27.827+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:27.828361+0000) 2024-02-19T20:25:28.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:27.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:28.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:27.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:28.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:27.968+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:28.016 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:27.968+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:28.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:27.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:28.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:27.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:28.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:27.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:28.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:27.851+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:28.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:28.253+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:28.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:28.253+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:28.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:28.253+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:28.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:28.254+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:28.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:28.522+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:28.523144+0000) 2024-02-19T20:25:29.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:28.739+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:28.740138+0000) 2024-02-19T20:25:29.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:28.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:29.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:28.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:29.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:28.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:29.013 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:28.852+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:29.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:28.829+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:28.830562+0000) 2024-02-19T20:25:29.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:29.213+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:29.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:29.213+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:29.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:29.213+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:29.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:29.214+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:29.269 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:29.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:29.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:29.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:29.270 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:29.016+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:29.737 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:29.524+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:29.525004+0000) 2024-02-19T20:25:29.993 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:29.740+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:29.741308+0000) 2024-02-19T20:25:29.993 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:29.830+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:29.831666+0000) 2024-02-19T20:25:29.993 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:29.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:29.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:29.843+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:29.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:29.843+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:29.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:29.843+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:29.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:29.844+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:30.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:29.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:30.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:29.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:30.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:29.997+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:30.515 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:25:30.262+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60535 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:25:30.516 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:30.260+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:30.261666+0000) 2024-02-19T20:25:30.516 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:30.259+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:30.260798+0000) 2024-02-19T20:25:30.516 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:30.250+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:30.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:30.250+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:30.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:30.250+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:30.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:30.251+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:30.517 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:30.260+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:30.261043+0000) 2024-02-19T20:25:30.820 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:30.740+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:30.741483+0000) 2024-02-19T20:25:30.820 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:30.525+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:30.526056+0000) 2024-02-19T20:25:31.115 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:30.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:31.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:30.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:31.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:30.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:31.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:30.876+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:31.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:30.830+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:30.831943+0000) 2024-02-19T20:25:31.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:31.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:31.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:31.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:31.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:31.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:31.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:31.042+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:31.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:31.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:31.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:31.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:31.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:31.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:31.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:31.259+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:31.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:31.527+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:31.528501+0000) 2024-02-19T20:25:32.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:31.743+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:31.744827+0000) 2024-02-19T20:25:32.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:31.833+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:31.834030+0000) 2024-02-19T20:25:32.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:31.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:32.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:31.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:32.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:31.828+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:32.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:31.830+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:32.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:32.244+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:32.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:32.244+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:32.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:32.244+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:32.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:32.245+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:32.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:32.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:32.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:32.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:32.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:32.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:32.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:32.068+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:32.743 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:32.530+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:32.531120+0000) 2024-02-19T20:25:33.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:32.747+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:32.748376+0000) 2024-02-19T20:25:33.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:32.835+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:32.836041+0000) 2024-02-19T20:25:33.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:32.840+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:33.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:32.840+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:33.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:32.840+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:33.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:32.841+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:33.388 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:33.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:33.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:33.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:33.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:33.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:33.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:33.259+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:33.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:33.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:33.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:33.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:33.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:33.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:33.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:33.090+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:33.746 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:33.532+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:33.533246+0000) 2024-02-19T20:25:34.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:33.749+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:33.750588+0000) 2024-02-19T20:25:34.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:33.837+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:33.838142+0000) 2024-02-19T20:25:34.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:33.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:34.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:33.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:34.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:33.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:34.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:33.799+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:34.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:34.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:34.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:34.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:34.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:34.118+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:34.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:34.119+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:34.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:34.242+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:34.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:34.242+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:34.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:34.242+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:34.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:34.242+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:34.747 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:34.533+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:34.534781+0000) 2024-02-19T20:25:35.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:34.751+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:34.752211+0000) 2024-02-19T20:25:35.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:34.838+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:34.839155+0000) 2024-02-19T20:25:35.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:34.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:35.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:34.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:35.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:34.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:35.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:34.808+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:35.385 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:25:35.269+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60576 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:25:35.385 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:35.266+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:35.267551+0000) 2024-02-19T20:25:35.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:35.209+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:35.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:35.209+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:35.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:35.209+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:35.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:35.209+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:35.386 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:35.266+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:35.267088+0000) 2024-02-19T20:25:35.386 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:35.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:35.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:35.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:35.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:35.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:35.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:35.131+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:35.387 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:35.267+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:35.268007+0000) 2024-02-19T20:25:35.748 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:35.534+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:35.535028+0000) 2024-02-19T20:25:36.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:35.751+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:35.752544+0000) 2024-02-19T20:25:36.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:35.838+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:35.839330+0000) 2024-02-19T20:25:36.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:35.801+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:36.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:35.801+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:36.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:35.801+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:36.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:35.802+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:36.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:36.221+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:36.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:36.221+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:36.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:36.221+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:36.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:36.221+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:36.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:36.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:36.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:36.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:36.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:36.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:36.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:36.176+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:36.751 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:36.536+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:36.537065+0000) 2024-02-19T20:25:37.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:36.754+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:36.755813+0000) 2024-02-19T20:25:37.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:36.840+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:36.841285+0000) 2024-02-19T20:25:37.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:36.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:37.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:36.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:37.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:36.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:37.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:36.760+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:37.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:37.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:37.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:37.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:37.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:37.258+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:37.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:37.258+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:37.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:37.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:37.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:37.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:37.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:37.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:37.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:37.156+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:37.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:37.537+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:37.539262+0000) 2024-02-19T20:25:37.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:37.726+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:38.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:37.757+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:37.758388+0000) 2024-02-19T20:25:38.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:37.842+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:37.843482+0000) 2024-02-19T20:25:38.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:37.726+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:38.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:37.726+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:38.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:37.727+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:38.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:38.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:38.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:38.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:38.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:38.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:38.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:38.221+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:38.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:38.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:38.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:38.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:38.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:38.145+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:38.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:38.145+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:38.734 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:38.540+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:38.541454+0000) 2024-02-19T20:25:39.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:38.759+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:38.760532+0000) 2024-02-19T20:25:39.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:38.844+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:38.845546+0000) 2024-02-19T20:25:39.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:38.738+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:39.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:38.738+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:39.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:38.738+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:39.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:38.739+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:39.353 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:39.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:39.353 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:39.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:39.353 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:39.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:39.353 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:39.229+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:39.353 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:39.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:39.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:39.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:39.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:39.097+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:39.354 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:39.098+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:39.726 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:39.541+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:39.542693+0000) 2024-02-19T20:25:40.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:39.760+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:39.761777+0000) 2024-02-19T20:25:40.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:39.845+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:39.846581+0000) 2024-02-19T20:25:40.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:39.730+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:40.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:39.730+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:40.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:39.730+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:40.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:39.731+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:40.389 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:25:40.277+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60616 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:25:40.389 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:40.274+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:40.275414+0000) 2024-02-19T20:25:40.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:40.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:40.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:40.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:40.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:40.188+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:40.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:40.189+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:40.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:40.274+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:40.275271+0000) 2024-02-19T20:25:40.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:40.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:40.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:40.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:40.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:40.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:40.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:40.074+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:40.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:40.274+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:40.275910+0000) 2024-02-19T20:25:40.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:40.541+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:40.542897+0000) 2024-02-19T20:25:40.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:40.696+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:40.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:40.696+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:41.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:40.761+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:40.761971+0000) 2024-02-19T20:25:41.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:40.845+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:40.846808+0000) 2024-02-19T20:25:41.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:40.696+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:41.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:40.697+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:41.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:41.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:41.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:41.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:41.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:41.096+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:41.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:41.097+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:41.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:41.183+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:41.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:41.183+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:41.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:41.183+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:41.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:41.185+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:41.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:41.545+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:41.546443+0000) 2024-02-19T20:25:42.014 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:41.763+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:41.764119+0000) 2024-02-19T20:25:42.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:41.847+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:41.848794+0000) 2024-02-19T20:25:42.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:41.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:42.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:41.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:42.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:41.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:42.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:41.726+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:42.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:42.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:42.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:42.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:42.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:42.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:42.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:42.150+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:42.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:42.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:42.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:42.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:42.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:42.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:42.393 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:42.082+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:42.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:42.547+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:42.548724+0000) 2024-02-19T20:25:43.014 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:42.849+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:42.850797+0000) 2024-02-19T20:25:43.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:42.765+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:42.766563+0000) 2024-02-19T20:25:43.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:42.693+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:43.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:42.693+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:43.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:42.693+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:43.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:42.694+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:43.311 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:43.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:43.311 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:43.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:43.311 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:43.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:43.311 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:43.164+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:43.312 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:43.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:43.312 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:43.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:43.312 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:43.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:43.312 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:43.056+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:43.576 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:43.550+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:43.551222+0000) 2024-02-19T20:25:43.830 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:43.767+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:43.768824+0000) 2024-02-19T20:25:43.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:43.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:43.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:43.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:43.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:43.670+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:43.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:43.671+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:44.116 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:43.851+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:43.852874+0000) 2024-02-19T20:25:44.116 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:44.080+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:44.116 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:44.080+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:44.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:44.080+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:44.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:44.080+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:44.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:44.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:44.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:44.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:44.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:44.189+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:44.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:44.190+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:44.691 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:44.551+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:44.552868+0000) 2024-02-19T20:25:45.014 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:44.768+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:44.769955+0000) 2024-02-19T20:25:45.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:44.853+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:44.854072+0000) 2024-02-19T20:25:45.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:44.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:45.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:44.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:45.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:44.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:45.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:44.696+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:45.389 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:25:45.284+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60657 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:25:45.390 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:45.281+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:45.282836+0000) 2024-02-19T20:25:45.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:45.151+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:45.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:45.151+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:45.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:45.151+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:45.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:45.151+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:45.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:45.282+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:45.283535+0000) 2024-02-19T20:25:45.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:45.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:45.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:45.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:45.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:45.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:45.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:45.110+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:45.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:45.280+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:45.281895+0000) 2024-02-19T20:25:45.704 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:45.552+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:45.553053+0000) 2024-02-19T20:25:46.014 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:45.769+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:45.770163+0000) 2024-02-19T20:25:46.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:45.853+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:45.854255+0000) 2024-02-19T20:25:46.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:45.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:46.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:45.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:46.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:45.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:46.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:45.709+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:46.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:46.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:46.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:46.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:46.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:46.072+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:46.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:46.073+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:46.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:46.165+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:46.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:46.165+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:46.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:46.165+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:46.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:46.166+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:46.680 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:46.554+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:46.555611+0000) 2024-02-19T20:25:47.014 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:46.855+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:46.856607+0000) 2024-02-19T20:25:47.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:46.771+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:46.772122+0000) 2024-02-19T20:25:47.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:46.684+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:47.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:46.684+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:47.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:46.684+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:47.016 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:46.685+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:47.389 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:47.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:47.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:47.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:47.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:47.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:47.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:47.119+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:47.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:47.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:47.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:47.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:47.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:47.172+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:47.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:47.172+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:47.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:47.557+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:47.558863+0000) 2024-02-19T20:25:47.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:47.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:47.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:47.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:47.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:47.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:47.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:47.637+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:48.107 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:47.773+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:47.774197+0000) 2024-02-19T20:25:48.107 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:47.858+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:47.859373+0000) 2024-02-19T20:25:48.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:48.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:48.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:48.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:48.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:48.111+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:48.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:48.111+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:48.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:48.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:48.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:48.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:48.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:48.216+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:48.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:48.216+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:48.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:48.559+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:48.560888+0000) 2024-02-19T20:25:48.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:48.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:48.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:48.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:48.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:48.625+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:48.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:48.626+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:49.116 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:48.776+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:48.777154+0000) 2024-02-19T20:25:49.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:48.860+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:48.861955+0000) 2024-02-19T20:25:49.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:49.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:49.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:49.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:49.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:49.166+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:49.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:49.167+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:49.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:49.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:49.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:49.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:49.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:49.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:49.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:49.131+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:49.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:49.560+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:49.561961+0000) 2024-02-19T20:25:49.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:49.675+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:49.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:49.675+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:49.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:49.675+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:50.014 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:49.676+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:50.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:49.862+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:49.863000+0000) 2024-02-19T20:25:50.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:49.777+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:49.778521+0000) 2024-02-19T20:25:50.390 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:25:50.291+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60697 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:25:50.391 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:50.288+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:50.289543+0000) 2024-02-19T20:25:50.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:50.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:50.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:50.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:50.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:50.130+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:50.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:50.130+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:50.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:50.289+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:50.290606+0000) 2024-02-19T20:25:50.392 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:50.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:50.393 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:50.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:50.393 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:50.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:50.393 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:50.150+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:50.393 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:50.289+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:50.290095+0000) 2024-02-19T20:25:50.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:50.561+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:50.562144+0000) 2024-02-19T20:25:50.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:50.634+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:50.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:50.634+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:50.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:50.634+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:50.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:50.635+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:51.117 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:50.777+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:50.778743+0000) 2024-02-19T20:25:51.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:50.862+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:50.863166+0000) 2024-02-19T20:25:51.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:51.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:51.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:51.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:51.390 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:51.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:51.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:51.137+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:51.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:51.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:51.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:51.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:51.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:51.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:51.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:51.176+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:51.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:51.563+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:51.564151+0000) 2024-02-19T20:25:51.766 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:51.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:51.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:51.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:51.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:51.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:51.767 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:51.619+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:52.117 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:51.779+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:51.780836+0000) 2024-02-19T20:25:52.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:51.865+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:51.866484+0000) 2024-02-19T20:25:52.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:52.180+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:52.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:52.180+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:52.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:52.180+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:52.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:52.181+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:52.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:52.162+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:52.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:52.162+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:52.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:52.162+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:52.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:52.162+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:52.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:52.565+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:52.566142+0000) 2024-02-19T20:25:52.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:52.620+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:52.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:52.620+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:52.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:52.620+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:52.765 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:52.621+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:53.117 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:52.781+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:52.782863+0000) 2024-02-19T20:25:53.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:52.868+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:52.869864+0000) 2024-02-19T20:25:53.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:53.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:53.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:53.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:53.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:53.176+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:53.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:53.177+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:53.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:53.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:53.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:53.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:53.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:53.136+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:53.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:53.137+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:53.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:53.567+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:53.568201+0000) 2024-02-19T20:25:53.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:53.644+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:53.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:53.785+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:53.786415+0000) 2024-02-19T20:25:53.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:53.871+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:53.872353+0000) 2024-02-19T20:25:53.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:53.644+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:53.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:53.644+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:53.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:53.645+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:54.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:54.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:54.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:54.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:54.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:54.158+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:54.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:54.159+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:54.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:54.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:54.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:54.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:54.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:54.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:54.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:54.115+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:54.840 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:54.786+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:54.787535+0000) 2024-02-19T20:25:54.840 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:54.568+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:54.569875+0000) 2024-02-19T20:25:54.840 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:54.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:54.840 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:54.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:54.840 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:54.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:54.841 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:54.675+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:55.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:54.872+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:54.873686+0000) 2024-02-19T20:25:55.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:55.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:55.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:55.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:55.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:55.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:55.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:55.088+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:55.390 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:25:55.298+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60740 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:25:55.391 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:55.296+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:55.297359+0000) 2024-02-19T20:25:55.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:55.295+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:55.296858+0000) 2024-02-19T20:25:55.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:55.167+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:55.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:55.167+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:55.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:55.167+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:55.392 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:55.168+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:55.392 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:55.296+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:55.297206+0000) 2024-02-19T20:25:55.662 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:55.569+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:55.570073+0000) 2024-02-19T20:25:55.663 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:55.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:55.663 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:55.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:55.663 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:55.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:56.014 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:55.667+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:56.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:55.786+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:55.787764+0000) 2024-02-19T20:25:56.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:55.872+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:55.873895+0000) 2024-02-19T20:25:56.312 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:56.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:56.312 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:56.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:56.312 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:56.138+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:56.312 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:56.138+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:56.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:56.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:56.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:56.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:56.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:56.046+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:56.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:56.047+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:56.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:56.789+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:56.789994+0000) 2024-02-19T20:25:56.843 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:56.571+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:56.572405+0000) 2024-02-19T20:25:56.843 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:56.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:56.843 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:56.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:56.844 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:56.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:56.844 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:56.697+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:57.108 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:56.875+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:56.876096+0000) 2024-02-19T20:25:57.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:57.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:57.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:57.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:57.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:57.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:57.109 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:57.030+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:57.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:57.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:57.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:57.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:57.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:57.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:57.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:57.113+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:57.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:57.573+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:57.574747+0000) 2024-02-19T20:25:57.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:57.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:57.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:57.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:57.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:57.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:58.014 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:57.791+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:57.792214+0000) 2024-02-19T20:25:58.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:57.726+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:58.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:57.877+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:57.878224+0000) 2024-02-19T20:25:58.390 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:58.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:58.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:58.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:58.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:58.086+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:58.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:58.087+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:58.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:58.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:58.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:58.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:58.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:58.075+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:58.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:58.076+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:58.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:58.577+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:58.577995+0000) 2024-02-19T20:25:59.014 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:58.793+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:58.794579+0000) 2024-02-19T20:25:59.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:58.879+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:58.880181+0000) 2024-02-19T20:25:59.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:58.692+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:59.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:58.692+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:59.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:58.692+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:59.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:58.693+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:59.312 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:59.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:59.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:59.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:59.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:59.051+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:25:59.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:59.052+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:25:59.313 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:59.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:59.313 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:59.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:59.313 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:59.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:25:59.314 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:59.051+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:25:59.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:25:59.795+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:59.796292+0000) 2024-02-19T20:25:59.844 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:59.578+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:59.579031+0000) 2024-02-19T20:25:59.844 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:59.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:59.844 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:59.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:59.844 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:59.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:25:59.844 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:25:59.722+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:00.117 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:00.067+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:00.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:00.067+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:00.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:00.067+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:00.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:00.068+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:00.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:25:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:25:59.880+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:59.881271+0000) 2024-02-19T20:26:00.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:00.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:00.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:00.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:00.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:00.077+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:00.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:00.077+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:00.391 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:26:00.306+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60779 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:26:00.391 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:00.303+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:00.304223+0000) 2024-02-19T20:26:00.391 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:00.303+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:00.304301+0000) 2024-02-19T20:26:00.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:00.303+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:00.304469+0000) 2024-02-19T20:26:00.723 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:00.578+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:00.579247+0000) 2024-02-19T20:26:01.014 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:00.795+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:00.796452+0000) 2024-02-19T20:26:01.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:00.880+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:00.881486+0000) 2024-02-19T20:26:01.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:00.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:01.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:00.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:01.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:00.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:01.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:00.728+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:01.312 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:01.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:01.313 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:01.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:01.313 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:01.062+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:01.313 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:01.063+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:01.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:01.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:01.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:01.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:01.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:01.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:01.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:01.083+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:01.846 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:01.798+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:01.799029+0000) 2024-02-19T20:26:01.846 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:01.581+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:01.582786+0000) 2024-02-19T20:26:01.847 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:01.683+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:01.847 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:01.683+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:01.847 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:01.683+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:01.847 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:01.684+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:02.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:02.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:02.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:02.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:02.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:02.063+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:02.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:02.064+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:02.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:01.884+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:01.885016+0000) 2024-02-19T20:26:02.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:02.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:02.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:02.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:02.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:02.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:02.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:02.072+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:02.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:02.583+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:02.584834+0000) 2024-02-19T20:26:02.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:02.728+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:02.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:02.728+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:02.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:02.728+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:03.014 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:02.729+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:03.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:02.800+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:02.801029+0000) 2024-02-19T20:26:03.015 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:02.887+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:02.888346+0000) 2024-02-19T20:26:03.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:03.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:03.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:03.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:03.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:03.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:03.313 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:03.083+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:03.314 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:03.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:03.314 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:03.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:03.314 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:03.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:03.314 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:03.052+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:03.847 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:03.802+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:03.803000+0000) 2024-02-19T20:26:03.847 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:03.587+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:03.588156+0000) 2024-02-19T20:26:03.848 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:03.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:03.848 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:03.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:03.848 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:03.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:03.848 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:03.754+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:04.118 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:03.889+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:03.890392+0000) 2024-02-19T20:26:04.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:04.012+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:04.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:04.012+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:04.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:04.012+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:04.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:04.013+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:04.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:04.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:04.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:04.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:04.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:04.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:04.392 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:04.125+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:04.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:04.588+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:04.589217+0000) 2024-02-19T20:26:05.000 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:04.890+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:04.891631+0000) 2024-02-19T20:26:05.001 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:04.803+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:04.804026+0000) 2024-02-19T20:26:05.002 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:04.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:05.002 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:04.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:05.002 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:04.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:05.002 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:04.721+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:05.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:05.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:05.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:05.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:05.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:05.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:05.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:05.005+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:05.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:05.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:05.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:05.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:05.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:05.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:05.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:05.130+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:05.584 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:26:05.313+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60821 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:26:05.585 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:05.311+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:05.312111+0000) 2024-02-19T20:26:05.585 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:05.310+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:05.311284+0000) 2024-02-19T20:26:05.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:05.310+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:05.311507+0000) 2024-02-19T20:26:05.847 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:05.803+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:05.804227+0000) 2024-02-19T20:26:05.848 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:05.588+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:05.589380+0000) 2024-02-19T20:26:05.848 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:05.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:05.848 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:05.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:05.848 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:05.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:05.848 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:05.760+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:06.105 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:06.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:06.105 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:06.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:06.105 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:06.029+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:06.105 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:06.029+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:06.105 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:05.890+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:05.891835+0000) 2024-02-19T20:26:06.106 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:06.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:06.106 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:06.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:06.106 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:06.109+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:06.391 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:06.109+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:06.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:06.591+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:06.591984+0000) 2024-02-19T20:26:07.001 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:06.805+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:06.806436+0000) 2024-02-19T20:26:07.001 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:06.744+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:07.001 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:06.744+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:07.001 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:06.744+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:07.002 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:06.745+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:07.002 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:06.894+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:06.895394+0000) 2024-02-19T20:26:07.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:07.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:07.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:07.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:07.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:07.005+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:07.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:07.006+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:07.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:07.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:07.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:07.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:07.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:07.099+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:07.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:07.100+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:07.850 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:07.807+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:07.808429+0000) 2024-02-19T20:26:07.851 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:07.593+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:07.594020+0000) 2024-02-19T20:26:07.851 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:07.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:07.851 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:07.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:07.851 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:07.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:07.851 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:07.766+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:08.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:07.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:08.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:07.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:08.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:07.967+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:08.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:07.969+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:08.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:07.897+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:07.898805+0000) 2024-02-19T20:26:08.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:08.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:08.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:08.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:08.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:08.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:08.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:08.079+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:08.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:08.595+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:08.596763+0000) 2024-02-19T20:26:09.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:08.810+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:08.811757+0000) 2024-02-19T20:26:09.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:08.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:09.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:08.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:09.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:08.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:09.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:08.939+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:09.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:08.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:09.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:08.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:09.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:08.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:09.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:08.810+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:09.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:08.900+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:08.901410+0000) 2024-02-19T20:26:09.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:09.034+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:09.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:09.034+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:09.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:09.034+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:09.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:09.034+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:09.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:09.597+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:09.598195+0000) 2024-02-19T20:26:10.014 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:09.902+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:09.903130+0000) 2024-02-19T20:26:10.014 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:09.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:10.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:09.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:10.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:09.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:10.015 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:09.821+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:10.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:09.812+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:09.813426+0000) 2024-02-19T20:26:10.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:09.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:10.015 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:09.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:10.016 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:09.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:10.016 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:09.913+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:10.310 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:10.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:10.310 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:10.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:10.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:10.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:10.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:10.042+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:10.593 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:26:10.322+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60860 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:26:10.593 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:10.318+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:10.319512+0000) 2024-02-19T20:26:10.594 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:10.317+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:10.318691+0000) 2024-02-19T20:26:10.594 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:10.317+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:10.318879+0000) 2024-02-19T20:26:10.851 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:10.812+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:10.813624+0000) 2024-02-19T20:26:10.851 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:10.597+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:10.598387+0000) 2024-02-19T20:26:10.851 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:10.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:10.851 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:10.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:10.852 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:10.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:10.852 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:10.792+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:11.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:10.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:11.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:10.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:11.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:10.919+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:11.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:10.919+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:11.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:10.902+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:10.903344+0000) 2024-02-19T20:26:11.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:11.006+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:11.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:11.006+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:11.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:11.006+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:11.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:11.007+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:11.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:11.599+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:11.600822+0000) 2024-02-19T20:26:12.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:11.832+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:12.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:11.832+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:12.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:11.832+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:12.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:11.833+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:12.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:11.904+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:11.905398+0000) 2024-02-19T20:26:12.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:11.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:12.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:11.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:12.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:11.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:12.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:11.991+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:12.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:11.815+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:11.816938+0000) 2024-02-19T20:26:12.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:11.928+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:12.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:11.928+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:12.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:11.928+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:12.122 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:11.929+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:12.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:12.601+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:12.602878+0000) 2024-02-19T20:26:13.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:12.819+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:12.820339+0000) 2024-02-19T20:26:13.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:12.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:13.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:12.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:13.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:12.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:13.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:12.895+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:13.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:12.795+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:13.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:12.795+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:13.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:12.795+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:13.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:12.795+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:13.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:12.906+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:12.907441+0000) 2024-02-19T20:26:13.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:12.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:13.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:12.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:13.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:12.995+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:13.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:12.996+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:13.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:13.603+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:13.604917+0000) 2024-02-19T20:26:14.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:13.827+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:14.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:13.827+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:14.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:13.827+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:14.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:13.828+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:14.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:13.821+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:13.822352+0000) 2024-02-19T20:26:14.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:13.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:14.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:13.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:14.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:13.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:14.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:13.912+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:14.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:13.908+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:13.909588+0000) 2024-02-19T20:26:14.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:13.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:14.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:13.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:14.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:13.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:14.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:13.988+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:14.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:14.605+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:14.606667+0000) 2024-02-19T20:26:15.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:14.784+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:15.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:14.784+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:15.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:14.784+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:15.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:14.785+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:15.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:14.910+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:14.911024+0000) 2024-02-19T20:26:15.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:15.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:15.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:15.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:15.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:15.013+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:15.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:14.822+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:14.823385+0000) 2024-02-19T20:26:15.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:14.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:15.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:14.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:15.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:14.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:15.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:14.921+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:15.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:15.013+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:15.601 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:26:15.329+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60902 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:26:15.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:15.327+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:15.328705+0000) 2024-02-19T20:26:15.602 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:15.326+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:15.327078+0000) 2024-02-19T20:26:15.602 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:15.326+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:15.327286+0000) 2024-02-19T20:26:15.855 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:15.822+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:15.823606+0000) 2024-02-19T20:26:15.855 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:15.605+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:15.606896+0000) 2024-02-19T20:26:15.855 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:15.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:15.855 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:15.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:15.855 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:15.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:15.856 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:15.809+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:16.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:15.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:16.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:15.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:16.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:15.885+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:16.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:15.886+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:16.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:15.910+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:15.911214+0000) 2024-02-19T20:26:16.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:16.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:16.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:16.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:16.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:16.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:16.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:16.008+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:16.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:16.608+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:16.609448+0000) 2024-02-19T20:26:17.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:16.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:17.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:16.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:17.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:16.774+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:17.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:16.775+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:17.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:16.912+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:16.913141+0000) 2024-02-19T20:26:17.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:16.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:17.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:16.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:17.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:16.986+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:17.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:16.986+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:17.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:16.824+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:16.825817+0000) 2024-02-19T20:26:17.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:16.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:17.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:16.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:17.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:16.889+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:17.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:16.890+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:17.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:17.611+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:17.612813+0000) 2024-02-19T20:26:18.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:17.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:18.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:17.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:18.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:17.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:18.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:17.812+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:18.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:17.826+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:17.827939+0000) 2024-02-19T20:26:18.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:17.865+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:18.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:17.865+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:18.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:17.865+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:18.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:17.866+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:18.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:17.914+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:17.915846+0000) 2024-02-19T20:26:18.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:17.983+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:18.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:17.983+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:18.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:17.983+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:18.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:17.984+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:18.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:18.614+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:18.615063+0000) 2024-02-19T20:26:19.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:18.844+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:19.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:18.844+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:19.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:18.844+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:19.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:18.845+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:19.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:18.829+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:18.830428+0000) 2024-02-19T20:26:19.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:18.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:19.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:18.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:19.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:18.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:19.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:18.860+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:19.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:18.917+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:18.918071+0000) 2024-02-19T20:26:19.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:18.970+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:19.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:18.970+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:19.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:18.970+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:19.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:18.971+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:19.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:19.615+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:19.616075+0000) 2024-02-19T20:26:20.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:19.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:20.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:19.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:20.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:19.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:20.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:19.803+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:20.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:19.831+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:19.832170+0000) 2024-02-19T20:26:20.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:19.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:20.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:19.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:20.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:19.866+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:20.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:19.867+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:20.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:19.918+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:19.919106+0000) 2024-02-19T20:26:20.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:19.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:20.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:19.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:20.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:19.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:20.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:19.998+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:20.392 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:26:20.336+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60941 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:26:20.392 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:20.333+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:20.334764+0000) 2024-02-19T20:26:20.393 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:20.333+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:20.334175+0000) 2024-02-19T20:26:20.393 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:20.333+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:20.334817+0000) 2024-02-19T20:26:20.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:20.615+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:20.616264+0000) 2024-02-19T20:26:21.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:20.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:21.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:20.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:21.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:20.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:21.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:20.847+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:21.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:20.831+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:20.832349+0000) 2024-02-19T20:26:21.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:20.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:21.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:20.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:21.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:20.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:21.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:20.855+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:21.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:20.918+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:20.919281+0000) 2024-02-19T20:26:21.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:21.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:21.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:21.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:21.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:21.043+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:21.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:21.043+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:21.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:21.617+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:21.618954+0000) 2024-02-19T20:26:22.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:21.920+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:21.921313+0000) 2024-02-19T20:26:22.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:22.040+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:22.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:22.040+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:22.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:22.040+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:22.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:22.041+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:22.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:21.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:22.121 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:21.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:22.121 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:21.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:22.121 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:21.871+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:22.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:21.834+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:21.835025+0000) 2024-02-19T20:26:22.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:21.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:22.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:21.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:22.122 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:21.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:22.122 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:21.852+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:22.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:22.621+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:22.622262+0000) 2024-02-19T20:26:23.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:22.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:23.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:22.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:23.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:22.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:23.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:22.893+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:23.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:22.836+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:22.837029+0000) 2024-02-19T20:26:23.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:22.858+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:23.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:22.858+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:23.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:22.858+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:23.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:22.859+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:23.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:22.923+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:22.924655+0000) 2024-02-19T20:26:23.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:22.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:23.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:22.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:23.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:22.996+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:23.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:22.996+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:23.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:23.623+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:23.624949+0000) 2024-02-19T20:26:24.119 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:23.838+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:23.838997+0000) 2024-02-19T20:26:24.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:23.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:24.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:23.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:24.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:23.894+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:24.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:23.894+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:24.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:23.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:24.121 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:23.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:24.121 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:23.845+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:24.121 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:23.845+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:24.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:23.926+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:23.926799+0000) 2024-02-19T20:26:24.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:24.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:24.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:24.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:24.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:24.023+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:24.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:24.023+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:24.685 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:24.626+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:24.626597+0000) 2024-02-19T20:26:24.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:24.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:24.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:24.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:24.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:24.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:24.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:24.844+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:24.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:24.926+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:24.928009+0000) 2024-02-19T20:26:24.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:24.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:24.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:24.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:24.995 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:24.997+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:24.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:24.839+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:24.840673+0000) 2024-02-19T20:26:24.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:24.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:24.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:24.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:24.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:24.849+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:24.995 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:24.850+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:25.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:24.997+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:25.621 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:26:25.343+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 60983 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:26:25.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:25.340+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:25.342221+0000) 2024-02-19T20:26:25.622 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:25.339+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:25.341652+0000) 2024-02-19T20:26:25.622 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:25.339+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:25.341625+0000) 2024-02-19T20:26:25.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:25.839+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:25.840899+0000) 2024-02-19T20:26:25.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:25.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:25.875 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:25.624+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:25.626855+0000) 2024-02-19T20:26:25.875 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:25.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:25.876 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:25.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:25.876 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:25.846+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:25.876 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:25.846+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:26.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:25.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:26.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:25.878+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:26.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:25.878+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:26.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:25.927+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:25.928266+0000) 2024-02-19T20:26:26.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:26.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:26.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:26.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:26.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:26.024+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:26.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:26.025+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:26.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:26.628+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:26.629194+0000) 2024-02-19T20:26:26.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:26.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:26.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:26.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:26.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:26.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:26.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:26.810+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:26.926 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:26.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:26.926 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:26.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:26.926 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:26.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:26.926 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:26.838+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:26.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:26.842+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:26.843060+0000) 2024-02-19T20:26:27.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:26.929+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:26.930843+0000) 2024-02-19T20:26:27.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:26.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:27.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:26.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:27.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:26.988+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:27.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:26.989+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:27.928 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:27.630+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:27.631306+0000) 2024-02-19T20:26:27.928 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:27.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:27.928 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:27.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:27.928 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:27.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:27.928 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:27.800+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:27.929 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:27.845+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:27.846587+0000) 2024-02-19T20:26:27.929 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:27.858+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:27.929 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:27.858+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:27.929 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:27.858+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:27.929 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:27.859+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:28.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:27.932+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:27.933416+0000) 2024-02-19T20:26:28.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:27.971+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:28.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:27.971+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:28.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:27.971+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:28.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:27.971+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:28.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:28.848+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:28.849161+0000) 2024-02-19T20:26:28.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:28.632+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:28.633918+0000) 2024-02-19T20:26:28.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:28.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:28.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:28.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:28.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:28.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:28.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:28.771+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:29.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:28.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:29.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:28.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:29.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:28.897+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:29.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:28.898+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:29.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:28.935+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:28.936055+0000) 2024-02-19T20:26:29.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:28.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:29.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:28.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:29.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:28.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:29.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:28.961+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:29.910 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:29.849+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:29.850249+0000) 2024-02-19T20:26:29.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:29.634+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:29.635477+0000) 2024-02-19T20:26:29.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:29.736+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:29.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:29.736+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:29.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:29.736+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:29.911 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:29.737+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:30.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:29.914+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:30.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:29.914+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:30.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:29.914+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:30.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:29.915+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:30.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:29.936+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:29.937490+0000) 2024-02-19T20:26:30.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:29.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:30.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:29.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:30.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:29.982+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:30.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:29.982+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:30.630 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:26:30.351+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61022 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:26:30.630 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:30.348+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:30.349313+0000) 2024-02-19T20:26:30.631 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:30.348+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:30.349742+0000) 2024-02-19T20:26:30.631 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:30.347+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:30.348834+0000) 2024-02-19T20:26:30.931 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:30.849+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:30.850451+0000) 2024-02-19T20:26:30.931 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:30.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:30.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:30.634+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:30.635687+0000) 2024-02-19T20:26:30.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:30.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:30.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:30.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:30.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:30.701+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:30.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:30.702+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:31.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:30.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:31.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:30.936+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:31.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:30.937+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:31.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:30.936+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:30.937701+0000) 2024-02-19T20:26:31.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:31.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:31.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:31.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:31.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:31.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:31.267 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:31.027+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:31.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:31.851+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:31.852532+0000) 2024-02-19T20:26:31.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:31.637+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:31.638138+0000) 2024-02-19T20:26:31.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:31.676+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:31.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:31.676+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:31.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:31.676+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:31.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:31.676+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:32.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:31.951+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:32.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:31.951+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:32.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:31.951+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:32.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:31.952+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:32.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:31.940+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:31.941271+0000) 2024-02-19T20:26:32.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:31.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:32.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:31.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:32.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:31.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:32.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:31.992+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:32.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:32.855+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:32.856029+0000) 2024-02-19T20:26:32.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:32.640+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:32.641464+0000) 2024-02-19T20:26:32.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:32.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:32.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:32.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:32.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:32.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:32.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:32.690+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:33.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:32.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:33.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:32.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:33.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:32.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:33.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:32.987+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:33.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:32.943+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:32.944611+0000) 2024-02-19T20:26:33.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:33.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:33.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:33.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:33.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:33.042+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:33.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:33.042+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:33.827 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:33.643+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:33.644939+0000) 2024-02-19T20:26:33.827 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:33.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:33.828 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:33.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:33.828 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:33.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:33.828 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:33.695+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:34.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:33.857+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:33.858040+0000) 2024-02-19T20:26:34.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:34.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:34.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:34.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:34.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:34.008+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:34.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:34.008+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:34.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:33.946+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:33.947898+0000) 2024-02-19T20:26:34.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:34.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:34.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:34.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:34.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:34.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:34.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:34.027+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:34.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:34.645+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:34.646702+0000) 2024-02-19T20:26:34.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:34.696+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:34.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:34.696+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:34.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:34.696+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:34.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:34.697+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:34.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:34.858+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:34.859538+0000) 2024-02-19T20:26:35.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:34.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:35.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:34.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:35.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:34.985+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:35.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:34.986+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:35.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:34.948+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:34.949763+0000) 2024-02-19T20:26:35.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:35.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:35.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:35.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:35.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:35.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:35.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:35.001+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:35.642 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:26:35.358+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61064 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:26:35.642 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:35.355+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:35.356876+0000) 2024-02-19T20:26:35.643 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:35.355+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:35.356633+0000) 2024-02-19T20:26:35.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:35.354+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:35.355951+0000) 2024-02-19T20:26:35.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:35.858+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:35.859746+0000) 2024-02-19T20:26:35.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:35.645+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:35.646900+0000) 2024-02-19T20:26:35.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:35.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:35.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:35.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:35.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:35.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:35.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:35.698+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:36.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:35.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:36.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:35.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:36.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:35.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:36.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:35.966+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:36.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:35.949+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:35.949972+0000) 2024-02-19T20:26:36.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:35.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:36.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:35.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:36.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:35.994+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:36.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:35.995+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:36.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:36.861+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:36.862147+0000) 2024-02-19T20:26:36.947 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:36.648+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:36.649285+0000) 2024-02-19T20:26:36.947 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:36.660+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:36.947 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:36.660+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:36.947 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:36.660+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:36.947 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:36.660+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:37.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:37.000+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:37.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:37.000+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:37.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:37.000+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:37.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:37.000+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:37.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:36.951+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:36.952395+0000) 2024-02-19T20:26:37.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:36.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:37.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:36.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:37.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:36.966+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:37.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:36.967+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:37.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:37.864+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:37.865402+0000) 2024-02-19T20:26:37.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:37.651+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:37.652531+0000) 2024-02-19T20:26:37.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:37.675+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:37.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:37.675+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:37.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:37.675+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:37.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:37.675+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:38.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:38.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:38.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:38.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:38.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:38.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:38.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:38.007+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:38.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:37.927+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:38.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:37.927+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:38.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:37.927+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:38.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:37.927+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:38.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:37.953+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:37.954963+0000) 2024-02-19T20:26:38.899 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:38.866+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:38.867474+0000) 2024-02-19T20:26:38.900 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:38.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:38.900 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:38.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:38.900 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:38.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:38.900 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:38.904+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:38.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:38.654+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:38.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:38.654+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:38.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:38.654+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:38.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:38.654+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:38.655317+0000) 2024-02-19T20:26:38.901 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:38.655+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:39.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:38.956+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:38.956986+0000) 2024-02-19T20:26:39.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:38.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:39.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:38.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:39.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:38.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:39.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:38.965+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:39.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:39.867+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:39.868934+0000) 2024-02-19T20:26:39.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:39.656+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:39.657112+0000) 2024-02-19T20:26:39.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:39.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:39.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:39.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:39.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:39.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:39.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:39.690+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:40.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:39.975+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:40.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:39.975+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:40.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:39.975+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:40.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:39.976+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:40.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:39.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:40.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:39.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:40.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:39.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:40.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:39.944+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:40.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:39.957+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:39.958027+0000) 2024-02-19T20:26:40.651 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:26:40.365+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61103 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:26:40.651 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:40.362+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:40.363495+0000) 2024-02-19T20:26:40.652 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:40.362+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:40.363220+0000) 2024-02-19T20:26:40.652 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:40.362+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:40.363389+0000) 2024-02-19T20:26:40.952 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:40.868+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:40.869116+0000) 2024-02-19T20:26:40.952 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:40.656+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:40.657319+0000) 2024-02-19T20:26:40.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:40.657+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:40.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:40.657+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:40.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:40.657+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:40.953 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:40.658+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:41.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:40.993+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:41.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:40.993+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:41.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:40.993+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:41.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:40.994+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:41.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:40.957+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:40.958200+0000) 2024-02-19T20:26:41.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:40.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:41.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:40.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:41.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:40.964+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:41.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:40.965+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:41.931 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:41.871+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:41.872292+0000) 2024-02-19T20:26:41.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:41.637+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:41.931 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:41.637+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:41.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:41.637+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:41.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:41.638+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:41.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:41.659+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:41.660028+0000) 2024-02-19T20:26:42.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:41.936+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:42.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:41.936+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:42.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:41.936+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:42.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:41.936+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:42.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:41.959+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:41.960586+0000) 2024-02-19T20:26:42.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:41.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:42.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:41.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:42.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:41.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:42.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:41.961+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:42.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:42.873+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:42.874397+0000) 2024-02-19T20:26:42.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:42.640+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:42.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:42.640+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:42.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:42.640+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:42.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:42.641+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:42.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:42.661+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:42.661996+0000) 2024-02-19T20:26:43.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:42.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:43.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:42.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:43.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:42.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:43.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:42.999+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:43.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:42.945+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:43.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:42.945+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:43.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:42.945+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:43.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:42.945+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:43.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:42.962+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:42.963106+0000) 2024-02-19T20:26:43.827 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:43.663+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:43.664454+0000) 2024-02-19T20:26:43.828 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:43.687+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:43.828 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:43.687+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:43.828 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:43.687+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:43.828 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:43.688+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:44.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:43.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:44.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:43.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:44.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:43.896+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:44.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:43.896+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:44.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:43.964+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:43.965096+0000) 2024-02-19T20:26:44.122 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:43.875+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:43.876458+0000) 2024-02-19T20:26:44.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:43.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:44.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:43.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:44.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:43.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:44.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:43.987+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:44.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:44.876+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:44.877581+0000) 2024-02-19T20:26:44.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:44.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:44.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:44.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:44.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:44.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:44.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:44.660+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:44.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:44.664+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:44.665573+0000) 2024-02-19T20:26:44.916 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:44.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:44.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:44.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:44.917 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:44.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:45.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:44.920+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:45.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:44.965+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:44.966330+0000) 2024-02-19T20:26:45.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:44.972+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:45.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:44.972+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:45.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:44.972+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:45.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:44.973+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:45.659 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:26:45.373+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61147 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:26:45.660 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:45.369+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:45.370779+0000) 2024-02-19T20:26:45.660 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:45.369+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:45.370734+0000) 2024-02-19T20:26:45.661 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:45.370+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:45.371420+0000) 2024-02-19T20:26:45.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:45.876+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:45.877777+0000) 2024-02-19T20:26:45.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:45.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:45.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:45.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:45.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:45.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:45.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:45.895+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:45.961 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:45.664+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:45.665770+0000) 2024-02-19T20:26:45.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:45.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:45.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:45.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:45.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:45.686+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:45.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:45.687+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:46.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:45.965+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:45.966497+0000) 2024-02-19T20:26:46.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:45.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:46.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:45.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:46.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:45.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:46.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:45.998+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:46.915 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:46.878+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:46.879843+0000) 2024-02-19T20:26:46.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:46.667+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:46.668393+0000) 2024-02-19T20:26:46.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:46.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:46.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:46.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:46.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:46.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:46.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:46.673+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:47.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:46.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:47.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:46.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:47.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:46.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:47.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:46.953+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:47.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:46.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:47.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:46.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:47.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:46.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:47.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:46.920+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:47.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:46.968+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:46.969058+0000) 2024-02-19T20:26:47.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:47.881+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:47.882288+0000) 2024-02-19T20:26:47.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:47.951+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:47.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:47.951+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:47.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:47.951+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:47.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:47.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:47.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:47.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:47.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:47.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:47.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:47.912+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:47.948 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:47.669+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:47.670926+0000) 2024-02-19T20:26:47.948 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:47.690+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:47.949 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:47.690+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:47.949 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:47.690+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:47.949 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:47.691+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:48.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:47.952+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:48.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:47.970+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:47.971691+0000) 2024-02-19T20:26:48.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:48.883+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:48.884743+0000) 2024-02-19T20:26:48.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:48.672+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:48.673629+0000) 2024-02-19T20:26:48.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:48.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:48.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:48.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:48.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:48.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:48.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:48.703+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:48.969 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:48.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:48.969 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:48.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:48.969 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:48.874+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:48.969 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:48.875+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:49.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:48.972+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:48.973885+0000) 2024-02-19T20:26:49.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:48.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:49.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:48.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:49.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:48.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:49.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:48.981+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:49.969 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:49.885+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:49.886738+0000) 2024-02-19T20:26:49.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:49.674+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:49.675019+0000) 2024-02-19T20:26:49.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:49.748+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:49.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:49.748+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:49.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:49.748+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:49.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:49.748+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:49.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:49.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:49.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:49.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:49.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:49.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:49.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:49.903+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:50.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:49.974+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:49.975101+0000) 2024-02-19T20:26:50.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:50.017+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:50.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:50.017+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:50.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:50.017+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:50.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:50.018+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:50.669 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:26:50.380+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61186 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:26:50.669 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:50.377+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:50.378170+0000) 2024-02-19T20:26:50.670 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:50.377+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:50.378355+0000) 2024-02-19T20:26:50.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:50.377+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:50.378858+0000) 2024-02-19T20:26:50.969 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:50.885+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:50.886925+0000) 2024-02-19T20:26:50.969 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:50.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:50.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:50.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:50.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:50.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:50.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:50.916+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:50.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:50.974+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:50.975307+0000) 2024-02-19T20:26:50.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:50.674+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:50.675439+0000) 2024-02-19T20:26:50.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:50.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:50.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:50.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:50.971 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:50.766+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:50.971 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:50.766+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:51.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:51.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:51.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:51.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:51.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:51.036+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:51.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:51.037+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:51.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:51.888+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:51.889258+0000) 2024-02-19T20:26:51.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:51.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:51.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:51.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:51.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:51.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:51.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:51.909+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:51.972 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:51.677+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:51.678009+0000) 2024-02-19T20:26:51.972 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:51.741+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:51.972 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:51.741+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:51.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:51.741+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:51.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:51.742+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:52.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:51.976+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:51.977481+0000) 2024-02-19T20:26:52.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:52.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:52.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:52.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:52.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:52.066+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:52.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:52.066+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:52.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:52.890+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:52.891254+0000) 2024-02-19T20:26:52.974 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:52.680+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:52.681379+0000) 2024-02-19T20:26:52.974 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:52.789+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:52.974 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:52.789+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:52.974 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:52.789+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:52.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:52.790+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:52.975 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:52.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:52.975 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:52.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:52.975 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:52.908+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:52.975 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:52.909+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:52.975 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:52.978+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:52.979768+0000) 2024-02-19T20:26:53.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:53.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:53.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:53.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:53.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:53.064+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:53.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:53.064+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:53.829 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:53.682+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:53.683685+0000) 2024-02-19T20:26:53.829 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:53.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:53.829 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:53.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:53.829 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:53.764+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:53.830 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:53.764+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:54.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:53.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:54.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:53.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:54.123 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:53.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:54.123 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:53.948+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:54.123 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:53.981+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:53.982483+0000) 2024-02-19T20:26:54.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:53.893+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:53.894717+0000) 2024-02-19T20:26:54.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:54.067+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:54.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:54.067+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:54.124 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:54.067+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:54.124 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:54.068+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:54.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:54.895+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:54.896618+0000) 2024-02-19T20:26:54.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:54.683+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:54.684877+0000) 2024-02-19T20:26:54.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:54.752+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:54.979 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:54.752+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:54.979 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:54.752+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:54.979 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:54.753+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:55.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:54.983+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:54.984400+0000) 2024-02-19T20:26:55.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:54.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:55.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:54.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:55.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:54.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:55.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:54.994+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:55.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:55.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:55.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:55.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:55.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:55.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:55.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:55.027+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:55.679 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:26:55.388+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61228 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:26:55.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:55.385+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:55.386518+0000) 2024-02-19T20:26:55.680 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:55.385+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:55.386831+0000) 2024-02-19T20:26:55.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:55.385+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:55.386298+0000) 2024-02-19T20:26:55.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:55.895+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:55.896838+0000) 2024-02-19T20:26:55.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:55.684+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:55.685077+0000) 2024-02-19T20:26:55.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:55.714+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:55.974 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:55.714+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:55.974 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:55.714+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:55.974 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:55.715+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:56.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:56.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:56.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:56.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:56.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:56.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:56.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:56.022+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:56.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:55.978+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:56.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:55.978+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:56.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:55.978+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:56.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:55.979+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:56.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:55.983+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:55.984491+0000) 2024-02-19T20:26:56.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:56.898+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:56.899483+0000) 2024-02-19T20:26:56.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:56.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:56.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:56.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:56.981 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:56.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:56.981 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:56.675+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:56.981 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:56.686+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:56.687189+0000) 2024-02-19T20:26:57.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:56.992+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:57.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:56.992+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:57.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:56.992+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:57.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:56.993+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:57.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:56.985+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:56.986460+0000) 2024-02-19T20:26:57.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:57.021+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:57.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:57.021+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:57.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:57.021+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:57.266 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:57.022+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:57.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:57.649+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:57.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:57.649+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:57.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:57.649+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:57.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:57.650+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:57.896 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:57.688+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:57.689567+0000) 2024-02-19T20:26:58.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:57.987+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:57.988482+0000) 2024-02-19T20:26:58.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:58.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:58.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:58.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:58.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:58.008+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:58.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:58.008+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:58.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:57.900+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:57.901496+0000) 2024-02-19T20:26:58.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:58.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:58.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:58.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:58.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:58.002+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:58.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:58.002+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:58.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:58.904+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:58.905021+0000) 2024-02-19T20:26:58.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:58.690+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:58.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:58.690+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:58.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:58.690+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:58.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:58.690+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:58.691532+0000) 2024-02-19T20:26:58.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:58.691+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:59.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:59.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:59.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:59.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:59.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:59.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:26:59.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:59.026+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:26:59.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:58.990+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:58.990962+0000) 2024-02-19T20:26:59.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:59.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:59.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:59.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:59.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:59.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:26:59.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:59.028+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:26:59.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:26:59.905+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:59.906943+0000) 2024-02-19T20:26:59.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:59.691+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:59.692825+0000) 2024-02-19T20:26:59.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:59.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:59.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:59.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:59.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:59.708+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:26:59.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:26:59.709+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:00.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:00.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:00.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:00.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:00.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:00.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:00.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:00.044+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:00.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:26:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:26:59.991+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:59.992305+0000) 2024-02-19T20:27:00.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:00.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:00.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:00.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:00.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:00.027+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:00.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:00.028+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:00.686 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:27:00.395+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61267 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:27:00.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:00.393+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:00.394146+0000) 2024-02-19T20:27:00.687 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:00.392+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:00.393278+0000) 2024-02-19T20:27:00.687 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:00.392+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:00.393278+0000) 2024-02-19T20:27:00.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:00.906+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:00.907128+0000) 2024-02-19T20:27:00.986 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:00.692+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:00.693050+0000) 2024-02-19T20:27:00.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:00.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:00.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:00.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:00.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:00.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:00.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:00.722+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:01.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:01.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:01.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:01.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:01.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:01.001+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:01.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:01.001+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:01.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:00.991+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:00.992522+0000) 2024-02-19T20:27:01.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:01.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:01.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:01.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:01.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:01.009+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:01.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:01.010+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:01.988 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:01.908+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:01.909627+0000) 2024-02-19T20:27:01.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:01.695+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:01.696194+0000) 2024-02-19T20:27:01.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:01.713+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:01.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:01.713+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:01.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:01.713+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:01.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:01.713+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:02.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:02.047+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:02.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:02.047+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:02.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:02.047+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:02.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:02.047+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:02.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:01.993+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:01.994592+0000) 2024-02-19T20:27:02.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:02.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:02.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:02.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:02.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:02.044+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:02.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:02.045+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:02.990 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:02.910+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:02.911827+0000) 2024-02-19T20:27:02.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:02.698+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:02.699485+0000) 2024-02-19T20:27:02.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:02.709+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:02.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:02.709+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:02.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:02.709+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:02.991 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:02.710+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:03.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:02.996+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:02.997120+0000) 2024-02-19T20:27:03.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:03.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:03.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:03.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:03.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:03.053+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:03.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:03.054+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:03.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:03.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:03.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:03.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:03.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:03.032+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:03.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:03.032+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:03.828 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:03.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:03.829 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:03.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:03.829 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:03.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:03.829 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:03.695+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:03.829 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:03.700+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:03.701588+0000) 2024-02-19T20:27:04.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:03.913+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:03.914469+0000) 2024-02-19T20:27:04.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:04.025+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:04.124 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:04.025+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:04.124 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:04.025+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:04.124 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:04.026+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:04.124 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:03.998+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:03.999345+0000) 2024-02-19T20:27:04.124 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:04.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:04.124 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:04.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:04.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:04.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:04.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:04.078+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:04.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:04.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:04.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:04.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:04.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:04.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:04.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:04.651+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:04.994 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:04.702+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:04.703522+0000) 2024-02-19T20:27:04.994 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:04.914+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:04.915798+0000) 2024-02-19T20:27:05.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:05.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:05.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:05.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:05.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:05.070+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:05.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:05.070+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:05.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:04.999+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:05.000778+0000) 2024-02-19T20:27:05.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:05.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:05.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:05.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:05.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:05.114+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:05.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:05.115+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:05.614 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:27:05.402+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61309 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:27:05.615 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:05.399+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:05.400155+0000) 2024-02-19T20:27:05.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:05.400+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:05.401623+0000) 2024-02-19T20:27:05.616 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:05.400+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:05.401037+0000) 2024-02-19T20:27:05.616 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:05.620+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:05.616 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:05.620+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:05.616 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:05.620+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:05.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:05.621+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:05.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:05.702+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:05.703733+0000) 2024-02-19T20:27:06.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:05.914+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:05.915969+0000) 2024-02-19T20:27:06.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:06.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:06.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:06.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:06.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:06.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:06.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:06.099+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:06.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:05.999+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:06.000970+0000) 2024-02-19T20:27:06.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:06.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:06.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:06.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:06.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:06.076+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:06.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:06.077+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:06.898 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:06.628+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:06.898 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:06.628+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:06.898 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:06.628+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:06.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:06.629+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:06.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:06.705+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:06.705977+0000) 2024-02-19T20:27:07.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:06.917+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:06.918023+0000) 2024-02-19T20:27:07.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:07.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:07.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:07.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:07.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:07.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:07.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:07.097+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:07.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:07.003+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:07.004152+0000) 2024-02-19T20:27:07.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:07.108+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:07.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:07.108+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:07.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:07.108+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:07.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:07.109+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:07.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:07.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:07.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:07.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:07.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:07.593+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:07.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:07.593+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:07.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:07.707+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:07.708541+0000) 2024-02-19T20:27:08.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:07.919+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:07.920074+0000) 2024-02-19T20:27:08.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:08.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:08.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:08.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:08.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:08.096+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:08.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:08.097+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:08.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:08.005+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:08.006564+0000) 2024-02-19T20:27:08.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:08.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:08.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:08.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:08.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:08.078+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:08.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:08.078+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:08.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:08.590+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:08.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:08.590+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:08.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:08.590+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:08.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:08.591+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:08.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:08.711+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:08.712101+0000) 2024-02-19T20:27:09.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:09.009+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:09.010147+0000) 2024-02-19T20:27:09.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:09.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:09.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:09.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:09.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:09.067+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:09.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:09.067+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:09.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:08.921+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:08.922472+0000) 2024-02-19T20:27:09.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:09.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:09.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:09.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:09.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:09.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:09.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:09.057+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:09.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:09.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:09.707 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:09.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:09.707 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:09.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:09.707 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:09.546+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:10.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:09.712+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:09.713215+0000) 2024-02-19T20:27:10.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:09.922+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:09.923855+0000) 2024-02-19T20:27:10.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:10.010+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:10.011405+0000) 2024-02-19T20:27:10.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:10.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:10.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:10.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:10.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:10.036+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:10.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:10.037+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:10.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:10.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:10.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:10.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:10.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:10.082+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:10.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:10.083+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:10.513 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:27:10.409+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61348 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:27:10.513 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:10.407+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:10.408135+0000) 2024-02-19T20:27:10.514 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:10.406+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:10.407422+0000) 2024-02-19T20:27:10.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:10.406+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:10.407738+0000) 2024-02-19T20:27:10.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:10.520+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:10.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:10.520+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:10.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:10.520+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:10.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:10.521+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:10.900 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:10.712+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:10.713408+0000) 2024-02-19T20:27:11.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:10.923+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:10.924033+0000) 2024-02-19T20:27:11.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:11.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:11.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:11.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:11.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:11.039+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:11.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:11.039+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:11.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:11.011+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:11.011658+0000) 2024-02-19T20:27:11.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:11.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:11.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:11.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:11.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:11.071+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:11.268 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:11.071+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:11.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:11.497+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:11.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:11.497+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:11.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:11.497+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:11.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:11.498+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:12.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:11.715+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:11.716022+0000) 2024-02-19T20:27:12.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:11.926+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:11.926376+0000) 2024-02-19T20:27:12.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:12.013+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:12.014020+0000) 2024-02-19T20:27:12.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:12.031+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:12.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:12.031+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:12.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:12.031+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:12.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:12.031+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:12.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:12.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:12.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:12.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:12.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:12.083+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:12.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:12.083+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:12.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:12.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:12.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:12.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:12.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:12.492+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:12.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:12.492+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:13.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:12.718+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:12.718190+0000) 2024-02-19T20:27:13.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:12.928+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:12.929090+0000) 2024-02-19T20:27:13.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:13.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:13.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:13.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:13.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:13.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:13.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:13.113+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:13.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:13.015+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:13.016130+0000) 2024-02-19T20:27:13.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:13.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:13.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:13.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:13.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:13.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:13.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:13.027+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:13.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:13.476+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:13.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:13.476+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:13.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:13.476+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:13.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:13.476+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:13.829 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:13.718+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:13.720969+0000) 2024-02-19T20:27:14.124 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:13.930+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:13.931261+0000) 2024-02-19T20:27:14.124 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:13.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:14.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:13.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:14.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:13.981+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:14.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:13.982+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:14.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:14.017+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:14.018952+0000) 2024-02-19T20:27:14.396 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:14.136+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:14.397 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:14.136+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:14.397 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:14.136+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:14.397 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:14.137+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:14.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:14.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:14.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:14.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:14.463+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:14.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:14.464+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:14.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:14.721+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:14.722107+0000) 2024-02-19T20:27:14.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:14.932+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:14.933265+0000) 2024-02-19T20:27:15.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:15.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:15.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:15.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:15.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:15.095+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:15.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:15.096+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:15.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:15.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:15.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:15.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:15.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:15.004+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:15.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:15.005+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:15.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:15.019+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:15.020153+0000) 2024-02-19T20:27:15.715 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:27:15.417+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61390 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:27:15.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:15.416+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:15.417866+0000) 2024-02-19T20:27:15.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:15.414+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:15.415576+0000) 2024-02-19T20:27:15.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:15.413+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:15.414555+0000) 2024-02-19T20:27:15.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:15.464+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:15.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:15.464+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:15.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:15.464+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:15.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:15.465+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:15.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:15.932+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:15.933490+0000) 2024-02-19T20:27:15.974 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:15.721+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:15.722347+0000) 2024-02-19T20:27:16.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:16.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:16.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:16.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:16.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:16.108+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:16.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:16.109+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:16.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:15.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:16.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:15.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:16.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:15.979+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:16.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:15.979+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:16.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:16.019+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:16.020346+0000) 2024-02-19T20:27:16.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:16.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:16.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:16.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:16.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:16.431+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:16.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:16.432+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:17.014 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:16.724+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:16.725373+0000) 2024-02-19T20:27:17.014 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:16.935+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:16.936084+0000) 2024-02-19T20:27:17.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:17.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:17.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:17.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:17.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:17.074+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:17.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:17.074+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:17.319 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:17.021+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:17.022442+0000) 2024-02-19T20:27:17.319 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:17.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:17.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:17.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:17.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:17.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:17.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:17.023+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:17.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:17.456+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:17.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:17.456+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:17.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:17.456+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:17.722 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:17.457+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:18.013 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:17.726+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:17.727873+0000) 2024-02-19T20:27:18.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:17.937+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:17.938139+0000) 2024-02-19T20:27:18.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:18.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:18.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:18.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:18.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:18.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:18.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:18.044+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:18.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:18.023+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:18.024766+0000) 2024-02-19T20:27:18.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:18.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:18.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:18.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:18.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:18.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:18.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:18.073+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:18.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:18.454+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:18.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:18.454+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:18.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:18.454+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:18.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:18.455+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:19.013 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:18.729+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:18.730644+0000) 2024-02-19T20:27:19.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:18.939+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:18.940585+0000) 2024-02-19T20:27:19.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:19.047+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:19.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:19.047+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:19.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:19.047+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:19.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:19.047+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:19.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:19.025+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:19.026884+0000) 2024-02-19T20:27:19.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:19.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:19.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:19.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:19.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:19.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:19.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:19.123+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:19.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:19.427+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:19.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:19.427+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:19.725 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:19.427+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:19.726 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:19.427+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:19.726 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:19.731+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:19.732023+0000) 2024-02-19T20:27:20.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:19.940+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:19.941930+0000) 2024-02-19T20:27:20.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:20.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:20.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:20.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:20.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:20.056+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:20.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:20.056+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:20.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:20.027+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:20.028865+0000) 2024-02-19T20:27:20.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:20.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:20.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:20.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:20.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:20.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:20.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:20.141+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:20.725 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:27:20.424+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61429 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:27:20.726 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:20.422+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:20.423163+0000) 2024-02-19T20:27:20.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:20.422+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:20.423066+0000) 2024-02-19T20:27:20.726 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:20.421+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:20.422961+0000) 2024-02-19T20:27:20.726 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:20.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:20.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:20.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:20.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:20.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:20.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:20.469+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:21.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:20.731+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:20.732235+0000) 2024-02-19T20:27:21.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:20.941+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:20.942109+0000) 2024-02-19T20:27:21.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:21.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:21.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:21.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:21.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:21.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:21.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:21.044+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:21.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:21.028+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:21.029027+0000) 2024-02-19T20:27:21.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:21.128+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:21.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:21.128+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:21.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:21.128+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:21.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:21.128+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:21.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:21.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:21.728 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:21.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:21.728 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:21.517+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:21.728 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:21.518+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:22.013 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:21.733+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:21.734418+0000) 2024-02-19T20:27:22.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:21.943+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:21.944835+0000) 2024-02-19T20:27:22.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:22.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:22.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:22.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:22.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:22.051+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:22.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:22.052+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:22.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:22.031+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:22.032357+0000) 2024-02-19T20:27:22.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:22.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:22.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:22.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:22.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:22.137+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:22.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:22.138+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:22.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:22.532+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:22.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:22.532+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:22.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:22.532+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:22.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:22.533+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:23.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:22.735+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:22.736568+0000) 2024-02-19T20:27:23.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:22.946+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:22.946976+0000) 2024-02-19T20:27:23.319 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:23.034+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:23.035731+0000) 2024-02-19T20:27:23.319 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:23.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:23.319 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:23.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:23.319 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:23.091+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:23.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:23.092+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:23.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:23.097+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:23.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:23.097+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:23.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:23.097+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:23.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:23.098+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:23.574 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:23.538+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:23.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:23.538+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:23.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:23.538+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:23.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:23.539+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:23.830 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:23.737+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:23.738700+0000) 2024-02-19T20:27:24.125 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:23.948+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:23.949039+0000) 2024-02-19T20:27:24.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:24.038+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:24.039275+0000) 2024-02-19T20:27:24.398 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:24.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:24.398 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:24.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:24.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:24.134+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:24.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:24.135+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:24.399 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:24.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:24.399 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:24.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:24.399 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:24.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:24.399 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:24.131+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:24.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:24.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:24.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:24.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:24.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:24.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:24.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:24.574+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:25.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:24.739+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:24.740022+0000) 2024-02-19T20:27:25.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:24.949+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:24.950203+0000) 2024-02-19T20:27:25.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:25.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:25.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:25.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:25.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:25.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:25.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:25.149+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:25.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:25.039+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:25.040734+0000) 2024-02-19T20:27:25.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:25.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:25.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:25.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:25.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:25.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:25.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:25.153+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:25.733 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:27:25.432+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61471 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:27:25.733 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:25.430+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:25.431076+0000) 2024-02-19T20:27:25.734 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:25.428+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:25.429969+0000) 2024-02-19T20:27:25.734 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:25.533+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:25.734 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:25.533+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:25.734 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:25.533+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:25.734 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:25.534+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:25.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:25.429+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:25.430285+0000) 2024-02-19T20:27:26.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:25.739+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:25.740227+0000) 2024-02-19T20:27:26.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:25.949+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:25.950404+0000) 2024-02-19T20:27:26.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:26.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:26.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:26.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:26.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:26.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:26.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:26.113+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:26.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:26.039+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:26.040896+0000) 2024-02-19T20:27:26.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:26.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:26.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:26.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:26.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:26.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:26.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:26.123+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:26.914 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:26.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:26.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:26.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:26.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:26.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:26.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:26.582+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:26.915 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:26.741+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:26.742691+0000) 2024-02-19T20:27:27.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:27.043+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:27.044276+0000) 2024-02-19T20:27:27.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:27.169+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:27.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:27.169+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:27.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:27.169+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:27.265 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:27.170+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:27.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:26.951+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:26.952518+0000) 2024-02-19T20:27:27.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:27.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:27.265 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:27.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:27.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:27.078+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:27.266 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:27.079+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:27.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:27.580+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:27.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:27.580+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:27.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:27.580+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:27.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:27.581+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:27.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:27.745+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:27.746051+0000) 2024-02-19T20:27:28.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:27.953+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:27.954767+0000) 2024-02-19T20:27:28.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:28.109+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:28.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:28.109+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:28.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:28.109+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:28.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:28.110+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:28.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:28.046+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:28.046998+0000) 2024-02-19T20:27:28.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:28.125+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:28.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:28.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:28.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:28.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:28.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:28.126+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:28.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:28.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:28.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:28.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:28.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:28.587+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:28.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:28.588+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:28.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:28.748+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:28.749490+0000) 2024-02-19T20:27:29.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:28.955+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:28.956737+0000) 2024-02-19T20:27:29.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:29.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:29.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:29.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:29.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:29.149+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:29.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:29.150+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:29.263 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:29.048+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:29.049198+0000) 2024-02-19T20:27:29.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:29.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:29.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:29.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:29.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:29.148+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:29.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:29.148+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:29.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:29.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:29.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:29.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:29.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:29.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:29.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:29.590+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:29.916 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:29.750+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:29.751422+0000) 2024-02-19T20:27:30.170 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:29.956+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:29.957940+0000) 2024-02-19T20:27:30.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:30.049+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:30.050436+0000) 2024-02-19T20:27:30.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:30.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:30.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:30.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:30.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:30.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:30.429 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:30.176+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:30.429 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:30.200+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:30.429 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:30.200+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:30.429 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:30.200+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:30.430 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:30.201+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:30.744 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:27:30.440+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61510 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:27:30.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:30.436+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:30.437188+0000) 2024-02-19T20:27:30.745 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:30.436+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:30.437973+0000) 2024-02-19T20:27:30.745 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:30.639+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:30.745 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:30.639+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:30.746 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:30.639+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:30.746 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:30.640+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:30.746 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:30.437+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:30.438390+0000) 2024-02-19T20:27:31.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:30.750+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:30.751613+0000) 2024-02-19T20:27:31.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:30.957+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:30.958107+0000) 2024-02-19T20:27:31.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:31.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:31.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:31.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:31.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:31.232+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:31.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:31.233+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:31.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:31.049+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:31.050608+0000) 2024-02-19T20:27:31.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:31.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:31.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:31.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:31.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:31.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:31.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:31.140+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:31.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:31.678+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:31.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:31.678+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:31.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:31.678+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:31.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:31.678+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:31.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:31.754+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:31.755020+0000) 2024-02-19T20:27:32.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:31.960+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:31.961665+0000) 2024-02-19T20:27:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:32.052+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:32.053966+0000) 2024-02-19T20:27:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:32.173+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:32.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:32.173+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:32.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:32.173+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:32.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:32.174+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:32.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:32.222+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:32.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:32.222+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:32.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:32.222+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:32.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:32.223+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:32.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:32.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:32.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:32.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:32.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:32.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:32.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:32.652+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:32.921 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:32.757+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:32.758476+0000) 2024-02-19T20:27:33.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:32.962+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:32.963698+0000) 2024-02-19T20:27:33.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:33.055+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:33.056519+0000) 2024-02-19T20:27:33.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:33.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:33.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:33.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:33.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:33.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:33.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:33.176+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:33.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:33.239+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:33.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:33.239+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:33.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:33.239+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:33.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:33.240+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:33.830 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:33.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:33.830 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:33.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:33.830 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:33.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:33.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:33.635+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:33.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:33.759+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:33.760687+0000) 2024-02-19T20:27:34.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:33.965+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:33.966212+0000) 2024-02-19T20:27:34.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:34.058+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:34.059027+0000) 2024-02-19T20:27:34.398 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:34.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:34.398 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:34.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:34.399 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:34.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:34.399 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:34.192+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:34.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:34.208+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:34.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:34.208+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:34.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:34.208+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:34.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:34.208+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:34.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:34.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:34.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:34.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:34.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:34.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:34.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:34.667+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:34.920 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:34.761+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:34.762006+0000) 2024-02-19T20:27:35.199 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:34.966+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:34.967627+0000) 2024-02-19T20:27:35.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:35.060+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:35.061097+0000) 2024-02-19T20:27:35.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:35.205+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:35.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:35.205+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:35.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:35.205+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:35.450 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:27:35.448+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61552 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:27:35.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:35.445+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:35.446103+0000) 2024-02-19T20:27:35.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:35.206+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:35.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:35.444+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:35.445534+0000) 2024-02-19T20:27:35.451 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:35.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:35.451 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:35.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:35.451 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:35.234+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:35.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:35.234+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:35.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:35.445+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:35.446431+0000) 2024-02-19T20:27:35.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:35.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:35.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:35.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:35.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:35.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:35.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:35.625+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:36.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:35.761+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:35.762168+0000) 2024-02-19T20:27:36.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:35.966+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:35.967832+0000) 2024-02-19T20:27:36.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:36.186+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:36.320 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:36.186+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:36.321 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:36.186+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:36.321 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:36.186+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:36.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:36.060+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:36.061272+0000) 2024-02-19T20:27:36.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:36.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:36.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:36.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:36.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:36.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:36.322 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:36.209+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:36.922 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:36.640+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:36.922 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:36.640+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:36.923 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:36.640+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:36.923 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:36.640+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:36.923 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:36.763+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:36.764468+0000) 2024-02-19T20:27:37.224 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:36.970+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:36.971423+0000) 2024-02-19T20:27:37.224 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:37.063+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:37.064215+0000) 2024-02-19T20:27:37.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:37.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:37.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:37.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:37.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:37.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:37.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:37.234+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:37.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:37.230+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:37.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:37.230+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:37.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:37.230+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:37.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:37.230+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:37.923 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:37.638+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:37.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:37.638+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:37.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:37.638+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:37.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:37.640+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:37.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:37.766+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:37.767063+0000) 2024-02-19T20:27:38.227 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:37.972+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:37.973592+0000) 2024-02-19T20:27:38.227 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:38.065+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:38.066323+0000) 2024-02-19T20:27:38.227 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:38.233+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:38.228 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:38.233+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:38.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:38.233+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:38.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:38.233+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:38.514 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:38.259+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:38.514 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:38.259+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:38.514 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:38.259+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:38.514 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:38.260+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:38.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:38.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:38.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:38.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:38.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:38.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:38.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:38.624+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:38.925 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:38.768+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:38.769525+0000) 2024-02-19T20:27:39.222 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:38.975+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:38.976016+0000) 2024-02-19T20:27:39.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:39.068+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:39.069741+0000) 2024-02-19T20:27:39.223 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:39.228+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:39.223 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:39.228+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:39.223 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:39.228+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:39.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:39.228+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:39.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:39.238+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:39.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:39.238+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:39.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:39.238+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:39.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:39.239+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:39.923 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:39.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:39.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:39.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:39.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:39.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:39.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:39.632+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:39.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:39.769+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:39.770818+0000) 2024-02-19T20:27:40.204 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:39.976+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:39.977913+0000) 2024-02-19T20:27:40.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:40.069+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:40.070923+0000) 2024-02-19T20:27:40.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:40.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:40.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:40.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:40.205 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:40.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:40.456 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:27:40.455+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61591 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:27:40.457 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:40.453+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:40.454506+0000) 2024-02-19T20:27:40.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:40.211+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:40.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:40.451+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:40.452862+0000) 2024-02-19T20:27:40.457 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:40.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:40.458 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:40.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:40.458 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:40.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:40.458 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:40.229+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:40.458 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:40.451+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:40.452816+0000) 2024-02-19T20:27:40.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:40.601+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:40.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:40.601+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:40.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:40.601+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:40.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:40.602+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:41.064 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:40.770+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:40.771002+0000) 2024-02-19T20:27:41.064 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:40.977+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:40.978116+0000) 2024-02-19T20:27:41.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:41.070+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:41.071091+0000) 2024-02-19T20:27:41.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:41.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:41.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:41.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:41.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:41.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:41.321 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:41.191+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:41.322 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:41.198+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:41.322 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:41.198+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:41.322 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:41.199+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:41.322 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:41.199+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:41.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:41.557+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:41.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:41.557+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:41.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:41.557+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:41.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:41.557+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:42.066 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:41.773+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:41.774529+0000) 2024-02-19T20:27:42.066 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:41.980+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:41.981625+0000) 2024-02-19T20:27:42.321 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:42.238+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:42.321 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:42.238+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:42.321 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:42.238+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:42.321 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:42.239+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:42.322 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:42.072+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:42.073327+0000) 2024-02-19T20:27:42.322 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:42.237+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:42.322 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:42.237+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:42.322 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:42.237+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:42.322 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:42.238+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:42.927 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:42.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:42.927 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:42.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:42.927 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:42.581+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:42.927 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:42.582+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:42.927 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:42.776+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:42.777149+0000) 2024-02-19T20:27:43.199 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:42.983+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:42.984155+0000) 2024-02-19T20:27:43.200 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:43.074+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:43.075895+0000) 2024-02-19T20:27:43.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:43.240+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:43.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:43.240+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:43.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:43.240+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:43.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:43.241+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:43.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:43.205+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:43.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:43.205+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:43.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:43.205+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:43.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:43.206+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:43.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:43.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:43.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:43.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:43.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:43.591+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:43.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:43.592+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:43.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:43.778+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:43.779133+0000) 2024-02-19T20:27:44.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:43.985+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:43.986655+0000) 2024-02-19T20:27:44.127 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:44.076+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:44.077893+0000) 2024-02-19T20:27:44.399 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:44.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:44.399 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:44.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:44.399 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:44.236+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:44.399 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:44.237+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:44.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:44.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:44.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:44.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:44.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:44.163+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:44.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:44.164+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:44.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:44.595+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:44.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:44.595+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:44.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:44.595+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:44.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:44.596+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:45.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:44.779+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:44.780712+0000) 2024-02-19T20:27:45.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:44.987+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:44.987983+0000) 2024-02-19T20:27:45.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:45.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:45.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:45.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:45.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:45.142+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:45.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:45.143+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:45.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:45.079+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:45.080000+0000) 2024-02-19T20:27:45.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:45.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:45.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:45.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:45.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:45.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:45.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:45.193+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:45.762 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:27:45.463+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61633 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:27:45.763 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:45.459+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:45.460829+0000) 2024-02-19T20:27:45.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:45.459+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:45.460547+0000) 2024-02-19T20:27:45.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:45.460+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:45.461215+0000) 2024-02-19T20:27:45.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:45.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:45.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:45.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:45.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:45.615+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:45.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:45.615+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:46.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:45.779+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:45.780939+0000) 2024-02-19T20:27:46.073 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:45.987+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:45.988159+0000) 2024-02-19T20:27:46.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:46.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:46.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:46.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:46.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:46.116+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:46.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:46.116+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:46.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:46.079+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:46.080188+0000) 2024-02-19T20:27:46.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:46.222+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:46.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:46.222+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:46.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:46.222+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:46.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:46.223+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:46.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:46.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:46.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:46.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:46.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:46.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:46.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:46.625+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:47.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:46.782+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:46.783494+0000) 2024-02-19T20:27:47.075 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:46.989+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:46.990346+0000) 2024-02-19T20:27:47.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:47.136+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:47.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:47.136+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:47.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:47.136+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:47.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:47.137+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:47.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:47.081+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:47.082518+0000) 2024-02-19T20:27:47.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:47.225+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:47.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:47.225+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:47.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:47.225+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:47.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:47.225+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:47.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:47.590+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:47.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:47.590+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:47.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:47.590+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:47.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:47.591+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:48.078 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:47.785+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:47.786093+0000) 2024-02-19T20:27:48.079 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:47.991+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:47.992381+0000) 2024-02-19T20:27:48.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:48.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:48.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:48.112+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:48.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:48.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:48.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:48.113+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:48.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:48.084+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:48.085830+0000) 2024-02-19T20:27:48.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:48.225+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:48.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:48.225+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:48.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:48.225+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:48.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:48.226+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:48.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:48.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:48.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:48.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:48.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:48.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:48.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:48.569+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:49.082 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:48.993+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:48.994812+0000) 2024-02-19T20:27:49.082 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:48.787+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:48.788194+0000) 2024-02-19T20:27:49.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:49.161+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:49.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:49.161+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:49.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:49.161+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:49.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:49.161+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:49.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:49.088+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:49.089326+0000) 2024-02-19T20:27:49.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:49.244+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:49.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:49.244+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:49.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:49.244+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:49.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:49.245+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:49.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:49.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:49.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:49.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:49.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:49.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:49.764 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:49.608+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:50.084 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:49.788+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:49.789834+0000) 2024-02-19T20:27:50.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:49.995+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:49.996848+0000) 2024-02-19T20:27:50.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:50.114+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:50.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:50.114+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:50.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:50.114+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:50.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:50.115+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:50.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:50.090+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:50.091177+0000) 2024-02-19T20:27:50.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:50.233+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:50.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:50.233+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:50.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:50.233+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:50.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:50.233+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:50.674 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:27:50.470+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61672 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:27:50.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:50.468+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:50.469309+0000) 2024-02-19T20:27:50.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:50.467+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:50.468765+0000) 2024-02-19T20:27:50.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:50.467+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:50.468694+0000) 2024-02-19T20:27:50.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:50.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:50.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:50.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:50.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:50.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:50.676 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:50.608+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:50.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:50.789+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:50.790017+0000) 2024-02-19T20:27:51.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:51.090+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:51.091365+0000) 2024-02-19T20:27:51.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:50.996+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:50.997029+0000) 2024-02-19T20:27:51.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:51.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:51.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:51.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:51.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:51.154+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:51.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:51.154+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:51.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:51.282+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:51.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:51.282+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:51.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:51.282+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:51.611 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:51.283+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:51.934 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:51.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:51.934 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:51.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:51.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:51.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:51.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:51.618+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:51.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:51.791+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:51.792381+0000) 2024-02-19T20:27:52.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:52.092+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:52.093540+0000) 2024-02-19T20:27:52.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:51.998+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:51.999688+0000) 2024-02-19T20:27:52.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:52.165+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:52.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:52.165+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:52.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:52.165+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:52.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:52.166+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:52.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:52.326+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:52.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:52.326+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:52.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:52.326+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:52.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:52.327+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:52.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:52.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:52.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:52.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:52.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:52.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:52.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:52.637+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:52.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:52.793+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:52.794551+0000) 2024-02-19T20:27:53.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:53.094+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:53.095528+0000) 2024-02-19T20:27:53.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:53.000+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:53.001730+0000) 2024-02-19T20:27:53.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:53.190+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:53.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:53.190+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:53.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:53.190+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:53.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:53.191+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:53.574 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:53.357+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:53.574 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:53.357+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:53.574 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:53.357+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:53.574 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:53.357+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:53.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:53.654+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:53.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:53.654+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:53.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:53.654+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:53.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:53.654+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:53.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:53.796+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:53.797193+0000) 2024-02-19T20:27:54.091 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:54.002+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:54.003943+0000) 2024-02-19T20:27:54.376 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:54.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:54.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:54.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:54.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:54.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:54.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:54.163+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:54.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:54.097+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:54.098839+0000) 2024-02-19T20:27:54.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:54.382+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:54.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:54.382+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:54.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:54.382+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:54.698 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:54.383+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:54.998 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:54.704+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:54.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:54.704+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:54.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:54.704+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:54.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:54.705+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:54.999 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:54.798+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:54.799262+0000) 2024-02-19T20:27:55.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:55.099+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:55.100888+0000) 2024-02-19T20:27:55.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:55.005+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:55.006042+0000) 2024-02-19T20:27:55.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:55.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:55.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:55.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:55.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:55.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:55.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:55.156+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:55.591 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:27:55.478+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61714 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:27:55.591 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:55.475+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:55.476155+0000) 2024-02-19T20:27:55.592 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:55.475+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:55.476051+0000) 2024-02-19T20:27:55.592 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:55.355+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:55.592 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:55.355+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:55.592 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:55.355+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:55.592 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:55.355+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:55.593 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:55.474+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:55.475611+0000) 2024-02-19T20:27:55.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:55.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:55.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:55.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:55.937 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:55.666+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:55.937 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:55.668+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:55.937 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:55.798+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:55.799429+0000) 2024-02-19T20:27:56.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:56.100+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:56.101098+0000) 2024-02-19T20:27:56.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:56.005+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:56.006240+0000) 2024-02-19T20:27:56.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:56.171+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:56.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:56.171+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:56.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:56.171+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:56.263 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:56.172+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:56.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:56.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:56.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:56.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:56.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:56.380+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:56.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:56.380+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:56.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:56.642+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:56.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:56.642+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:56.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:56.642+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:56.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:56.643+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:56.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:56.801+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:56.802766+0000) 2024-02-19T20:27:57.197 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:57.102+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:57.103030+0000) 2024-02-19T20:27:57.197 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:57.008+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:57.009581+0000) 2024-02-19T20:27:57.197 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:57.203+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:57.197 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:57.203+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:57.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:57.203+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:57.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:57.204+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:57.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:57.416+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:57.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:57.416+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:57.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:57.416+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:57.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:57.417+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:57.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:57.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:57.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:57.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:57.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:57.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:57.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:57.624+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:57.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:57.803+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:57.804778+0000) 2024-02-19T20:27:58.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:58.104+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:58.105095+0000) 2024-02-19T20:27:58.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:58.010+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:58.011505+0000) 2024-02-19T20:27:58.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:58.229+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:58.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:58.229+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:58.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:58.229+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:58.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:58.229+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:58.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:58.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:58.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:58.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:58.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:58.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:58.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:58.401+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:58.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:58.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:58.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:58.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:58.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:58.651+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:58.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:58.652+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:58.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:58.805+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:58.806921+0000) 2024-02-19T20:27:59.225 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:59.012+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:59.013922+0000) 2024-02-19T20:27:59.225 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:59.231+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:59.225 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:59.231+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:59.225 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:59.231+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:27:59.226 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:59.106+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:59.107202+0000) 2024-02-19T20:27:59.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:27:59.232+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:27:59.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:59.364+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:59.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:59.364+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:59.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:59.364+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:27:59.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:27:59.365+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:59.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:59.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:59.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:59.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:59.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:59.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:27:59.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:59.692+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:27:59.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:27:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:27:59.807+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:59.808202+0000) 2024-02-19T20:28:00.250 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:00.015+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:00.016018+0000) 2024-02-19T20:28:00.251 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:00.107+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:00.108468+0000) 2024-02-19T20:28:00.512 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:28:00.486+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61753 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:28:00.512 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:00.482+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:00.483713+0000) 2024-02-19T20:28:00.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:00.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:00.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:00.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:00.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:00.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:00.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:00.257+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:00.513 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:00.482+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:00.483202+0000) 2024-02-19T20:28:00.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:00.355+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:00.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:00.355+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:00.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:00.355+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:00.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:00.355+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:00.514 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:00.482+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:00.483058+0000) 2024-02-19T20:28:00.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:00.693+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:00.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:00.693+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:00.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:00.693+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:00.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:00.693+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:00.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:00.806+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:00.808376+0000) 2024-02-19T20:28:01.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:01.014+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:01.016198+0000) 2024-02-19T20:28:01.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:01.106+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:01.108697+0000) 2024-02-19T20:28:01.671 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:01.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:01.671 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:01.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:01.671 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:01.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:01.671 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:01.276+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:01.672 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:01.323+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:01.672 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:01.323+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:01.672 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:01.323+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:01.672 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:01.323+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:01.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:01.677+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:01.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:01.677+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:01.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:01.677+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:01.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:01.678+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:01.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:01.809+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:01.810830+0000) 2024-02-19T20:28:02.264 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:02.017+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:02.018278+0000) 2024-02-19T20:28:02.264 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:02.109+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:02.110819+0000) 2024-02-19T20:28:02.710 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:02.300+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:02.710 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:02.300+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:02.710 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:02.300+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:02.710 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:02.300+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:02.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:02.300+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:02.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:02.300+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:02.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:02.300+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:02.711 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:02.301+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:02.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:02.716+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:02.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:02.716+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:02.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:02.716+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:02.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:02.717+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:02.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:02.812+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:02.813249+0000) 2024-02-19T20:28:03.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:03.019+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:03.020434+0000) 2024-02-19T20:28:03.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:03.113+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:03.114239+0000) 2024-02-19T20:28:03.574 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:03.322+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:03.574 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:03.322+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:03.574 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:03.322+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:03.574 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:03.323+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:03.575 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:03.289+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:03.575 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:03.289+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:03.575 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:03.289+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:03.575 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:03.290+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:03.831 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:03.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:03.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:03.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:03.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:03.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:03.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:03.675+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:03.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:03.814+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:03.815643+0000) 2024-02-19T20:28:04.111 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:04.022+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:04.023024+0000) 2024-02-19T20:28:04.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:04.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:04.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:04.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:04.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:04.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:04.401 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:04.350+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:04.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:04.115+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:04.116868+0000) 2024-02-19T20:28:04.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:04.326+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:04.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:04.326+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:04.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:04.326+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:04.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:04.326+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:05.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:04.723+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:05.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:04.723+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:05.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:04.723+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:05.013 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:04.724+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:05.013 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:04.816+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:04.817095+0000) 2024-02-19T20:28:05.322 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:05.023+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:05.024310+0000) 2024-02-19T20:28:05.323 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:05.117+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:05.118213+0000) 2024-02-19T20:28:05.604 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:28:05.493+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61795 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:28:05.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:05.490+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:05.491505+0000) 2024-02-19T20:28:05.605 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:05.396+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:05.605 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:05.396+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:05.605 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:05.396+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:05.605 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:05.397+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:05.606 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:05.491+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:05.492286+0000) 2024-02-19T20:28:05.606 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:05.330+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:05.606 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:05.330+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:05.606 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:05.330+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:05.607 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:05.330+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:05.607 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:05.490+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:05.491278+0000) 2024-02-19T20:28:05.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:05.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:05.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:05.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:05.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:05.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:05.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:05.691+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:05.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:05.816+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:05.817253+0000) 2024-02-19T20:28:06.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:06.117+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:06.118387+0000) 2024-02-19T20:28:06.262 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:06.023+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:06.024504+0000) 2024-02-19T20:28:06.706 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:06.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:06.706 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:06.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:06.706 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:06.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:06.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:06.394+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:06.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:06.379+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:06.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:06.379+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:06.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:06.379+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:06.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:06.379+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:07.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:06.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:07.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:06.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:07.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:06.712+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:07.013 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:06.713+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:07.013 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:06.818+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:06.819302+0000) 2024-02-19T20:28:07.323 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:07.025+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:07.026646+0000) 2024-02-19T20:28:07.323 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:07.119+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:07.120968+0000) 2024-02-19T20:28:07.730 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:07.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:07.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:07.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:07.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:07.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:07.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:07.360+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:07.731 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:07.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:07.731 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:07.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:07.731 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:07.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:07.731 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:07.429+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:08.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:07.737+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:08.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:07.737+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:08.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:07.737+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:08.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:07.737+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:08.013 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:07.820+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:07.821665+0000) 2024-02-19T20:28:08.323 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:08.028+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:08.029352+0000) 2024-02-19T20:28:08.323 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:08.122+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:08.123518+0000) 2024-02-19T20:28:08.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:08.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:08.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:08.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:08.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:08.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:08.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:08.332+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:08.731 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:08.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:08.732 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:08.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:08.732 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:08.427+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:08.732 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:08.428+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:09.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:08.737+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:09.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:08.737+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:09.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:08.737+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:09.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:08.738+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:09.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:08.824+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:08.825217+0000) 2024-02-19T20:28:09.288 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:09.125+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:09.126095+0000) 2024-02-19T20:28:09.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:09.030+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:09.031619+0000) 2024-02-19T20:28:09.288 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:09.294+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:09.289 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:09.294+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:09.289 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:09.294+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:09.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:09.295+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:09.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:09.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:09.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:09.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:09.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:09.411+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:09.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:09.411+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:10.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:09.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:10.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:09.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:10.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:09.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:10.013 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:09.763+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:10.013 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:09.825+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:09.826941+0000) 2024-02-19T20:28:10.323 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:10.032+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:10.033305+0000) 2024-02-19T20:28:10.323 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:10.126+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:10.127693+0000) 2024-02-19T20:28:10.612 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:28:10.500+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61834 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:28:10.613 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:10.498+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:10.499266+0000) 2024-02-19T20:28:10.613 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:10.344+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:10.613 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:10.344+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:10.613 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:10.344+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:10.613 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:10.345+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:10.614 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:10.498+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:10.498985+0000) 2024-02-19T20:28:10.614 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:10.426+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:10.614 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:10.426+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:10.614 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:10.426+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:10.614 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:10.426+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:10.614 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:10.497+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:10.498689+0000) 2024-02-19T20:28:10.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:10.752+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:10.884 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:10.752+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:10.885 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:10.752+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:10.885 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:10.752+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:10.885 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:10.826+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:10.827130+0000) 2024-02-19T20:28:11.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:11.032+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:11.033523+0000) 2024-02-19T20:28:11.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:11.126+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:11.127908+0000) 2024-02-19T20:28:11.432 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:11.333+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:11.433 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:11.333+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:11.433 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:11.333+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:11.433 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:11.334+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:11.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:11.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:11.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:11.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:11.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:11.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:11.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:11.439+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:12.122 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:12.034+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:12.035930+0000) 2024-02-19T20:28:12.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:11.789+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:12.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:11.789+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:12.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:11.789+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:12.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:11.790+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:12.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:11.828+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:11.829565+0000) 2024-02-19T20:28:12.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:12.128+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:12.129948+0000) 2024-02-19T20:28:12.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:12.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:12.403 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:12.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:12.403 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:12.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:12.403 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:12.346+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:12.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:12.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:12.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:12.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:12.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:12.458+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:12.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:12.458+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:13.030 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:12.773+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:13.030 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:12.773+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:13.030 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:12.773+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:13.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:12.774+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:13.031 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:12.830+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:12.831754+0000) 2024-02-19T20:28:13.323 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:13.037+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:13.038010+0000) 2024-02-19T20:28:13.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:13.132+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:13.133200+0000) 2024-02-19T20:28:13.574 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:13.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:13.574 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:13.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:13.574 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:13.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:13.574 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:13.359+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:13.574 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:13.416+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:13.575 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:13.416+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:13.575 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:13.416+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:13.575 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:13.416+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:13.826 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:13.784+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:13.826 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:13.784+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:13.826 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:13.784+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:13.826 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:13.785+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:14.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:13.832+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:13.833799+0000) 2024-02-19T20:28:14.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:14.039+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:14.040429+0000) 2024-02-19T20:28:14.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:14.134+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:14.135284+0000) 2024-02-19T20:28:14.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:14.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:14.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:14.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:14.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:14.336+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:14.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:14.337+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:14.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:14.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:14.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:14.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:14.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:14.423+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:14.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:14.423+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:15.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:15.041+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:15.042321+0000) 2024-02-19T20:28:15.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:14.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:15.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:14.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:15.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:14.820+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:15.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:14.821+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:15.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:14.835+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:14.835993+0000) 2024-02-19T20:28:15.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:15.135+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:15.136417+0000) 2024-02-19T20:28:15.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:15.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:15.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:15.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:15.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:15.317+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:15.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:15.318+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:15.762 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:28:15.508+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61877 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:28:15.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:15.506+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:15.507388+0000) 2024-02-19T20:28:15.763 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:15.504+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:15.505465+0000) 2024-02-19T20:28:15.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:15.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:15.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:15.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:15.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:15.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:15.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:15.469+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:15.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:15.505+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:15.506089+0000) 2024-02-19T20:28:16.129 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:16.041+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:16.042494+0000) 2024-02-19T20:28:16.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:15.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:16.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:15.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:16.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:15.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:16.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:15.816+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:16.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:15.835+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:15.836174+0000) 2024-02-19T20:28:16.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:16.135+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:16.136663+0000) 2024-02-19T20:28:16.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:16.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:16.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:16.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:16.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:16.277+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:16.403 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:16.278+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:16.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:16.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:16.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:16.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:16.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:16.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:16.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:16.499+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:17.130 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:17.044+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:17.044994+0000) 2024-02-19T20:28:17.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:16.837+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:16.838127+0000) 2024-02-19T20:28:17.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:16.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:17.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:16.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:17.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:16.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:17.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:16.855+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:17.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:17.138+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:17.139000+0000) 2024-02-19T20:28:17.403 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:17.247+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:17.403 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:17.247+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:17.403 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:17.247+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:17.403 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:17.247+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:17.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:17.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:17.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:17.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:17.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:17.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:17.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:17.514+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:18.130 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:18.046+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:18.047027+0000) 2024-02-19T20:28:18.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:17.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:18.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:17.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:18.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:17.811+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:18.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:17.812+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:18.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:17.839+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:17.840212+0000) 2024-02-19T20:28:18.402 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:18.140+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:18.141626+0000) 2024-02-19T20:28:18.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:18.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:18.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:18.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:18.403 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:18.282+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:18.403 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:18.283+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:18.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:18.539+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:18.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:18.539+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:18.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:18.539+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:18.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:18.539+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:19.041 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:18.776+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:19.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:18.776+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:19.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:18.776+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:19.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:18.777+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:19.042 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:18.841+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:18.842186+0000) 2024-02-19T20:28:19.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:19.142+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:19.143798+0000) 2024-02-19T20:28:19.324 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:19.048+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:19.049088+0000) 2024-02-19T20:28:19.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:19.238+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:19.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:19.238+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:19.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:19.238+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:19.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:19.239+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:19.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:19.493+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:19.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:19.493+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:19.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:19.493+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:19.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:19.493+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:20.043 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:19.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:20.043 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:19.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:20.043 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:19.771+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:20.043 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:19.771+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:20.043 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:19.842+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:19.843674+0000) 2024-02-19T20:28:20.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:20.144+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:20.145173+0000) 2024-02-19T20:28:20.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:20.049+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:20.050733+0000) 2024-02-19T20:28:20.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:20.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:20.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:20.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:20.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:20.235+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:20.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:20.236+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:20.628 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:28:20.515+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61916 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:28:20.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:20.512+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:20.513740+0000) 2024-02-19T20:28:20.629 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:20.512+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:20.513845+0000) 2024-02-19T20:28:20.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:20.456+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:20.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:20.456+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:20.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:20.456+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:20.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:20.456+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:20.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:20.512+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:20.513079+0000) 2024-02-19T20:28:20.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:20.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:20.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:20.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:20.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:20.802+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:20.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:20.803+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:20.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:20.842+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:20.843881+0000) 2024-02-19T20:28:21.213 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:21.144+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:21.145361+0000) 2024-02-19T20:28:21.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:21.050+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:21.050955+0000) 2024-02-19T20:28:21.213 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:21.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:21.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:21.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:21.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:21.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:21.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:21.220+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:21.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:21.438+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:21.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:21.438+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:21.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:21.438+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:21.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:21.439+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:22.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:21.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:22.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:21.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:22.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:21.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:22.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:21.765+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:22.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:21.844+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:21.845908+0000) 2024-02-19T20:28:22.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:22.147+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:22.148931+0000) 2024-02-19T20:28:22.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:22.052+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:22.053017+0000) 2024-02-19T20:28:22.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:22.254+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:22.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:22.254+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:22.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:22.254+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:22.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:22.255+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:22.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:22.438+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:22.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:22.438+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:22.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:22.438+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:22.743 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:22.439+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:23.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:22.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:23.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:22.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:23.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:22.749+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:23.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:22.749+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:23.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:22.847+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:22.848015+0000) 2024-02-19T20:28:23.324 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:23.150+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:23.151298+0000) 2024-02-19T20:28:23.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:23.054+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:23.055090+0000) 2024-02-19T20:28:23.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:23.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:23.326 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:23.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:23.326 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:23.233+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:23.326 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:23.234+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:23.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:23.396+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:23.815 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:23.396+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:23.815 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:23.396+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:23.815 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:23.397+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:24.049 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:23.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:24.049 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:23.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:24.049 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:23.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:24.049 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:23.793+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:24.050 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:23.849+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:23.850408+0000) 2024-02-19T20:28:24.325 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:24.152+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:24.153812+0000) 2024-02-19T20:28:24.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:24.056+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:24.057089+0000) 2024-02-19T20:28:24.326 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:24.239+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:24.326 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:24.239+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:24.326 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:24.239+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:24.327 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:24.240+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:24.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:24.359+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:24.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:24.359+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:24.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:24.359+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:24.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:24.360+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:25.130 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:25.057+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:25.058758+0000) 2024-02-19T20:28:25.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:24.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:25.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:24.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:25.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:24.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:25.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:24.826+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:25.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:24.850+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:24.851694+0000) 2024-02-19T20:28:25.402 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:25.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:25.403 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:25.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:25.403 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:25.220+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:25.403 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:25.221+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:25.403 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:25.154+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:25.155579+0000) 2024-02-19T20:28:25.403 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:25.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:25.403 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:25.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:25.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:25.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:25.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:25.313+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:25.761 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:28:25.523+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61958 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:28:25.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:25.519+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:25.520902+0000) 2024-02-19T20:28:25.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:25.519+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:25.520457+0000) 2024-02-19T20:28:25.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:25.519+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:25.520862+0000) 2024-02-19T20:28:26.051 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:25.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:26.051 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:25.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:26.052 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:25.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:26.052 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:25.804+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:26.052 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:25.850+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:25.851884+0000) 2024-02-19T20:28:26.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:26.058+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:26.058973+0000) 2024-02-19T20:28:26.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:26.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:26.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:26.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:26.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:26.263+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:26.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:26.264+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:26.326 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:26.154+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:26.155781+0000) 2024-02-19T20:28:26.326 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:26.275+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:26.326 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:26.275+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:26.326 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:26.275+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:26.326 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:26.275+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:27.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:27.060+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:27.061296+0000) 2024-02-19T20:28:27.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:26.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:27.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:26.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:27.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:26.815+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:27.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:26.816+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:27.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:26.853+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:26.854429+0000) 2024-02-19T20:28:27.403 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:27.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:27.403 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:27.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:27.403 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:27.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:27.403 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:27.285+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:27.403 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:27.156+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:27.157850+0000) 2024-02-19T20:28:27.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:27.262+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:27.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:27.262+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:27.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:27.262+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:27.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:27.263+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:28.055 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:27.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:28.055 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:27.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:28.056 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:27.797+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:28.056 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:27.797+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:28.056 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:27.855+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:27.856865+0000) 2024-02-19T20:28:28.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:28.062+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:28.063332+0000) 2024-02-19T20:28:28.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:28.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:28.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:28.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:28.319 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:28.325+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:28.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:28.160+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:28.161328+0000) 2024-02-19T20:28:28.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:28.282+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:28.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:28.282+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:28.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:28.282+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:28.320 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:28.282+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:28.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:28.326+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:29.059 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:28.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:29.059 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:28.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:29.059 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:28.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:29.059 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:28.786+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:29.059 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:28.857+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:28.858850+0000) 2024-02-19T20:28:29.328 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:29.065+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:29.066832+0000) 2024-02-19T20:28:29.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:29.162+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:29.163723+0000) 2024-02-19T20:28:29.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:29.275+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:29.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:29.275+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:29.328 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:29.275+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:29.329 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:29.275+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:29.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:29.357+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:29.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:29.357+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:29.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:29.357+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:29.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:29.357+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:30.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:29.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:30.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:29.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:30.013 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:29.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:30.013 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:29.768+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:30.013 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:29.859+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:29.860373+0000) 2024-02-19T20:28:30.315 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:30.067+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:30.068565+0000) 2024-02-19T20:28:30.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:30.164+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:30.165083+0000) 2024-02-19T20:28:30.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:30.321+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:30.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:30.321+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:30.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:30.322+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:30.635 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:28:30.533+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 61997 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:28:30.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:30.529+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:30.530317+0000) 2024-02-19T20:28:30.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:30.322+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:30.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:30.530+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:30.531376+0000) 2024-02-19T20:28:30.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:30.400+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:30.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:30.400+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:30.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:30.400+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:30.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:30.401+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:30.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:30.530+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:30.531453+0000) 2024-02-19T20:28:30.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:30.773+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:30.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:30.773+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:30.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:30.773+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:30.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:30.774+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:30.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:30.859+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:30.860581+0000) 2024-02-19T20:28:31.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:31.067+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:31.068748+0000) 2024-02-19T20:28:31.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:31.164+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:31.165274+0000) 2024-02-19T20:28:31.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:31.347+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:31.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:31.347+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:31.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:31.347+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:31.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:31.348+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:31.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:31.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:31.763 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:31.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:31.763 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:31.419+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:31.763 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:31.420+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:32.064 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:31.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:32.064 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:31.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:32.064 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:31.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:32.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:31.817+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:32.065 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:31.861+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:31.862472+0000) 2024-02-19T20:28:32.314 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:32.070+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:32.071951+0000) 2024-02-19T20:28:32.314 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:32.166+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:32.167893+0000) 2024-02-19T20:28:32.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:32.406+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:32.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:32.406+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:32.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:32.406+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:32.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:32.407+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:32.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:32.321+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:32.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:32.321+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:32.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:32.321+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:32.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:32.321+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:33.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:33.073+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:33.074061+0000) 2024-02-19T20:28:33.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:32.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:33.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:32.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:33.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:32.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:33.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:32.837+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:33.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:32.865+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:32.866018+0000) 2024-02-19T20:28:33.403 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:33.169+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:33.170348+0000) 2024-02-19T20:28:33.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:33.335+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:33.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:33.335+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:33.405 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:33.335+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:33.405 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:33.336+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:33.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:33.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:33.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:33.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:33.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:33.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:33.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:33.416+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:34.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:34.075+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:34.076125+0000) 2024-02-19T20:28:34.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:33.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:34.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:33.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:34.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:33.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:34.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:33.831+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:34.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:33.867+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:33.868164+0000) 2024-02-19T20:28:34.383 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:34.171+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:34.172788+0000) 2024-02-19T20:28:34.383 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:34.305+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:34.383 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:34.305+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:34.384 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:34.305+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:34.384 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:34.306+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:34.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:34.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:34.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:34.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:34.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:34.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:34.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:34.391+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:35.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:35.076+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:35.077879+0000) 2024-02-19T20:28:35.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:34.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:35.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:34.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:35.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:34.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:35.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:34.856+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:35.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:34.868+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:34.869409+0000) 2024-02-19T20:28:35.403 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:35.173+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:35.174531+0000) 2024-02-19T20:28:35.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:35.330+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:35.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:35.330+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:35.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:35.330+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:35.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:35.331+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:35.761 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:28:35.540+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62039 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:28:35.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:35.537+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:35.538253+0000) 2024-02-19T20:28:35.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:35.537+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:35.538899+0000) 2024-02-19T20:28:35.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:35.423+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:35.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:35.423+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:35.763 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:35.423+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:35.763 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:35.424+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:35.763 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:35.536+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:35.537848+0000) 2024-02-19T20:28:36.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:36.077+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:36.078044+0000) 2024-02-19T20:28:36.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:35.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:36.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:35.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:36.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:35.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:36.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:35.862+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:36.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:35.868+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:35.869594+0000) 2024-02-19T20:28:36.383 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:36.173+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:36.174734+0000) 2024-02-19T20:28:36.384 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:36.362+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:36.384 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:36.362+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:36.384 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:36.362+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:36.384 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:36.363+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:36.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:36.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:36.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:36.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:36.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:36.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:36.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:36.391+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:37.073 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:36.871+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:36.872963+0000) 2024-02-19T20:28:37.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:36.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:37.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:36.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:37.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:36.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:37.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:36.879+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:37.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:37.080+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:37.081477+0000) 2024-02-19T20:28:37.326 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:37.177+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:37.178359+0000) 2024-02-19T20:28:37.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:37.369+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:37.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:37.369+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:37.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:37.369+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:37.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:37.369+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:37.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:37.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:37.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:37.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:37.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:37.408+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:37.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:37.408+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:38.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:38.082+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:38.083711+0000) 2024-02-19T20:28:38.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:37.875+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:37.876215+0000) 2024-02-19T20:28:38.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:37.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:38.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:37.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:38.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:37.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:38.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:37.908+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:38.403 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:38.179+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:38.180419+0000) 2024-02-19T20:28:38.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:38.330+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:38.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:38.330+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:38.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:38.330+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:38.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:38.331+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:38.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:38.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:38.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:38.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:38.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:38.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:38.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:38.440+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:39.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:39.086+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:39.087105+0000) 2024-02-19T20:28:39.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:38.877+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:38.878488+0000) 2024-02-19T20:28:39.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:38.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:39.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:38.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:39.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:38.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:39.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:38.889+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:39.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:39.181+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:39.182885+0000) 2024-02-19T20:28:39.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:39.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:39.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:39.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:39.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:39.296+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:39.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:39.297+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:39.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:39.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:39.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:39.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:39.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:39.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:39.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:39.490+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:40.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:40.088+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:40.089296+0000) 2024-02-19T20:28:40.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:39.878+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:39.879734+0000) 2024-02-19T20:28:40.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:39.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:40.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:39.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:40.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:39.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:40.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:39.933+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:40.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:40.183+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:40.184633+0000) 2024-02-19T20:28:40.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:40.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:40.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:40.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:40.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:40.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:40.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:40.276+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:40.658 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:40.545+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:40.546709+0000) 2024-02-19T20:28:40.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:40.544+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:40.545587+0000) 2024-02-19T20:28:40.659 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:28:40.547+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62078 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:28:40.659 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:40.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:40.660 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:40.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:40.660 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:40.471+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:40.660 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:40.471+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:40.660 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:40.545+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:40.546062+0000) 2024-02-19T20:28:40.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:40.878+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:40.879931+0000) 2024-02-19T20:28:40.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:40.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:40.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:40.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:40.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:40.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:40.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:40.910+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:41.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:41.088+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:41.089494+0000) 2024-02-19T20:28:41.262 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:41.183+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:41.184824+0000) 2024-02-19T20:28:41.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:41.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:41.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:41.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:41.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:41.312+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:41.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:41.313+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:41.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:41.477+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:41.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:41.477+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:41.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:41.477+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:41.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:41.477+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:42.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:42.090+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:42.091953+0000) 2024-02-19T20:28:42.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:41.881+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:41.882020+0000) 2024-02-19T20:28:42.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:41.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:42.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:41.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:42.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:41.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:42.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:41.886+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:42.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:42.187+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:42.188108+0000) 2024-02-19T20:28:42.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:42.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:42.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:42.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:42.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:42.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:42.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:42.351+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:42.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:42.503+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:42.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:42.503+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:42.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:42.503+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:42.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:42.504+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:43.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:43.094+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:43.095260+0000) 2024-02-19T20:28:43.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:42.883+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:42.884218+0000) 2024-02-19T20:28:43.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:42.901+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:43.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:42.901+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:43.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:42.901+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:43.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:42.901+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:43.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:43.190+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:43.191625+0000) 2024-02-19T20:28:43.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:43.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:43.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:43.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:43.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:43.327+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:43.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:43.328+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:43.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:43.481+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:43.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:43.481+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:43.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:43.481+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:43.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:43.481+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:44.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:43.885+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:43.886754+0000) 2024-02-19T20:28:44.089 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:43.949+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:44.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:43.949+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:44.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:43.949+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:44.090 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:43.951+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:44.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:44.192+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:44.193823+0000) 2024-02-19T20:28:44.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:44.096+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:44.097652+0000) 2024-02-19T20:28:44.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:44.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:44.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:44.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:44.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:44.285+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:44.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:44.286+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:44.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:44.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:44.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:44.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:44.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:44.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:44.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:44.458+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:45.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:45.098+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:45.099369+0000) 2024-02-19T20:28:45.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:44.887+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:44.888220+0000) 2024-02-19T20:28:45.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:44.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:45.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:44.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:45.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:44.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:45.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:44.904+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:45.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:45.195+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:45.196077+0000) 2024-02-19T20:28:45.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:45.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:45.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:45.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:45.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:45.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:45.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:45.289+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:45.666 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:28:45.555+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62120 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:28:45.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:45.551+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:45.552902+0000) 2024-02-19T20:28:45.667 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:45.552+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:45.553524+0000) 2024-02-19T20:28:45.667 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:45.414+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:45.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:45.414+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:45.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:45.414+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:45.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:45.415+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:45.668 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:45.551+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:45.552730+0000) 2024-02-19T20:28:45.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:45.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:45.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:45.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:45.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:45.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:45.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:45.863+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:45.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:45.887+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:45.888394+0000) 2024-02-19T20:28:46.230 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:46.195+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:46.196261+0000) 2024-02-19T20:28:46.230 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:46.098+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:46.099603+0000) 2024-02-19T20:28:46.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:46.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:46.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:46.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:46.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:46.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:46.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:46.327+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:46.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:46.413+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:46.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:46.413+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:46.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:46.413+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:46.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:46.414+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:47.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:47.100+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:47.101626+0000) 2024-02-19T20:28:47.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:46.889+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:46.890410+0000) 2024-02-19T20:28:47.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:46.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:47.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:46.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:47.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:46.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:47.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:46.909+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:47.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:47.353+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:47.404 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:47.353+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:47.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:47.353+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:47.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:47.353+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:47.405 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:47.198+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:47.199027+0000) 2024-02-19T20:28:47.405 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:47.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:47.405 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:47.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:47.405 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:47.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:47.406 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:47.372+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:48.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:48.103+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:48.104256+0000) 2024-02-19T20:28:48.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:47.892+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:47.893071+0000) 2024-02-19T20:28:48.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:47.919+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:48.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:47.919+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:48.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:47.919+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:48.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:47.920+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:48.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:48.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:48.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:48.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:48.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:48.347+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:48.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:48.347+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:48.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:48.201+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:48.201043+0000) 2024-02-19T20:28:48.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:48.348+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:48.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:48.348+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:48.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:48.348+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:48.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:48.349+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:49.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:49.105+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:49.106889+0000) 2024-02-19T20:28:49.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:48.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:49.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:48.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:49.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:48.889+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:49.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:48.889+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:49.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:48.894+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:48.895480+0000) 2024-02-19T20:28:49.383 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:49.203+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:49.204641+0000) 2024-02-19T20:28:49.384 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:49.332+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:49.384 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:49.332+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:49.384 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:49.332+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:49.384 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:49.333+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:49.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:49.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:49.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:49.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:49.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:49.390+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:49.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:49.390+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:50.100 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:49.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:50.100 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:49.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:50.100 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:49.842+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:50.100 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:49.843+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:50.100 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:49.896+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:49.896978+0000) 2024-02-19T20:28:50.365 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:50.107+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:50.108250+0000) 2024-02-19T20:28:50.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:50.205+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:50.206175+0000) 2024-02-19T20:28:50.675 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:28:50.563+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62159 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:28:50.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:50.560+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:50.561570+0000) 2024-02-19T20:28:50.676 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:50.430+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:50.676 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:50.430+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:50.676 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:50.430+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:50.676 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:50.431+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:50.676 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:50.559+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:50.560568+0000) 2024-02-19T20:28:50.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:50.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:50.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:50.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:50.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:50.372+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:50.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:50.373+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:50.677 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:50.560+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:50.561524+0000) 2024-02-19T20:28:50.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:50.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:50.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:50.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:50.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:50.874+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:50.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:50.875+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:50.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:50.896+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:50.897181+0000) 2024-02-19T20:28:51.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:51.107+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:51.108430+0000) 2024-02-19T20:28:51.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:51.205+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:51.206330+0000) 2024-02-19T20:28:51.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:51.383+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:51.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:51.383+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:51.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:51.383+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:51.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:51.384+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:51.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:51.398+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:51.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:51.398+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:51.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:51.398+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:51.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:51.398+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:52.133 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:52.109+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:52.110417+0000) 2024-02-19T20:28:52.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:51.884+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:52.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:51.884+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:52.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:51.884+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:52.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:51.885+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:52.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:51.898+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:51.899168+0000) 2024-02-19T20:28:52.404 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:52.207+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:52.208722+0000) 2024-02-19T20:28:52.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:52.442+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:52.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:52.442+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:52.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:52.442+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:52.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:52.442+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:52.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:52.429+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:52.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:52.429+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:52.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:52.429+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:52.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:52.429+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:53.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:53.111+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:53.112427+0000) 2024-02-19T20:28:53.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:52.899+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:53.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:52.899+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:53.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:52.899+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:53.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:52.900+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:52.901179+0000) 2024-02-19T20:28:53.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:52.900+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:53.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:53.210+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:53.211204+0000) 2024-02-19T20:28:53.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:53.477+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:53.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:53.477+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:53.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:53.477+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:53.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:53.478+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:53.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:53.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:53.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:53.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:53.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:53.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:53.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:53.479+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:54.106 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:53.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:54.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:53.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:54.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:53.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:54.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:53.861+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:54.107 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:53.902+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:53.903884+0000) 2024-02-19T20:28:54.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:54.113+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:54.114845+0000) 2024-02-19T20:28:54.405 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:54.212+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:54.213267+0000) 2024-02-19T20:28:54.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:54.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:54.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:54.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:54.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:54.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:54.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:54.462+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:54.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:54.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:54.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:54.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:54.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:54.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:54.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:54.472+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:55.133 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:55.115+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:55.116599+0000) 2024-02-19T20:28:55.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:54.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:55.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:54.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:55.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:54.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:55.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:54.889+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:55.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:54.904+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:54.905262+0000) 2024-02-19T20:28:55.405 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:55.213+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:55.214607+0000) 2024-02-19T20:28:55.683 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:28:55.571+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62201 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:28:55.684 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:55.567+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:55.568859+0000) 2024-02-19T20:28:55.684 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:55.506+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:55.684 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:55.506+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:55.684 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:55.506+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:55.685 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:55.507+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:55.685 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:55.567+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:55.568193+0000) 2024-02-19T20:28:55.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:55.443+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:55.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:55.443+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:55.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:55.443+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:55.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:55.443+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:55.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:55.567+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:55.568676+0000) 2024-02-19T20:28:55.987 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:55.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:55.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:55.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:55.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:55.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:55.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:55.899+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:55.988 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:55.904+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:55.905479+0000) 2024-02-19T20:28:56.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:56.115+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:56.116828+0000) 2024-02-19T20:28:56.261 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:56.213+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:56.214809+0000) 2024-02-19T20:28:56.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:56.472+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:56.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:56.472+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:56.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:56.472+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:56.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:56.472+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:56.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:56.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:56.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:56.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:56.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:56.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:56.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:56.487+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:57.133 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:57.118+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:57.119354+0000) 2024-02-19T20:28:57.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:56.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:57.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:56.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:57.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:56.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:57.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:56.874+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:57.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:56.906+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:56.907501+0000) 2024-02-19T20:28:57.405 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:57.216+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:57.216953+0000) 2024-02-19T20:28:57.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:57.495+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:57.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:57.495+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:57.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:57.495+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:57.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:57.496+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:57.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:57.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:57.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:57.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:57.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:57.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:57.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:57.511+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:58.133 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:58.120+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:58.121439+0000) 2024-02-19T20:28:58.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:57.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:58.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:57.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:58.134 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:57.876+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:58.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:57.877+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:58.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:57.910+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:57.910998+0000) 2024-02-19T20:28:58.405 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:58.219+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:58.220419+0000) 2024-02-19T20:28:58.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:58.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:58.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:58.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:58.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:58.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:58.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:58.480+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:58.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:58.507+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:58.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:58.507+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:58.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:58.507+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:58.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:58.507+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:28:59.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:59.123+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:59.124845+0000) 2024-02-19T20:28:59.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:58.911+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:58.912966+0000) 2024-02-19T20:28:59.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:58.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:59.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:58.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:59.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:58.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:28:59.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:58.923+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:59.483 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:59.222+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:59.223911+0000) 2024-02-19T20:28:59.483 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:59.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:59.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:59.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:59.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:59.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:28:59.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:28:59.491+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:28:59.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:59.521+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:59.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:59.521+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:59.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:59.521+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:28:59.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:28:59.522+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:00.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:00.126+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:00.127161+0000) 2024-02-19T20:29:00.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:59.913+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:59.914286+0000) 2024-02-19T20:29:00.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:59.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:00.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:59.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:00.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:59.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:00.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:28:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:28:59.940+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:00.497 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:00.225+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:00.226124+0000) 2024-02-19T20:29:00.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:00.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:00.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:00.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:00.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:00.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:00.761 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:29:00.579+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62240 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:29:00.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:00.577+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:00.578034+0000) 2024-02-19T20:29:00.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:00.505+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:00.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:00.575+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:00.576098+0000) 2024-02-19T20:29:00.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:00.540+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:00.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:00.540+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:00.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:00.540+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:00.763 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:00.540+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:00.763 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:00.575+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:00.576579+0000) 2024-02-19T20:29:01.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:01.126+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:01.127382+0000) 2024-02-19T20:29:01.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:00.913+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:00.914453+0000) 2024-02-19T20:29:01.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:00.920+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:01.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:00.920+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:01.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:00.920+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:01.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:00.921+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:01.505 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:01.225+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:01.226341+0000) 2024-02-19T20:29:01.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:01.544+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:01.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:01.544+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:01.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:01.544+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:01.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:01.544+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:01.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:01.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:01.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:01.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:01.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:01.512+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:01.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:01.513+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:02.220 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:02.128+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:02.129630+0000) 2024-02-19T20:29:02.220 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:01.916+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:01.917866+0000) 2024-02-19T20:29:02.220 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:01.968+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:02.220 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:01.968+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:02.220 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:01.968+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:02.221 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:01.969+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:02.511 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:02.227+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:02.228392+0000) 2024-02-19T20:29:02.511 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:02.470+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:02.511 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:02.470+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:02.511 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:02.470+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:02.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:02.471+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:02.911 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:02.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:02.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:02.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:02.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:02.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:02.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:02.591+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:03.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:03.130+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:03.131739+0000) 2024-02-19T20:29:03.223 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:02.919+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:02.920160+0000) 2024-02-19T20:29:03.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:02.952+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:03.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:02.952+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:03.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:02.952+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:03.224 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:02.953+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:03.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:03.231+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:03.232003+0000) 2024-02-19T20:29:03.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:03.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:03.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:03.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:03.498 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:03.505+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:03.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:03.505+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:03.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:03.557+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:03.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:03.557+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:03.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:03.557+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:03.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:03.558+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:04.126 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:03.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:04.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:03.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:04.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:03.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:04.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:03.908+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:04.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:03.921+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:03.922288+0000) 2024-02-19T20:29:04.405 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:04.134+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:04.135272+0000) 2024-02-19T20:29:04.406 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:04.234+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:04.235474+0000) 2024-02-19T20:29:04.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:04.558+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:04.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:04.558+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:04.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:04.558+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:04.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:04.559+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:04.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:04.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:04.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:04.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:04.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:04.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:04.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:04.488+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:05.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:05.136+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:05.137274+0000) 2024-02-19T20:29:05.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:04.897+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:05.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:04.897+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:05.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:04.897+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:05.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:04.898+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:05.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:04.922+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:04.923822+0000) 2024-02-19T20:29:05.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:05.236+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:05.237659+0000) 2024-02-19T20:29:05.459 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:05.467+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:05.460 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:05.467+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:05.761 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:29:05.587+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62282 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:29:05.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:05.585+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:05.586303+0000) 2024-02-19T20:29:05.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:05.525+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:05.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:05.525+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:05.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:05.525+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:05.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:05.525+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:05.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:05.583+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:05.584707+0000) 2024-02-19T20:29:05.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:05.467+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:05.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:05.467+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:05.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:05.583+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:05.584965+0000) 2024-02-19T20:29:06.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:05.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:06.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:05.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:06.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:05.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:06.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:05.872+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:06.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:05.923+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:05.924020+0000) 2024-02-19T20:29:06.406 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:06.136+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:06.137495+0000) 2024-02-19T20:29:06.406 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:06.236+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:06.237868+0000) 2024-02-19T20:29:06.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:06.446+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:06.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:06.446+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:06.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:06.446+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:06.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:06.446+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:06.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:06.486+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:06.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:06.486+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:06.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:06.486+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:06.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:06.487+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:07.158 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:07.138+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:07.139872+0000) 2024-02-19T20:29:07.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:06.901+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:07.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:06.901+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:07.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:06.901+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:07.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:06.902+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:07.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:06.926+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:06.927493+0000) 2024-02-19T20:29:07.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:07.239+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:07.240036+0000) 2024-02-19T20:29:07.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:07.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:07.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:07.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:07.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:07.415+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:07.470 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:07.416+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:07.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:07.477+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:07.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:07.477+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:07.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:07.477+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:07.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:07.478+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:08.233 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:08.140+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:08.141947+0000) 2024-02-19T20:29:08.234 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:07.929+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:07.930683+0000) 2024-02-19T20:29:08.234 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:07.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:08.234 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:07.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:08.234 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:07.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:08.234 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:07.943+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:08.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:08.470+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:08.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:08.470+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:08.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:08.470+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:08.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:08.471+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:08.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:08.241+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:08.242173+0000) 2024-02-19T20:29:08.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:08.443+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:08.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:08.443+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:08.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:08.443+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:08.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:08.444+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:09.237 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:09.143+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:09.144601+0000) 2024-02-19T20:29:09.237 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:08.931+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:08.932756+0000) 2024-02-19T20:29:09.237 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:08.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:09.238 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:08.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:09.238 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:08.982+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:09.238 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:08.983+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:09.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:09.422+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:09.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:09.422+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:09.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:09.422+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:09.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:09.423+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:09.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:09.244+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:09.245683+0000) 2024-02-19T20:29:09.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:09.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:09.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:09.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:09.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:09.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:09.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:09.487+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:10.238 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:10.145+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:10.146109+0000) 2024-02-19T20:29:10.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:09.933+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:09.934284+0000) 2024-02-19T20:29:10.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:09.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:10.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:09.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:10.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:09.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:10.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:09.957+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:10.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:10.376+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:10.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:10.376+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:10.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:10.376+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:10.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:10.376+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:10.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:10.246+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:10.247002+0000) 2024-02-19T20:29:10.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:10.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:10.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:10.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:10.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:10.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:10.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:10.483+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:10.926 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:29:10.594+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62322 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:29:10.927 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:10.590+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:10.591578+0000) 2024-02-19T20:29:10.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:10.591+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:10.592343+0000) 2024-02-19T20:29:10.928 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:10.590+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:10.591709+0000) 2024-02-19T20:29:11.238 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:11.145+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:11.146306+0000) 2024-02-19T20:29:11.238 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:10.933+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:10.934467+0000) 2024-02-19T20:29:11.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:10.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:11.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:10.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:11.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:10.971+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:11.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:10.972+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:11.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:11.374+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:11.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:11.374+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:11.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:11.374+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:11.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:11.375+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:11.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:11.246+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:11.247171+0000) 2024-02-19T20:29:11.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:11.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:11.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:11.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:11.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:11.439+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:11.513 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:11.439+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:12.242 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:12.148+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:12.149638+0000) 2024-02-19T20:29:12.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:11.936+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:11.937562+0000) 2024-02-19T20:29:12.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:11.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:12.242 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:11.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:12.243 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:11.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:12.243 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:11.991+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:12.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:12.343+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:12.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:12.343+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:12.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:12.343+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:12.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:12.344+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:12.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:12.249+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:12.250481+0000) 2024-02-19T20:29:12.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:12.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:12.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:12.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:12.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:12.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:12.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:12.475+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:13.244 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:13.152+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:13.152997+0000) 2024-02-19T20:29:13.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:12.939+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:12.940574+0000) 2024-02-19T20:29:13.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:13.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:13.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:13.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:13.245 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:13.025+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:13.245 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:13.026+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:13.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:13.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:13.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:13.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:13.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:13.342+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:13.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:13.343+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:13.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:13.251+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:13.252737+0000) 2024-02-19T20:29:13.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:13.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:13.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:13.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:13.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:13.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:13.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:13.440+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:14.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:13.942+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:13.943086+0000) 2024-02-19T20:29:14.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:14.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:14.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:14.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:14.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:14.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:14.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:14.012+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:14.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:14.254+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:14.254988+0000) 2024-02-19T20:29:14.407 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:14.155+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:14.156369+0000) 2024-02-19T20:29:14.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:14.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:14.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:14.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:14.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:14.359+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:14.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:14.360+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:14.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:14.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:14.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:14.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:14.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:14.482+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:14.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:14.483+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:15.247 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:15.156+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:15.157622+0000) 2024-02-19T20:29:15.248 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:14.944+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:14.945013+0000) 2024-02-19T20:29:15.248 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:14.968+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:15.248 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:14.968+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:15.248 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:14.968+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:15.248 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:14.969+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:15.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:15.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:15.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:15.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:15.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:15.348+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:15.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:15.348+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:15.511 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:15.255+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:15.256338+0000) 2024-02-19T20:29:15.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:15.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:15.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:15.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:15.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:15.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:15.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:15.488+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:15.936 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:29:15.601+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62364 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:29:15.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:15.599+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:15.600013+0000) 2024-02-19T20:29:15.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:15.599+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:15.600371+0000) 2024-02-19T20:29:15.937 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:15.599+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:15.600207+0000) 2024-02-19T20:29:16.248 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:16.156+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:16.157833+0000) 2024-02-19T20:29:16.248 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:15.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:16.248 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:15.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:16.248 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:15.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:16.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:15.944+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:15.945198+0000) 2024-02-19T20:29:16.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:15.944+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:16.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:16.255+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:16.256568+0000) 2024-02-19T20:29:16.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:16.363+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:16.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:16.363+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:16.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:16.363+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:16.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:16.364+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:16.904 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:16.521+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:16.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:16.521+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:16.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:16.521+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:16.905 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:16.522+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:17.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:17.160+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:17.161302+0000) 2024-02-19T20:29:17.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:16.912+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:17.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:16.912+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:17.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:16.912+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:17.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:16.913+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:17.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:16.947+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:16.948586+0000) 2024-02-19T20:29:17.469 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:17.257+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:17.258890+0000) 2024-02-19T20:29:17.469 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:17.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:17.469 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:17.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:17.470 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:17.350+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:17.470 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:17.351+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:17.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:17.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:17.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:17.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:17.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:17.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:17.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:17.477+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:18.252 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:18.162+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:18.163468+0000) 2024-02-19T20:29:18.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:17.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:18.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:17.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:18.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:17.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:18.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:17.927+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:18.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:17.951+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:17.952063+0000) 2024-02-19T20:29:18.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:18.399+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:18.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:18.399+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:18.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:18.399+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:18.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:18.400+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:18.511 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:18.260+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:18.261077+0000) 2024-02-19T20:29:18.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:18.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:18.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:18.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:18.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:18.486+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:18.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:18.487+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:19.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:18.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:19.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:18.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:19.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:18.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:19.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:18.910+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:19.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:18.954+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:18.955599+0000) 2024-02-19T20:29:19.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:19.263+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:19.264251+0000) 2024-02-19T20:29:19.407 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:19.164+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:19.165453+0000) 2024-02-19T20:29:19.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:19.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:19.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:19.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:19.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:19.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:19.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:19.409+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:19.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:19.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:19.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:19.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:19.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:19.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:19.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:19.511+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:20.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:19.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:20.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:19.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:20.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:19.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:20.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:19.901+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:20.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:19.956+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:19.957886+0000) 2024-02-19T20:29:20.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:20.265+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:20.266191+0000) 2024-02-19T20:29:20.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:20.166+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:20.167191+0000) 2024-02-19T20:29:20.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:20.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:20.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:20.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:20.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:20.415+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:20.512 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:20.416+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:20.853 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:29:20.609+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62403 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:29:20.853 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:20.606+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:20.607020+0000) 2024-02-19T20:29:20.854 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:20.606+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:20.607291+0000) 2024-02-19T20:29:20.854 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:20.523+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:20.854 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:20.523+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:20.854 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:20.523+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:20.855 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:20.524+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:20.855 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:20.605+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:20.606390+0000) 2024-02-19T20:29:21.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:20.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:21.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:20.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:21.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:20.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:21.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:20.861+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:21.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:20.957+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:20.958064+0000) 2024-02-19T20:29:21.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:21.265+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:21.266367+0000) 2024-02-19T20:29:21.407 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:21.166+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:21.167390+0000) 2024-02-19T20:29:21.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:21.376+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:21.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:21.376+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:21.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:21.376+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:21.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:21.376+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:21.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:21.552+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:21.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:21.552+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:21.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:21.552+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:21.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:21.552+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:22.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:21.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:22.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:21.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:22.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:21.908+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:22.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:21.909+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:22.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:21.959+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:21.960578+0000) 2024-02-19T20:29:22.502 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:22.267+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:22.268881+0000) 2024-02-19T20:29:22.502 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:22.168+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:22.169790+0000) 2024-02-19T20:29:22.502 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:22.399+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:22.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:22.399+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:22.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:22.400+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:22.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:22.400+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:22.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:22.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:22.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:22.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:22.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:22.510+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:22.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:22.510+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:23.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:22.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:23.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:22.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:23.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:22.869+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:23.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:22.870+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:23.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:22.962+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:22.963184+0000) 2024-02-19T20:29:23.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:23.270+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:23.271280+0000) 2024-02-19T20:29:23.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:23.172+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:23.173115+0000) 2024-02-19T20:29:23.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:23.364+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:23.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:23.364+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:23.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:23.364+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:23.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:23.365+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:23.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:23.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:23.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:23.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:23.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:23.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:23.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:23.507+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:24.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:23.890+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:24.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:23.891+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:24.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:23.891+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:24.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:23.892+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:24.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:23.964+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:23.965488+0000) 2024-02-19T20:29:24.407 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:24.272+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:24.273492+0000) 2024-02-19T20:29:24.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:24.175+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:24.176564+0000) 2024-02-19T20:29:24.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:24.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:24.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:24.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:24.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:24.320+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:24.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:24.321+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:24.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:24.533+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:24.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:24.533+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:24.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:24.533+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:24.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:24.533+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:25.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:24.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:25.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:24.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:25.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:24.893+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:25.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:24.895+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:25.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:24.966+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:24.967140+0000) 2024-02-19T20:29:25.408 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:25.274+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:25.275956+0000) 2024-02-19T20:29:25.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:25.177+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:25.178865+0000) 2024-02-19T20:29:25.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:25.289+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:25.408 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:25.289+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:25.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:25.289+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:25.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:25.290+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:25.727 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:29:25.617+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62445 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:29:25.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:25.613+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:25.614427+0000) 2024-02-19T20:29:25.728 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:25.614+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:25.614976+0000) 2024-02-19T20:29:25.728 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:25.564+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:25.728 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:25.564+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:25.728 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:25.564+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:25.728 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:25.564+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:25.728 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:25.614+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:25.615603+0000) 2024-02-19T20:29:26.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:25.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:26.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:25.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:26.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:25.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:26.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:25.936+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:26.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:25.966+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:25.967332+0000) 2024-02-19T20:29:26.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:26.178+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:26.179033+0000) 2024-02-19T20:29:26.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:26.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:26.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:26.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:26.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:26.257+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:26.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:26.257+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:26.596 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:26.275+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:26.276136+0000) 2024-02-19T20:29:26.901 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:26.604+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:26.902 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:26.604+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:26.902 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:26.604+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:26.902 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:26.604+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:27.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:26.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:27.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:26.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:27.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:26.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:27.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:26.910+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:27.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:26.969+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:26.970760+0000) 2024-02-19T20:29:27.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:27.180+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:27.181744+0000) 2024-02-19T20:29:27.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:27.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:27.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:27.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:27.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:27.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:27.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:27.275+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:27.512 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:27.277+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:27.278554+0000) 2024-02-19T20:29:27.888 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:27.578+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:27.888 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:27.578+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:27.888 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:27.578+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:27.889 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:27.579+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:28.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:27.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:28.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:27.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:28.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:27.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:28.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:27.897+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:28.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:27.973+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:27.974212+0000) 2024-02-19T20:29:28.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:28.279+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:28.280637+0000) 2024-02-19T20:29:28.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:28.182+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:28.183948+0000) 2024-02-19T20:29:28.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:28.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:28.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:28.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:28.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:28.283+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:28.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:28.283+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:28.928 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:28.614+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:28.928 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:28.614+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:28.928 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:28.614+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:28.928 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:28.615+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:29.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:28.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:29.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:28.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:29.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:28.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:29.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:28.936+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:29.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:28.975+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:28.976727+0000) 2024-02-19T20:29:29.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:29.283+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:29.284272+0000) 2024-02-19T20:29:29.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:29.185+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:29.186919+0000) 2024-02-19T20:29:29.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:29.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:29.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:29.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:29.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:29.331+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:29.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:29.332+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:29.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:29.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:29.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:29.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:29.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:29.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:29.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:29.572+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:30.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:30.188+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:30.189166+0000) 2024-02-19T20:29:30.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:29.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:30.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:29.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:30.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:29.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:30.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:29.973+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:30.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:29.978+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:29.979108+0000) 2024-02-19T20:29:30.560 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:30.284+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:30.285663+0000) 2024-02-19T20:29:30.560 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:30.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:30.560 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:30.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:30.561 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:30.345+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:30.561 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:30.346+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:30.561 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:30.346+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:30.966 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:29:30.624+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62484 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:29:30.966 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:30.621+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:30.622070+0000) 2024-02-19T20:29:30.967 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:30.621+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:30.622001+0000) 2024-02-19T20:29:30.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:30.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:30.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:30.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:30.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:30.568+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:30.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:30.621+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:30.622877+0000) 2024-02-19T20:29:31.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:31.188+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:31.189358+0000) 2024-02-19T20:29:31.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:30.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:31.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:30.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:31.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:30.974+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:31.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:30.975+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:31.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:30.978+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:30.979295+0000) 2024-02-19T20:29:31.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:31.369+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:31.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:31.369+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:31.542 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:31.369+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:31.543 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:31.370+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:31.543 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:31.284+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:31.285835+0000) 2024-02-19T20:29:31.543 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:31.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:31.543 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:31.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:31.543 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:31.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:31.918 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:31.550+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:32.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:31.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:32.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:31.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:32.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:31.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:32.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:31.927+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:32.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:31.980+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:31.981900+0000) 2024-02-19T20:29:32.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:32.287+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:32.288288+0000) 2024-02-19T20:29:32.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:32.191+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:32.191989+0000) 2024-02-19T20:29:32.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:32.369+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:32.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:32.369+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:32.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:32.369+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:32.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:32.369+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:32.891 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:32.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:32.891 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:32.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:32.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:32.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:32.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:32.527+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:33.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:32.899+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:33.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:32.899+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:33.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:32.899+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:33.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:32.900+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:33.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:32.983+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:32.984327+0000) 2024-02-19T20:29:33.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:33.289+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:33.290418+0000) 2024-02-19T20:29:33.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:33.194+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:33.195500+0000) 2024-02-19T20:29:33.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:33.357+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:33.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:33.357+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:33.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:33.357+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:33.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:33.357+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:33.836 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:33.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:33.836 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:33.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:33.837 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:33.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:33.837 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:33.570+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:34.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:33.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:34.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:33.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:34.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:33.875+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:34.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:33.876+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:34.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:33.985+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:33.986894+0000) 2024-02-19T20:29:34.408 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:34.292+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:34.292973+0000) 2024-02-19T20:29:34.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:34.197+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:34.198774+0000) 2024-02-19T20:29:34.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:34.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:34.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:34.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:34.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:34.319+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:34.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:34.320+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:34.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:34.616+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:34.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:34.616+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:34.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:34.616+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:34.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:34.617+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:35.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:34.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:35.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:34.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:35.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:34.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:35.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:34.850+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:35.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:34.987+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:34.988775+0000) 2024-02-19T20:29:35.408 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:35.293+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:35.294813+0000) 2024-02-19T20:29:35.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:35.199+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:35.200397+0000) 2024-02-19T20:29:35.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:35.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:35.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:35.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:35.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:35.366+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:35.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:35.367+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:35.743 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:29:35.631+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62525 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:29:35.744 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:35.629+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:35.630413+0000) 2024-02-19T20:29:35.744 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:35.628+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:35.629647+0000) 2024-02-19T20:29:35.745 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:35.628+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:35.629408+0000) 2024-02-19T20:29:35.745 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:35.656+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:35.745 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:35.656+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:35.745 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:35.656+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:35.745 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:35.656+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:36.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:35.891+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:36.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:35.891+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:36.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:35.891+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:36.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:35.892+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:36.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:35.987+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:35.988971+0000) 2024-02-19T20:29:36.286 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:36.199+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:36.200608+0000) 2024-02-19T20:29:36.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:36.294+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:36.295029+0000) 2024-02-19T20:29:36.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:36.648+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:36.642 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:36.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:36.642 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:36.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:36.642 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:36.349+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:36.643 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:36.350+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:36.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:36.648+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:36.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:36.648+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:36.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:36.649+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:36.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:36.867+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:36.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:36.867+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:36.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:36.867+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:36.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:36.868+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:37.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:36.990+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:36.992518+0000) 2024-02-19T20:29:37.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:37.202+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:37.203953+0000) 2024-02-19T20:29:37.637 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:37.296+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:37.297517+0000) 2024-02-19T20:29:37.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:37.391+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:37.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:37.391+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:37.638 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:37.391+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:37.638 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:37.392+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:37.899 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:37.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:37.899 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:37.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:37.900 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:37.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:37.900 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:37.645+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:38.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:37.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:38.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:37.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:38.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:37.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:38.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:37.908+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:38.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:37.993+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:37.994767+0000) 2024-02-19T20:29:38.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:38.298+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:38.299643+0000) 2024-02-19T20:29:38.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:38.205+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:38.206358+0000) 2024-02-19T20:29:38.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:38.418+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:38.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:38.418+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:38.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:38.418+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:38.511 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:38.419+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:38.875 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:38.621+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:38.875 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:38.621+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:38.875 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:38.621+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:38.875 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:38.622+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:39.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:38.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:39.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:38.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:39.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:38.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:39.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:38.883+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:39.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:38.995+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:38.996951+0000) 2024-02-19T20:29:39.406 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:39.207+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:39.208907+0000) 2024-02-19T20:29:39.406 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:39.300+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:39.301749+0000) 2024-02-19T20:29:39.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:39.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:39.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:39.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:39.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:39.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:39.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:39.414+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:39.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:39.641+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:39.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:39.641+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:39.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:39.641+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:39.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:39.642+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:40.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:39.859+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:40.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:39.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:40.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:39.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:40.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:39.861+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:40.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:39.997+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:39.998767+0000) 2024-02-19T20:29:40.409 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:40.302+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:40.303373+0000) 2024-02-19T20:29:40.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:40.209+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:40.210353+0000) 2024-02-19T20:29:40.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:40.379+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:40.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:40.379+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:40.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:40.379+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:40.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:40.380+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:40.749 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:29:40.639+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62565 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:29:40.750 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:40.636+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:40.637295+0000) 2024-02-19T20:29:40.750 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:40.636+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:40.637068+0000) 2024-02-19T20:29:40.751 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:40.635+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:40.636731+0000) 2024-02-19T20:29:40.751 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:40.659+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:40.751 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:40.659+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:40.751 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:40.659+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:40.751 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:40.659+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:41.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:40.867+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:41.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:40.867+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:41.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:40.867+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:41.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:40.869+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:41.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:40.998+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:40.999039+0000) 2024-02-19T20:29:41.290 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:41.209+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:41.210529+0000) 2024-02-19T20:29:41.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:41.302+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:41.303540+0000) 2024-02-19T20:29:41.622 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:41.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:41.622 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:41.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:41.622 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:41.393+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:41.623 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:41.394+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:41.992 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:41.832+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:41.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:41.832+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:41.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:41.832+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:41.993 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:41.833+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:41.993 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:41.629+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:41.993 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:41.629+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:41.993 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:41.629+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:41.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:41.630+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:42.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:42.000+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:42.001625+0000) 2024-02-19T20:29:42.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:42.211+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:42.212756+0000) 2024-02-19T20:29:42.617 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:42.418+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:42.617 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:42.418+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:42.617 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:42.418+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:42.618 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:42.419+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:42.618 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:42.305+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:42.306198+0000) 2024-02-19T20:29:42.618 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:42.625+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:42.618 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:42.625+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:42.618 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:42.625+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:42.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:42.625+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:42.995 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:42.789+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:42.995 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:42.789+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:42.995 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:42.789+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:42.995 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:42.790+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:43.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:43.215+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:43.216291+0000) 2024-02-19T20:29:43.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:43.002+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:43.003766+0000) 2024-02-19T20:29:43.572 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:43.307+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:43.308318+0000) 2024-02-19T20:29:43.572 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:43.449+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:43.573 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:43.449+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:43.573 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:43.449+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:43.573 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:43.450+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:43.826 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:43.582+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:43.826 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:43.582+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:43.826 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:43.582+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:43.826 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:43.583+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:44.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:43.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:44.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:43.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:44.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:43.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:44.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:43.835+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:44.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:44.005+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:44.006024+0000) 2024-02-19T20:29:44.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:44.218+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:44.219869+0000) 2024-02-19T20:29:44.409 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:44.309+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:44.310678+0000) 2024-02-19T20:29:44.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:44.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:44.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:44.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:44.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:44.461+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:44.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:44.462+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:44.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:44.551+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:44.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:44.551+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:44.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:44.551+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:44.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:44.552+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:45.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:44.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:45.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:44.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:45.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:44.799+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:45.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:44.800+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:45.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:45.006+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:45.007583+0000) 2024-02-19T20:29:45.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:45.221+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:45.222214+0000) 2024-02-19T20:29:45.409 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:45.311+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:45.312198+0000) 2024-02-19T20:29:45.756 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:29:45.647+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62608 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:29:45.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:45.643+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:45.644759+0000) 2024-02-19T20:29:45.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:45.466+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:45.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:45.466+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:45.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:45.466+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:45.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:45.467+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:45.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:45.644+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:45.645138+0000) 2024-02-19T20:29:45.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:45.580+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:45.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:45.580+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:45.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:45.580+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:45.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:45.581+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:45.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:45.643+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:45.643993+0000) 2024-02-19T20:29:46.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:45.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:46.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:45.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:46.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:45.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:46.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:45.851+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:46.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:46.006+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:46.007768+0000) 2024-02-19T20:29:46.303 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:46.221+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:46.222373+0000) 2024-02-19T20:29:46.605 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:46.495+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:46.605 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:46.495+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:46.605 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:46.495+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:46.605 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:46.496+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:46.606 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:46.311+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:46.312383+0000) 2024-02-19T20:29:46.606 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:46.613+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:46.606 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:46.613+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:46.606 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:46.613+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:47.001 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:46.613+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:47.001 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:46.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:47.001 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:46.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:47.001 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:46.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:47.001 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:46.850+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:47.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:47.009+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:47.010022+0000) 2024-02-19T20:29:47.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:47.223+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:47.224729+0000) 2024-02-19T20:29:47.590 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:47.314+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:47.315962+0000) 2024-02-19T20:29:47.590 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:47.545+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:47.590 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:47.545+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:47.590 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:47.545+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:47.591 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:47.546+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:48.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:47.598+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:48.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:47.598+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:48.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:47.598+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:48.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:47.598+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:48.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:47.829+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:48.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:47.829+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:48.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:47.829+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:48.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:47.830+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:48.261 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:48.011+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:48.012324+0000) 2024-02-19T20:29:48.261 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:48.227+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:48.228274+0000) 2024-02-19T20:29:48.565 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:48.317+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:48.317991+0000) 2024-02-19T20:29:48.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:48.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:48.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:48.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:48.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:48.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:48.858 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:48.574+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:48.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:48.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:48.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:48.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:48.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:48.645+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:48.858 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:48.645+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:49.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:48.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:49.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:48.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:49.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:48.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:49.139 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:48.866+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:49.139 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:49.013+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:49.014718+0000) 2024-02-19T20:29:49.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:49.229+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:49.230350+0000) 2024-02-19T20:29:49.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:49.319+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:49.320121+0000) 2024-02-19T20:29:49.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:49.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:49.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:49.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:49.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:49.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:49.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:49.590+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:49.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:49.601+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:49.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:49.601+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:49.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:49.601+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:49.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:49.602+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:50.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:49.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:50.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:49.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:50.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:49.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:50.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:49.837+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:50.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:50.015+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:50.016202+0000) 2024-02-19T20:29:50.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:50.231+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:50.232164+0000) 2024-02-19T20:29:50.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:50.321+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:50.322056+0000) 2024-02-19T20:29:50.662 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:29:50.654+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62648 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:29:50.663 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:50.651+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:50.652584+0000) 2024-02-19T20:29:50.663 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:50.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:50.663 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:50.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:50.663 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:50.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:50.664 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:50.594+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:50.664 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:50.650+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:50.651706+0000) 2024-02-19T20:29:50.664 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:50.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:50.665 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:50.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:50.665 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:50.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:50.665 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:50.554+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:50.665 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:50.651+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:50.652047+0000) 2024-02-19T20:29:51.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:50.805+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:51.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:50.805+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:51.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:50.805+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:51.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:50.805+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:51.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:51.015+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:51.016381+0000) 2024-02-19T20:29:51.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:51.231+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:51.232354+0000) 2024-02-19T20:29:51.581 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:51.321+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:51.322224+0000) 2024-02-19T20:29:51.581 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:51.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:51.581 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:51.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:51.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:51.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:51.582 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:51.558+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:51.895 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:51.589+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:51.895 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:51.589+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:51.895 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:51.589+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:51.895 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:51.590+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:51.895 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:51.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:51.896 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:51.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:51.896 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:51.782+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:51.896 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:51.783+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:52.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:52.017+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:52.018627+0000) 2024-02-19T20:29:52.510 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:52.233+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:52.234546+0000) 2024-02-19T20:29:52.510 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:52.323+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:52.324869+0000) 2024-02-19T20:29:52.806 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:52.562+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:52.806 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:52.562+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:52.806 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:52.562+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:52.807 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:52.562+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:52.807 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:52.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:52.807 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:52.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:52.807 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:52.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:52.807 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:52.587+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:53.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:52.814+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:53.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:52.814+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:53.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:52.814+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:53.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:52.815+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:53.139 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:53.021+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:53.022135+0000) 2024-02-19T20:29:53.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:53.235+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:53.236753+0000) 2024-02-19T20:29:53.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:53.326+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:53.327095+0000) 2024-02-19T20:29:53.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:53.560+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:53.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:53.560+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:53.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:53.560+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:53.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:53.560+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:53.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:53.594+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:53.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:53.594+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:53.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:53.594+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:53.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:53.595+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:54.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:53.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:54.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:53.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:54.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:53.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:54.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:53.809+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:54.139 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:54.023+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:54.024741+0000) 2024-02-19T20:29:54.410 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:54.328+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:54.329235+0000) 2024-02-19T20:29:54.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:54.237+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:54.238932+0000) 2024-02-19T20:29:54.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:54.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:54.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:54.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:54.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:54.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:54.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:54.571+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:54.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:54.628+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:54.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:54.628+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:54.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:54.628+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:54.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:54.629+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:55.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:54.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:55.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:54.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:55.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:54.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:55.017 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:54.771+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:55.324 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:55.025+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:55.026381+0000) 2024-02-19T20:29:55.324 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:55.239+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:55.240586+0000) 2024-02-19T20:29:55.647 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:55.329+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:55.330852+0000) 2024-02-19T20:29:55.647 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:55.543+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:55.648 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:55.543+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:55.648 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:55.543+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:55.648 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:55.544+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:56.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:55.658+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:55.659100+0000) 2024-02-19T20:29:56.011 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:29:55.661+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62689 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:29:56.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:55.657+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:55.658967+0000) 2024-02-19T20:29:56.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:55.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:56.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:55.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:56.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:55.759+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:56.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:55.760+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:56.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:55.658+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:55.659850+0000) 2024-02-19T20:29:56.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:55.666+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:56.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:55.666+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:56.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:55.666+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:56.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:55.666+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:56.321 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:56.025+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:56.026622+0000) 2024-02-19T20:29:56.322 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:56.239+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:56.240809+0000) 2024-02-19T20:29:56.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:56.330+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:56.331043+0000) 2024-02-19T20:29:56.650 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:56.541+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:56.650 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:56.541+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:56.650 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:56.541+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:56.650 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:56.542+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:56.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:56.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:56.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:56.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:56.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:56.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:56.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:56.786+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:56.947 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:56.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:56.947 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:56.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:56.947 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:56.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:56.947 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:56.694+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:57.233 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:57.027+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:57.028671+0000) 2024-02-19T20:29:57.502 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:57.332+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:57.333173+0000) 2024-02-19T20:29:57.502 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:57.241+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:57.242844+0000) 2024-02-19T20:29:57.503 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:57.510+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:57.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:57.677+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:57.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:57.677+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:57.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:57.677+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:57.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:57.678+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:57.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:57.744+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:57.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:57.744+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:57.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:57.744+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:57.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:57.745+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:57.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:57.510+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:57.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:57.510+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:57.762 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:57.511+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:58.326 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:58.029+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:58.030901+0000) 2024-02-19T20:29:58.326 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:58.244+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:58.245056+0000) 2024-02-19T20:29:58.691 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:58.545+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:58.691 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:58.545+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:58.691 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:58.545+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:58.691 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:58.546+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:29:58.691 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:58.334+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:58.335325+0000) 2024-02-19T20:29:58.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:58.699+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:58.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:58.699+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:58.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:58.699+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:29:59.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:58.699+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:59.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:58.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:59.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:58.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:59.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:58.756+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:29:59.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:58.757+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:29:59.328 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:59.032+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:59.033029+0000) 2024-02-19T20:29:59.328 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:59.246+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:59.247028+0000) 2024-02-19T20:29:59.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:59.336+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:59.337481+0000) 2024-02-19T20:29:59.735 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:59.548+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:59.736 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:59.548+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:59.736 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:59.548+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:29:59.736 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:29:59.549+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:00.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:59.755+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:00.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:59.755+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:00.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:59.755+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:00.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:29:59.756+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:00.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:59.743+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:00.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:59.743+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:00.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:59.743+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:00.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:29:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:29:59.744+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:00.329 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:00.033+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:00.034602+0000) 2024-02-19T20:30:00.330 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:00.248+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:00.249085+0000) 2024-02-19T20:30:00.654 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:00.337+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:00.338950+0000) 2024-02-19T20:30:00.655 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:00.542+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:00.655 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:00.542+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:00.655 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:00.542+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:00.655 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:00.543+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:01.010 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:30:00.669+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62730 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:30:01.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:00.666+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:00.667814+0000) 2024-02-19T20:30:01.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:00.665+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:00.666734+0000) 2024-02-19T20:30:01.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:00.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:01.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:00.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:01.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:00.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:01.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:00.759+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:01.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:00.666+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:00.667060+0000) 2024-02-19T20:30:01.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:00.705+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:01.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:00.705+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:01.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:00.705+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:01.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:00.705+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:01.329 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:01.248+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:01.249256+0000) 2024-02-19T20:30:01.330 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:01.033+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:01.034780+0000) 2024-02-19T20:30:01.654 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:01.585+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:01.655 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:01.585+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:01.655 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:01.585+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:01.655 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:01.586+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:01.655 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:01.338+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:01.339175+0000) 2024-02-19T20:30:01.655 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:01.663+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:01.656 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:01.663+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:01.656 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:01.663+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:02.001 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:01.663+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:02.001 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:01.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:02.001 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:01.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:02.001 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:01.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:02.002 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:01.807+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:02.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:02.036+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:02.037002+0000) 2024-02-19T20:30:02.260 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:02.250+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:02.251846+0000) 2024-02-19T20:30:02.584 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:02.340+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:02.341338+0000) 2024-02-19T20:30:02.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:02.592+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:02.845 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:02.592+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:02.845 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:02.592+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:02.845 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:02.593+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:02.845 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:02.693+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:02.845 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:02.693+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:02.845 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:02.693+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:02.846 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:02.694+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:03.139 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:02.853+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:03.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:02.853+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:03.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:02.853+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:03.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:02.853+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:03.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:03.038+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:03.039116+0000) 2024-02-19T20:30:03.411 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:03.253+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:03.254335+0000) 2024-02-19T20:30:03.411 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:03.342+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:03.343423+0000) 2024-02-19T20:30:03.733 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:03.601+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:03.734 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:03.601+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:03.734 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:03.601+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:03.734 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:03.602+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:04.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:03.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:04.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:03.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:04.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:03.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:04.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:03.827+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:04.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:03.741+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:04.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:03.741+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:04.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:03.741+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:04.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:03.742+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:04.333 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:04.040+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:04.041181+0000) 2024-02-19T20:30:04.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:04.255+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:04.256449+0000) 2024-02-19T20:30:04.618 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:04.344+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:04.345486+0000) 2024-02-19T20:30:05.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:04.627+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:05.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:04.627+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:05.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:04.627+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:05.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:04.627+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:05.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:04.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:05.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:04.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:05.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:04.800+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:05.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:04.801+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:05.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:04.718+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:05.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:04.718+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:05.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:04.718+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:05.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:04.719+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:05.333 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:05.041+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:05.042973+0000) 2024-02-19T20:30:05.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:05.257+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:05.258022+0000) 2024-02-19T20:30:05.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:05.346+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:05.347234+0000) 2024-02-19T20:30:06.010 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:30:05.676+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62770 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:30:06.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:05.672+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:05.673714+0000) 2024-02-19T20:30:06.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:05.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:06.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:05.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:06.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:05.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:06.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:05.761+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:06.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:05.673+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:05.674090+0000) 2024-02-19T20:30:06.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:05.752+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:06.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:05.752+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:06.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:05.752+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:06.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:05.752+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:06.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:05.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:06.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:05.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:06.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:05.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:06.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:05.631+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:06.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:05.672+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:05.673906+0000) 2024-02-19T20:30:06.333 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:06.042+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:06.043171+0000) 2024-02-19T20:30:06.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:06.257+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:06.258193+0000) 2024-02-19T20:30:06.587 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:06.346+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:06.347417+0000) 2024-02-19T20:30:07.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:06.731+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:07.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:06.731+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:07.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:06.731+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:07.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:06.732+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:07.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:06.596+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:07.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:06.596+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:07.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:06.596+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:07.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:06.597+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:07.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:06.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:07.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:06.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:07.014 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:06.731+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:07.014 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:06.731+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:07.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:07.044+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:07.045570+0000) 2024-02-19T20:30:07.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:07.259+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:07.260207+0000) 2024-02-19T20:30:07.605 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:07.348+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:07.349427+0000) 2024-02-19T20:30:08.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:07.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:08.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:07.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:08.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:07.718+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:08.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:07.718+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:08.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:07.613+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:08.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:07.613+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:08.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:07.613+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:08.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:07.614+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:08.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:07.687+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:08.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:07.687+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:08.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:07.687+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:08.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:07.687+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:08.333 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:08.046+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:08.047613+0000) 2024-02-19T20:30:08.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:08.261+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:08.262243+0000) 2024-02-19T20:30:08.597 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:08.350+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:08.351514+0000) 2024-02-19T20:30:09.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:08.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:09.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:08.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:09.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:08.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:09.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:08.649+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:09.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:08.685+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:09.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:08.685+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:09.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:08.685+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:09.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:08.685+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:09.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:08.605+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:09.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:08.605+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:09.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:08.605+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:09.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:08.607+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:09.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:09.049+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:09.050945+0000) 2024-02-19T20:30:09.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:09.264+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:09.265136+0000) 2024-02-19T20:30:09.612 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:09.352+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:09.353769+0000) 2024-02-19T20:30:10.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:09.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:10.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:09.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:10.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:09.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:10.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:09.673+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:10.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:09.620+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:10.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:09.620+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:10.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:09.620+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:10.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:09.621+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:10.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:09.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:10.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:09.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:10.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:09.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:10.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:09.658+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:10.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:10.051+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:10.052616+0000) 2024-02-19T20:30:10.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:10.266+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:10.267004+0000) 2024-02-19T20:30:10.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:10.355+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:10.356330+0000) 2024-02-19T20:30:11.010 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:30:10.684+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62811 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:30:11.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:10.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:11.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:10.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:11.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:10.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:11.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:10.664+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:11.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:10.679+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:10.680627+0000) 2024-02-19T20:30:11.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:10.638+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:11.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:10.638+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:11.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:10.638+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:11.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:10.639+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:11.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:10.681+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:10.682576+0000) 2024-02-19T20:30:11.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:10.680+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:10.681863+0000) 2024-02-19T20:30:11.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:10.705+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:11.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:10.705+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:11.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:10.705+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:11.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:10.706+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:11.338 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:11.051+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:11.052872+0000) 2024-02-19T20:30:11.374 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:11.266+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:11.267174+0000) 2024-02-19T20:30:11.609 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:11.355+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:11.356532+0000) 2024-02-19T20:30:12.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:11.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:12.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:11.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:12.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:11.659+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:12.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:11.659+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:12.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:11.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:12.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:11.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:12.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:11.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:12.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:11.618+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:12.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:11.704+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:12.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:11.704+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:12.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:11.704+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:12.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:11.705+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:12.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:12.055+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:12.056412+0000) 2024-02-19T20:30:12.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:12.268+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:12.269748+0000) 2024-02-19T20:30:12.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:12.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:12.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:12.359+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:12.360105+0000) 2024-02-19T20:30:13.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:12.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:13.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:12.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:13.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:12.644+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:13.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:12.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:13.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:12.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:13.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:12.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:13.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:12.662+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:13.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:12.711+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:13.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:12.711+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:13.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:12.711+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:13.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:12.712+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:13.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:13.057+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:13.058639+0000) 2024-02-19T20:30:13.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:13.272+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:13.273269+0000) 2024-02-19T20:30:13.640 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:13.361+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:13.362584+0000) 2024-02-19T20:30:14.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:13.682+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:14.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:13.682+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:14.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:13.682+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:14.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:13.683+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:14.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:13.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:14.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:13.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:14.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:13.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:14.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:13.650+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:14.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:13.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:14.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:13.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:14.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:13.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:14.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:13.736+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:14.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:14.060+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:14.061025+0000) 2024-02-19T20:30:14.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:14.274+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:14.275264+0000) 2024-02-19T20:30:14.622 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:14.365+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:14.366095+0000) 2024-02-19T20:30:15.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:14.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:15.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:14.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:15.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:14.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:15.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:14.631+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:15.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:14.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:15.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:14.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:15.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:14.721+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:15.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:14.721+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:15.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:14.688+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:15.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:14.688+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:15.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:14.689+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:15.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:14.689+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:15.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:15.062+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:15.063647+0000) 2024-02-19T20:30:15.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:15.276+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:15.277029+0000) 2024-02-19T20:30:15.662 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:15.367+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:15.368645+0000) 2024-02-19T20:30:16.010 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:30:15.691+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62851 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:30:16.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:15.687+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:15.688699+0000) 2024-02-19T20:30:16.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:15.707+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:16.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:15.707+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:16.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:15.707+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:16.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:15.708+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:16.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:15.688+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:15.689653+0000) 2024-02-19T20:30:16.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:15.709+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:16.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:15.709+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:16.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:15.709+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:16.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:15.710+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:16.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:15.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:16.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:15.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:16.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:15.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:16.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:15.671+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:16.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:15.689+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:15.690263+0000) 2024-02-19T20:30:16.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:16.062+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:16.063811+0000) 2024-02-19T20:30:16.334 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:16.276+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:16.277241+0000) 2024-02-19T20:30:16.635 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:16.367+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:16.368818+0000) 2024-02-19T20:30:17.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:16.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:17.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:16.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:17.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:16.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:17.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:16.644+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:17.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:16.726+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:17.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:16.726+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:17.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:16.726+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:17.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:16.727+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:17.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:16.679+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:17.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:16.679+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:17.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:16.679+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:17.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:16.679+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:17.334 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:17.065+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:17.066220+0000) 2024-02-19T20:30:17.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:17.279+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:17.280782+0000) 2024-02-19T20:30:17.639 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:17.370+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:17.371314+0000) 2024-02-19T20:30:18.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:17.679+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:18.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:17.679+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:18.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:17.679+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:18.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:17.680+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:18.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:17.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:18.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:17.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:18.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:17.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:18.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:17.648+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:18.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:17.716+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:18.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:17.716+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:18.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:17.716+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:18.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:17.716+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:18.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:18.067+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:18.068409+0000) 2024-02-19T20:30:18.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:18.282+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:18.283120+0000) 2024-02-19T20:30:18.644 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:18.373+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:18.374761+0000) 2024-02-19T20:30:19.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:18.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:19.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:18.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:19.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:18.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:19.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:18.766+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:19.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:18.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:19.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:18.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:19.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:18.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:19.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:18.658+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:19.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:18.653+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:19.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:18.653+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:19.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:18.653+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:19.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:18.654+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:19.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:19.070+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:19.071022+0000) 2024-02-19T20:30:19.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:19.284+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:19.285498+0000) 2024-02-19T20:30:19.640 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:19.375+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:19.376899+0000) 2024-02-19T20:30:20.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:19.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:20.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:19.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:20.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:19.725+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:20.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:19.726+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:20.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:19.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:20.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:19.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:20.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:19.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:20.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:19.658+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:20.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:19.649+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:20.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:19.649+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:20.013 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:19.649+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:20.014 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:19.649+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:20.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:20.071+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:20.072745+0000) 2024-02-19T20:30:20.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:20.286+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:20.287392+0000) 2024-02-19T20:30:20.638 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:20.378+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:20.379567+0000) 2024-02-19T20:30:20.639 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:20.647+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:20.639 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:20.647+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:20.639 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:20.647+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:21.010 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:30:20.699+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62892 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:30:21.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:20.647+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:21.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:20.695+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:20.696872+0000) 2024-02-19T20:30:21.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:20.696+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:20.697858+0000) 2024-02-19T20:30:21.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:20.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:21.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:20.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:21.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:20.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:21.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:20.701+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:21.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:20.695+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:20.696817+0000) 2024-02-19T20:30:21.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:20.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:21.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:20.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:21.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:20.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:21.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:20.768+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:21.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:21.072+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:21.072959+0000) 2024-02-19T20:30:21.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:21.286+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:21.287591+0000) 2024-02-19T20:30:21.628 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:21.378+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:21.379762+0000) 2024-02-19T20:30:22.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:21.652+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:22.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:21.652+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:22.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:21.652+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:22.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:21.653+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:22.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:21.637+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:22.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:21.637+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:22.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:21.637+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:22.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:21.637+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:22.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:21.720+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:22.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:21.720+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:22.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:21.720+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:22.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:21.721+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:22.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:22.075+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:22.076029+0000) 2024-02-19T20:30:22.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:22.288+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:22.289755+0000) 2024-02-19T20:30:22.606 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:22.380+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:22.381969+0000) 2024-02-19T20:30:23.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:22.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:23.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:22.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:23.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:22.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:23.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:22.759+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:23.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:22.656+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:23.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:22.656+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:23.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:22.656+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:23.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:22.657+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:23.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:22.615+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:23.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:22.615+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:23.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:22.615+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:23.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:22.616+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:23.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:23.078+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:23.079554+0000) 2024-02-19T20:30:23.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:23.291+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:23.292141+0000) 2024-02-19T20:30:23.589 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:23.383+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:23.384003+0000) 2024-02-19T20:30:23.589 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:23.597+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:23.589 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:23.597+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:23.590 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:23.597+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:24.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:23.598+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:24.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:23.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:24.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:23.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:24.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:23.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:24.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:23.702+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:24.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:23.733+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:24.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:23.733+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:24.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:23.733+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:24.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:23.734+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:24.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:24.292+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:24.294545+0000) 2024-02-19T20:30:24.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:24.080+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:24.082363+0000) 2024-02-19T20:30:24.616 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:24.384+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:24.386429+0000) 2024-02-19T20:30:24.616 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:24.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:25.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:24.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:25.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:24.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:25.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:24.624+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:25.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:24.684+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:25.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:24.684+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:25.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:24.684+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:25.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:24.685+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:25.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:24.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:25.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:24.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:25.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:24.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:25.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:24.670+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:25.335 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:25.296+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:25.297089+0000) 2024-02-19T20:30:25.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:25.083+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:25.084215+0000) 2024-02-19T20:30:25.615 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:25.387+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:25.388015+0000) 2024-02-19T20:30:25.616 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:25.586+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:25.616 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:25.586+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:25.616 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:25.586+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:25.616 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:25.587+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:26.010 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:30:25.707+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62932 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:30:26.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:25.703+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:25.704673+0000) 2024-02-19T20:30:26.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:25.703+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:25.704914+0000) 2024-02-19T20:30:26.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:25.714+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:26.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:25.714+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:26.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:25.714+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:26.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:25.715+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:26.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:25.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:26.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:25.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:26.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:25.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:26.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:25.625+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:26.013 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:25.704+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:25.705126+0000) 2024-02-19T20:30:26.287 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:26.083+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:26.084406+0000) 2024-02-19T20:30:26.559 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:26.296+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:26.297267+0000) 2024-02-19T20:30:26.559 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:26.387+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:26.388195+0000) 2024-02-19T20:30:27.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:26.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:27.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:26.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:27.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:26.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:27.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:26.568+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:27.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:26.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:27.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:26.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:27.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:26.754+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:27.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:26.755+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:27.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:26.629+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:27.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:26.629+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:27.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:26.629+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:27.012 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:26.630+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:27.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:27.085+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:27.086799+0000) 2024-02-19T20:30:27.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:27.298+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:27.299781+0000) 2024-02-19T20:30:27.623 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:27.390+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:27.391772+0000) 2024-02-19T20:30:27.623 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:27.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:27.623 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:27.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:27.623 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:27.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:27.623 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:27.570+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:28.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:27.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:28.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:27.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:28.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:27.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:28.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:27.632+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:28.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:27.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:28.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:27.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:28.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:27.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:28.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:27.759+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:28.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:28.088+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:28.089232+0000) 2024-02-19T20:30:28.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:28.300+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:28.301913+0000) 2024-02-19T20:30:28.591 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:28.393+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:28.393994+0000) 2024-02-19T20:30:29.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:28.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:29.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:28.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:29.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:28.727+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:29.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:28.728+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:29.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:28.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:29.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:28.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:29.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:28.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:29.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:28.601+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:29.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:28.615+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:29.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:28.615+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:29.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:28.615+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:29.012 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:28.616+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:29.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:29.090+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:29.091783+0000) 2024-02-19T20:30:29.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:29.303+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:29.304063+0000) 2024-02-19T20:30:29.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:29.395+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:29.396429+0000) 2024-02-19T20:30:29.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:29.580+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:29.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:29.580+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:29.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:29.580+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:29.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:29.581+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:30.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:29.746+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:30.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:29.746+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:30.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:29.746+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:30.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:29.747+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:30.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:29.638+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:30.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:29.638+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:30.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:29.638+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:30.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:29.639+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:30.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:30.092+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:30.093434+0000) 2024-02-19T20:30:30.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:30.305+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:30.306698+0000) 2024-02-19T20:30:30.594 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:30.396+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:30.397959+0000) 2024-02-19T20:30:30.594 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:30.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:30.594 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:30.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:30.594 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:30.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:30.594 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:30.563+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:31.009 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:30:30.714+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 62973 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:30:31.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:30.712+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:30.713370+0000) 2024-02-19T20:30:31.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:30.602+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:31.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:30.602+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:31.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:30.602+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:31.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:30.603+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:31.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:30.711+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:30.712357+0000) 2024-02-19T20:30:31.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:30.711+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:30.712903+0000) 2024-02-19T20:30:31.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:30.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:31.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:30.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:31.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:30.770+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:31.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:30.770+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:31.335 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:31.092+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:31.093598+0000) 2024-02-19T20:30:31.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:31.305+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:31.306893+0000) 2024-02-19T20:30:31.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:31.584+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:31.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:31.584+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:31.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:31.584+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:31.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:31.585+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:31.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:31.397+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:31.398139+0000) 2024-02-19T20:30:31.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:31.525+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:31.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:31.525+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:31.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:31.525+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:31.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:31.525+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:32.077 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:31.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:32.078 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:31.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:32.078 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:31.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:32.078 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:31.787+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:32.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:32.095+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:32.096889+0000) 2024-02-19T20:30:32.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:32.308+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:32.309253+0000) 2024-02-19T20:30:32.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:32.564+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:32.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:32.564+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:32.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:32.564+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:32.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:32.565+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:32.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:32.400+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:32.401445+0000) 2024-02-19T20:30:32.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:32.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:32.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:32.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:32.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:32.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:32.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:32.569+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:33.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:32.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:33.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:32.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:33.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:32.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:33.080 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:32.805+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:33.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:33.099+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:33.100111+0000) 2024-02-19T20:30:33.337 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:33.310+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:33.311212+0000) 2024-02-19T20:30:33.597 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:33.402+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:33.403335+0000) 2024-02-19T20:30:33.598 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:33.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:33.598 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:33.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:33.598 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:33.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:33.598 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:33.554+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:34.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:33.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:34.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:33.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:34.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:33.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:34.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:33.809+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:34.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:33.606+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:34.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:33.606+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:34.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:33.606+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:34.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:33.606+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:34.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:34.101+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:34.102223+0000) 2024-02-19T20:30:34.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:34.312+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:34.313367+0000) 2024-02-19T20:30:34.586 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:34.404+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:34.405338+0000) 2024-02-19T20:30:34.587 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:34.580+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:34.587 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:34.580+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:34.587 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:34.580+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:34.587 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:34.581+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:34.842 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:34.595+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:34.842 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:34.595+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:34.842 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:34.595+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:34.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:34.596+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:35.094 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:34.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:35.095 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:34.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:35.095 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:34.851+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:35.095 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:34.852+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:35.095 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:35.103+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:35.104267+0000) 2024-02-19T20:30:35.397 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:35.314+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:35.315006+0000) 2024-02-19T20:30:35.708 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:35.585+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:35.708 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:35.585+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:35.708 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:35.585+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:35.708 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:35.586+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:35.708 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:35.406+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:35.407307+0000) 2024-02-19T20:30:35.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:35.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:35.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:35.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:35.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:35.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:35.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:35.559+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:35.734 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:30:35.722+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63013 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:30:35.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:35.720+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:35.721672+0000) 2024-02-19T20:30:35.735 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:35.719+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:35.720419+0000) 2024-02-19T20:30:35.735 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:35.719+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:35.720649+0000) 2024-02-19T20:30:36.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:35.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:36.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:35.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:36.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:35.836+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:36.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:35.837+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:36.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:36.103+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:36.104470+0000) 2024-02-19T20:30:36.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:36.314+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:36.315177+0000) 2024-02-19T20:30:36.612 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:36.406+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:36.407520+0000) 2024-02-19T20:30:36.612 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:36.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:36.612 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:36.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:36.612 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:36.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:36.612 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:36.540+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:37.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:36.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:37.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:36.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:37.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:36.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:37.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:36.865+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:37.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:36.620+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:37.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:36.620+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:37.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:36.620+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:37.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:36.621+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:37.336 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:37.317+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:37.318747+0000) 2024-02-19T20:30:37.337 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:37.105+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:37.106655+0000) 2024-02-19T20:30:37.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:37.408+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:37.409703+0000) 2024-02-19T20:30:37.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:37.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:37.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:37.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:37.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:37.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:37.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:37.523+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:38.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:37.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:38.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:37.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:38.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:37.850+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:38.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:37.850+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:38.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:37.629+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:38.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:37.629+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:38.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:37.629+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:38.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:37.630+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:38.336 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:38.107+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:38.108863+0000) 2024-02-19T20:30:38.337 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:38.319+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:38.320920+0000) 2024-02-19T20:30:38.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:38.584+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:38.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:38.584+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:38.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:38.584+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:38.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:38.585+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:38.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:38.412+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:38.413143+0000) 2024-02-19T20:30:38.763 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:38.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:38.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:38.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:38.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:38.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:38.764 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:38.540+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:39.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:38.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:39.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:38.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:39.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:38.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:39.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:38.893+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:39.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:39.111+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:39.112630+0000) 2024-02-19T20:30:39.406 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:39.322+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:39.323649+0000) 2024-02-19T20:30:39.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:39.592+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:39.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:39.592+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:39.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:39.592+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:39.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:39.593+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:39.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:39.415+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:39.416314+0000) 2024-02-19T20:30:39.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:39.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:39.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:39.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:39.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:39.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:39.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:39.531+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:40.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:39.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:40.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:39.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:40.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:39.915+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:40.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:39.916+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:40.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:40.114+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:40.115321+0000) 2024-02-19T20:30:40.484 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:40.324+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:40.325455+0000) 2024-02-19T20:30:40.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:40.417+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:40.418246+0000) 2024-02-19T20:30:40.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:40.493+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:40.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:40.493+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:40.485 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:40.493+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:40.738 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:30:40.730+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63054 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:30:40.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:40.726+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:40.727498+0000) 2024-02-19T20:30:40.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:40.493+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:40.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:40.726+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:40.727647+0000) 2024-02-19T20:30:40.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:40.551+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:40.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:40.551+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:40.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:40.551+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:40.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:40.551+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:40.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:40.726+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:40.727603+0000) 2024-02-19T20:30:41.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:40.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:41.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:40.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:41.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:40.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:41.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:40.958+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:41.337 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:41.114+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:41.115543+0000) 2024-02-19T20:30:41.337 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:41.324+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:41.325675+0000) 2024-02-19T20:30:41.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:41.551+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:41.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:41.551+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:41.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:41.551+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:41.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:41.552+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:41.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:41.417+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:41.418404+0000) 2024-02-19T20:30:41.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:41.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:41.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:41.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:41.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:41.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:41.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:41.446+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:42.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:41.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:42.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:41.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:42.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:41.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:42.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:41.973+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:42.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:42.116+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:42.117451+0000) 2024-02-19T20:30:42.571 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:42.326+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:42.327717+0000) 2024-02-19T20:30:42.571 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:42.419+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:42.420838+0000) 2024-02-19T20:30:42.571 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:42.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:42.571 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:42.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:42.572 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:42.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:42.572 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:42.459+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:42.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:42.580+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:42.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:42.580+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:42.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:42.580+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:42.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:42.581+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:43.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:42.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:43.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:42.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:43.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:42.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:43.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:42.974+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:43.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:43.118+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:43.119440+0000) 2024-02-19T20:30:43.571 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:43.329+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:43.330256+0000) 2024-02-19T20:30:43.572 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:43.423+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:43.424055+0000) 2024-02-19T20:30:43.572 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:43.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:43.572 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:43.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:43.572 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:43.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:43.572 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:43.469+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:43.838 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:43.613+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:43.838 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:43.613+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:43.839 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:43.613+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:43.839 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:43.613+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:44.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:43.931+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:44.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:43.931+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:44.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:43.931+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:44.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:43.932+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:44.415 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:44.120+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:44.121386+0000) 2024-02-19T20:30:44.415 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:44.332+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:44.333846+0000) 2024-02-19T20:30:44.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:44.655+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:44.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:44.655+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:44.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:44.655+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:44.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:44.656+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:44.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:44.426+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:44.427003+0000) 2024-02-19T20:30:44.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:44.437+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:44.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:44.437+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:44.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:44.437+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:44.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:44.437+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:45.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:44.931+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:45.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:44.931+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:45.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:44.931+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:45.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:44.932+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:45.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:45.122+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:45.123026+0000) 2024-02-19T20:30:45.694 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:45.334+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:45.335557+0000) 2024-02-19T20:30:45.694 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:45.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:45.694 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:45.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:45.694 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:45.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:45.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:45.400+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:45.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:45.400+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:45.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:45.400+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:45.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:45.400+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:45.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:45.427+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:45.428944+0000) 2024-02-19T20:30:46.009 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:30:45.738+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63094 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:30:46.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:45.735+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:45.736207+0000) 2024-02-19T20:30:46.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:45.703+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:46.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:45.735+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:45.736049+0000) 2024-02-19T20:30:46.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:45.734+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:45.735128+0000) 2024-02-19T20:30:46.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:45.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:46.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:45.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:46.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:45.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:46.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:45.923+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:46.415 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:46.122+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:46.123215+0000) 2024-02-19T20:30:46.415 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:46.334+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:46.335761+0000) 2024-02-19T20:30:46.722 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:46.428+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:46.429125+0000) 2024-02-19T20:30:46.723 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:46.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:46.723 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:46.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:46.723 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:46.428+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:46.723 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:46.429+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:47.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:46.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:47.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:46.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:47.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:46.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:47.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:46.919+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:47.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:46.731+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:47.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:46.731+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:47.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:46.731+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:47.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:46.732+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:47.415 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:47.124+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:47.125608+0000) 2024-02-19T20:30:47.415 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:47.337+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:47.338013+0000) 2024-02-19T20:30:47.731 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:47.431+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:47.432697+0000) 2024-02-19T20:30:47.732 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:47.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:47.732 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:47.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:47.732 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:47.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:47.732 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:47.436+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:48.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:47.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:48.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:47.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:48.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:47.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:48.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:47.741+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:48.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:47.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:48.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:47.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:48.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:47.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:48.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:47.948+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:48.409 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:48.126+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:48.127797+0000) 2024-02-19T20:30:48.410 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:48.339+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:48.340639+0000) 2024-02-19T20:30:48.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:48.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:48.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:48.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:48.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:48.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:48.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:48.419+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:48.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:48.435+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:48.436256+0000) 2024-02-19T20:30:49.093 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:48.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:49.093 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:48.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:49.093 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:48.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:49.093 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:48.777+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:49.094 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:48.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:49.094 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:48.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:49.094 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:48.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:49.094 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:48.908+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:49.415 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:49.128+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:49.129922+0000) 2024-02-19T20:30:49.415 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:49.341+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:49.342822+0000) 2024-02-19T20:30:49.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:49.425+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:49.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:49.425+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:49.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:49.425+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:49.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:49.425+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:49.742 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:49.437+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:49.438543+0000) 2024-02-19T20:30:50.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:49.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:50.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:49.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:50.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:49.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:50.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:49.942+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:50.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:49.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:50.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:49.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:50.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:49.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:50.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:49.751+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:50.415 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:50.130+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:50.131598+0000) 2024-02-19T20:30:50.416 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:50.344+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:50.345474+0000) 2024-02-19T20:30:50.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:50.439+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:50.440205+0000) 2024-02-19T20:30:50.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:50.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:50.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:50.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:50.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:50.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:50.731 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:50.453+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:51.009 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:30:50.745+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63135 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:30:51.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:50.742+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:50.743785+0000) 2024-02-19T20:30:51.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:50.742+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:50.743435+0000) 2024-02-19T20:30:51.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:50.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:51.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:50.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:51.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:50.757+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:51.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:50.758+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:51.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:50.741+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:50.742785+0000) 2024-02-19T20:30:51.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:50.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:51.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:50.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:51.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:50.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:51.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:50.931+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:51.409 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:51.130+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:51.131784+0000) 2024-02-19T20:30:51.409 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:51.344+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:51.345681+0000) 2024-02-19T20:30:51.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:51.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:51.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:51.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:51.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:51.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:51.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:51.418+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:51.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:51.439+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:51.440296+0000) 2024-02-19T20:30:52.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:51.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:52.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:51.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:52.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:51.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:52.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:51.905+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:52.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:51.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:52.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:51.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:52.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:51.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:52.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:51.735+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:52.415 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:52.346+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:52.347788+0000) 2024-02-19T20:30:52.416 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:52.134+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:52.135125+0000) 2024-02-19T20:30:52.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:52.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:52.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:52.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:52.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:52.431+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:52.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:52.432+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:52.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:52.441+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:52.442497+0000) 2024-02-19T20:30:53.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:52.867+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:53.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:52.867+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:53.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:52.867+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:53.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:52.867+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:53.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:52.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:53.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:52.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:53.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:52.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:53.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:52.767+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:53.415 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:53.137+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:53.138556+0000) 2024-02-19T20:30:53.416 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:53.348+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:53.349858+0000) 2024-02-19T20:30:53.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:53.443+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:53.444537+0000) 2024-02-19T20:30:53.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:53.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:53.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:53.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:53.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:53.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:53.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:53.477+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:54.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:53.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:54.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:53.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:54.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:53.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:54.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:53.909+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:54.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:53.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:54.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:53.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:54.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:53.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:54.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:53.735+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:54.415 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:54.141+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:54.142085+0000) 2024-02-19T20:30:54.416 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:54.352+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:54.353247+0000) 2024-02-19T20:30:54.679 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:54.447+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:54.448001+0000) 2024-02-19T20:30:54.679 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:54.514+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:54.679 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:54.514+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:54.679 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:54.514+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:54.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:54.515+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:55.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:54.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:55.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:54.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:55.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:54.907+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:55.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:54.908+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:55.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:54.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:55.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:54.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:55.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:54.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:55.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:54.689+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:55.338 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:55.142+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:55.143856+0000) 2024-02-19T20:30:55.642 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:55.354+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:55.355834+0000) 2024-02-19T20:30:55.643 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:55.651+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:55.643 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:55.651+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:55.643 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:55.651+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:55.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:55.448+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:55.449706+0000) 2024-02-19T20:30:55.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:55.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:55.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:55.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:55.644 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:55.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:55.644 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:55.514+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:56.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:55.750+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:55.751062+0000) 2024-02-19T20:30:56.010 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:30:55.753+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63175 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:30:56.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:55.652+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:56.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:55.749+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:55.750424+0000) 2024-02-19T20:30:56.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:55.749+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:55.750676+0000) 2024-02-19T20:30:56.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:55.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:56.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:55.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:56.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:55.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:56.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:55.899+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:56.338 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:56.143+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:56.144025+0000) 2024-02-19T20:30:56.622 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:56.355+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:56.356015+0000) 2024-02-19T20:30:56.622 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:56.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:56.623 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:56.448+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:56.449879+0000) 2024-02-19T20:30:56.623 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:56.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:56.623 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:56.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:56.623 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:56.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:56.623 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:56.556+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:56.919 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:56.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:56.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:56.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:56.920 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:56.632+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:57.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:56.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:57.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:56.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:57.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:56.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:57.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:56.929+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:57.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:57.144+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:57.145923+0000) 2024-02-19T20:30:57.442 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:57.358+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:57.359385+0000) 2024-02-19T20:30:57.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:57.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:57.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:57.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:57.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:57.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:57.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:57.657+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:57.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:57.451+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:57.452322+0000) 2024-02-19T20:30:57.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:57.533+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:57.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:57.533+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:57.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:57.533+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:57.762 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:57.534+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:58.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:57.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:58.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:57.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:58.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:57.950+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:58.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:57.951+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:58.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:58.147+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:58.148340+0000) 2024-02-19T20:30:58.648 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:58.361+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:58.362850+0000) 2024-02-19T20:30:58.649 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:58.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:58.649 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:58.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:58.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:58.454+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:58.455702+0000) 2024-02-19T20:30:58.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:58.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:58.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:58.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:58.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:58.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:58.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:58.523+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:58.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:58.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:58.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:58.658+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:59.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:58.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:59.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:58.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:59.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:58.909+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:30:59.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:58.910+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:30:59.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:59.149+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:59.150537+0000) 2024-02-19T20:30:59.448 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:59.365+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:59.366307+0000) 2024-02-19T20:30:59.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:59.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:59.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:59.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:59.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:59.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:30:59.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:30:59.616+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:30:59.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:59.457+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:59.458311+0000) 2024-02-19T20:30:59.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:59.524+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:59.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:59.524+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:59.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:59.524+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:30:59.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:30:59.524+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:00.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:59.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:00.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:59.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:00.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:59.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:00.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:30:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:30:59.949+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:00.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:00.151+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:00.152354+0000) 2024-02-19T20:31:00.648 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:00.367+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:00.368285+0000) 2024-02-19T20:31:00.649 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:00.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:00.649 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:00.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:00.649 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:00.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:00.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:00.459+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:00.460698+0000) 2024-02-19T20:31:00.649 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:00.571+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:00.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:00.571+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:00.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:00.571+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:00.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:00.572+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:00.980 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:31:00.760+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63216 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:31:00.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:00.758+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:00.759231+0000) 2024-02-19T20:31:00.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:00.658+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:00.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:00.757+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:00.758137+0000) 2024-02-19T20:31:00.982 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:00.757+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:00.758745+0000) 2024-02-19T20:31:00.982 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:00.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:00.982 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:00.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:00.982 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:00.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:01.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:00.991+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:01.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:01.151+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:01.152585+0000) 2024-02-19T20:31:01.602 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:01.367+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:01.368478+0000) 2024-02-19T20:31:01.603 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:01.459+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:01.460899+0000) 2024-02-19T20:31:01.603 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:01.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:01.603 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:01.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:01.603 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:01.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:01.603 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:01.537+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:01.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:01.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:01.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:01.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:01.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:01.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:01.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:01.612+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:02.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:01.981+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:02.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:01.981+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:02.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:01.981+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:02.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:01.981+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:02.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:02.155+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:02.156079+0000) 2024-02-19T20:31:02.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:02.370+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:02.371774+0000) 2024-02-19T20:31:02.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:02.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:02.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:02.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:02.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:02.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:02.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:02.570+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:02.610 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:02.463+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:02.464324+0000) 2024-02-19T20:31:02.610 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:02.533+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:02.610 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:02.533+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:02.610 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:02.533+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:02.610 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:02.534+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:03.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:03.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:03.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:03.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:03.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:03.018+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:03.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:03.019+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:03.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:03.157+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:03.158870+0000) 2024-02-19T20:31:03.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:03.466+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:03.467682+0000) 2024-02-19T20:31:03.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:03.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:03.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:03.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:03.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:03.506+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:03.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:03.507+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:03.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:03.374+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:03.375157+0000) 2024-02-19T20:31:03.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:03.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:03.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:03.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:03.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:03.553+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:03.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:03.554+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:04.106 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:04.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:04.106 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:04.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:04.106 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:04.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:04.106 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:04.021+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:04.366 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:04.161+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:04.162387+0000) 2024-02-19T20:31:04.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:04.376+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:04.377176+0000) 2024-02-19T20:31:04.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:04.535+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:04.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:04.535+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:04.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:04.535+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:04.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:04.536+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:04.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:04.468+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:04.469728+0000) 2024-02-19T20:31:04.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:04.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:04.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:04.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:04.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:04.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:04.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:04.556+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:05.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:04.986+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:05.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:04.986+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:05.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:04.986+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:05.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:04.987+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:05.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:05.163+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:05.164007+0000) 2024-02-19T20:31:05.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:05.378+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:05.379081+0000) 2024-02-19T20:31:05.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:05.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:05.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:05.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:05.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:05.487+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:05.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:05.488+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:05.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:05.470+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:05.471811+0000) 2024-02-19T20:31:05.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:05.524+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:05.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:05.524+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:05.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:05.524+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:05.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:05.524+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:06.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:05.767+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:05.768573+0000) 2024-02-19T20:31:06.009 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:31:05.768+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63256 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:31:06.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:05.766+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:05.767280+0000) 2024-02-19T20:31:06.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:05.766+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:05.767066+0000) 2024-02-19T20:31:06.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:05.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:06.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:05.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:06.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:05.998+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:06.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:05.999+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:06.369 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:06.163+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:06.164178+0000) 2024-02-19T20:31:06.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:06.471+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:06.471996+0000) 2024-02-19T20:31:06.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:06.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:06.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:06.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:06.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:06.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:06.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:06.556+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:06.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:06.378+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:06.379279+0000) 2024-02-19T20:31:06.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:06.522+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:06.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:06.522+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:06.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:06.522+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:06.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:06.523+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:07.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:07.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:07.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:07.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:07.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:07.011+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:07.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:07.012+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:07.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:07.165+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:07.166169+0000) 2024-02-19T20:31:07.668 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:07.381+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:07.382493+0000) 2024-02-19T20:31:07.669 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:07.505+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:07.669 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:07.505+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:07.669 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:07.505+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:07.669 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:07.506+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:07.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:07.474+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:07.475239+0000) 2024-02-19T20:31:07.669 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:07.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:07.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:07.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:07.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:07.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:07.670 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:07.562+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:08.339 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:08.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:08.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:08.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:08.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:08.035+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:08.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:08.036+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:08.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:08.168+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:08.168987+0000) 2024-02-19T20:31:08.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:08.385+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:08.385976+0000) 2024-02-19T20:31:08.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:08.529+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:08.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:08.529+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:08.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:08.529+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:08.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:08.529+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:08.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:08.477+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:08.478521+0000) 2024-02-19T20:31:08.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:08.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:08.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:08.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:08.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:08.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:08.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:08.527+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:09.339 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:09.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:09.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:09.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:09.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:09.022+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:09.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:09.023+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:09.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:09.170+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:09.171128+0000) 2024-02-19T20:31:09.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:09.480+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:09.481204+0000) 2024-02-19T20:31:09.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:09.551+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:09.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:09.551+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:09.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:09.551+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:09.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:09.551+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:09.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:09.387+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:09.388443+0000) 2024-02-19T20:31:09.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:09.502+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:09.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:09.502+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:09.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:09.502+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:09.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:09.503+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:10.339 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:10.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:10.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:10.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:10.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:10.033+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:10.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:10.034+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:10.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:10.171+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:10.172827+0000) 2024-02-19T20:31:10.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:10.390+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:10.391184+0000) 2024-02-19T20:31:10.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:10.529+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:10.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:10.529+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:10.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:10.529+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:10.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:10.530+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:10.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:10.482+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:10.483863+0000) 2024-02-19T20:31:10.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:10.563+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:10.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:10.563+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:10.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:10.563+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:10.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:10.563+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:11.038 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:31:10.776+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63297 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:31:11.038 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:10.773+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:10.774354+0000) 2024-02-19T20:31:11.038 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:10.772+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:10.773175+0000) 2024-02-19T20:31:11.039 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:10.773+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:10.774634+0000) 2024-02-19T20:31:11.339 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:11.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:11.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:11.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:11.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:11.047+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:11.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:11.048+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:11.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:11.172+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:11.173040+0000) 2024-02-19T20:31:11.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:11.390+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:11.391364+0000) 2024-02-19T20:31:11.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:11.508+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:11.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:11.508+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:11.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:11.508+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:11.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:11.509+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:11.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:11.483+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:11.484066+0000) 2024-02-19T20:31:11.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:11.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:11.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:11.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:11.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:11.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:11.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:11.519+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:12.339 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:12.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:12.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:12.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:12.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:12.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:12.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:12.036+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:12.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:12.173+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:12.175590+0000) 2024-02-19T20:31:12.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:12.484+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:12.486586+0000) 2024-02-19T20:31:12.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:12.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:12.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:12.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:12.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:12.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:12.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:12.549+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:12.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:12.391+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:12.393943+0000) 2024-02-19T20:31:12.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:12.518+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:12.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:12.518+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:12.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:12.518+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:12.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:12.518+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:13.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:13.012+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:13.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:13.012+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:13.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:13.012+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:13.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:13.013+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:13.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:13.177+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:13.178137+0000) 2024-02-19T20:31:13.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:13.396+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:13.397458+0000) 2024-02-19T20:31:13.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:13.550+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:13.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:13.550+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:13.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:13.550+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:13.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:13.550+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:13.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:13.487+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:13.488742+0000) 2024-02-19T20:31:13.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:13.597+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:13.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:13.597+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:13.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:13.597+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:13.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:13.597+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:14.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:14.050+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:14.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:14.050+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:14.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:14.050+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:14.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:14.051+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:14.389 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:14.179+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:14.180713+0000) 2024-02-19T20:31:14.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:14.398+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:14.399541+0000) 2024-02-19T20:31:14.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:14.519+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:14.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:14.519+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:14.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:14.519+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:14.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:14.520+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:14.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:14.491+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:14.492280+0000) 2024-02-19T20:31:14.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:14.639+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:14.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:14.639+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:14.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:14.639+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:14.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:14.639+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:15.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:15.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:15.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:15.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:15.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:15.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:15.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:15.011+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:15.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:15.181+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:15.182578+0000) 2024-02-19T20:31:15.607 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:15.494+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:15.495089+0000) 2024-02-19T20:31:15.607 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:15.616+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:15.607 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:15.616+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:15.608 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:15.616+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:15.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:15.400+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:15.401807+0000) 2024-02-19T20:31:15.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:15.554+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:15.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:15.554+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:15.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:15.554+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:15.608 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:15.555+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:15.891 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:31:15.783+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63337 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:31:15.891 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:15.780+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:15.781083+0000) 2024-02-19T20:31:15.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:15.780+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:15.781471+0000) 2024-02-19T20:31:15.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:15.617+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:15.892 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:15.781+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:15.782090+0000) 2024-02-19T20:31:16.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:15.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:16.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:15.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:16.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:15.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:16.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:15.962+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:16.417 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:16.181+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:16.182769+0000) 2024-02-19T20:31:16.418 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:16.401+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:16.402000+0000) 2024-02-19T20:31:16.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:16.598+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:16.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:16.598+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:16.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:16.598+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:16.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:16.598+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:16.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:16.494+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:16.495266+0000) 2024-02-19T20:31:16.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:16.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:16.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:16.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:16.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:16.572+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:16.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:16.572+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:17.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:16.927+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:17.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:16.927+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:17.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:16.927+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:17.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:16.928+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:17.487 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:17.184+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:17.185329+0000) 2024-02-19T20:31:17.487 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:17.403+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:17.404531+0000) 2024-02-19T20:31:17.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:17.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:17.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:17.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:17.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:17.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:17.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:17.594+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:17.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:17.496+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:17.497363+0000) 2024-02-19T20:31:17.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:17.602+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:17.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:17.602+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:17.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:17.602+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:17.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:17.602+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:18.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:17.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:18.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:17.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:18.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:17.959+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:18.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:17.960+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:18.396 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:18.187+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:18.188874+0000) 2024-02-19T20:31:18.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:18.499+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:18.500906+0000) 2024-02-19T20:31:18.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:18.585+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:18.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:18.585+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:18.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:18.585+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:18.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:18.585+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:18.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:18.405+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:18.406779+0000) 2024-02-19T20:31:18.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:18.558+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:18.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:18.558+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:18.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:18.558+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:18.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:18.559+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:19.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:18.999+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:19.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:18.999+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:19.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:18.999+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:19.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:18.999+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:19.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:19.189+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:19.190951+0000) 2024-02-19T20:31:19.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:19.502+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:19.503409+0000) 2024-02-19T20:31:19.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:19.547+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:19.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:19.547+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:19.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:19.547+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:19.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:19.548+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:19.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:19.409+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:19.410083+0000) 2024-02-19T20:31:19.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:19.520+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:19.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:19.520+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:19.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:19.520+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:19.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:19.521+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:20.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:20.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:20.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:20.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:20.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:20.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:20.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:20.011+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:20.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:20.191+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:20.192636+0000) 2024-02-19T20:31:20.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:20.411+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:20.412753+0000) 2024-02-19T20:31:20.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:20.492+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:20.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:20.492+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:20.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:20.492+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:20.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:20.492+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:20.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:20.504+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:20.505153+0000) 2024-02-19T20:31:20.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:20.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:20.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:20.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:20.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:20.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:20.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:20.570+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:21.026 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:31:20.791+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63378 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:31:21.027 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:20.788+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:20.789311+0000) 2024-02-19T20:31:21.027 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:20.787+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:20.788758+0000) 2024-02-19T20:31:21.027 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:20.787+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:20.788767+0000) 2024-02-19T20:31:21.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:21.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:21.340 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:21.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:21.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:21.036+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:21.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:21.036+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:21.341 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:21.191+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:21.192807+0000) 2024-02-19T20:31:21.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:21.504+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:21.505313+0000) 2024-02-19T20:31:21.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:21.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:21.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:21.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:21.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:21.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:21.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:21.588+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:21.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:21.411+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:21.412968+0000) 2024-02-19T20:31:21.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:21.443+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:21.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:21.443+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:21.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:21.443+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:21.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:21.444+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:22.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:22.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:22.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:22.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:22.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:22.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:22.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:22.011+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:22.260 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:22.195+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:22.196157+0000) 2024-02-19T20:31:22.610 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:22.506+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:22.507800+0000) 2024-02-19T20:31:22.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:22.415+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:22.416459+0000) 2024-02-19T20:31:22.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:22.470+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:22.610 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:22.470+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:22.611 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:22.470+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:22.611 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:22.471+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:22.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:22.619+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:22.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:22.619+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:22.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:22.619+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:22.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:22.620+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:23.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:22.993+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:23.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:22.993+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:23.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:22.993+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:23.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:22.994+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:23.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:23.197+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:23.198353+0000) 2024-02-19T20:31:23.643 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:23.509+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:23.510497+0000) 2024-02-19T20:31:23.643 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:23.418+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:23.419097+0000) 2024-02-19T20:31:23.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:23.520+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:23.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:23.520+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:23.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:23.520+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:23.644 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:23.521+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:23.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:23.652+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:23.975 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:23.652+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:23.975 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:23.652+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:23.975 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:23.653+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:24.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:23.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:24.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:23.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:24.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:23.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:24.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:23.984+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:24.259 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:24.199+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:24.200658+0000) 2024-02-19T20:31:24.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:24.512+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:24.513079+0000) 2024-02-19T20:31:24.600 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:24.609+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:24.601 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:24.610+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:24.601 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:24.610+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:24.601 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:24.420+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:24.421531+0000) 2024-02-19T20:31:24.601 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:24.532+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:24.601 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:24.532+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:24.601 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:24.532+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:24.602 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:24.533+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:24.933 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:24.610+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:25.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:24.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:25.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:24.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:25.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:24.942+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:25.192 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:24.943+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:25.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:25.201+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:25.202695+0000) 2024-02-19T20:31:25.505 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:25.423+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:25.424371+0000) 2024-02-19T20:31:25.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:25.561+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:25.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:25.561+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:25.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:25.561+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:25.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:25.562+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:25.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:25.514+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:25.515798+0000) 2024-02-19T20:31:25.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:25.608+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:25.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:25.608+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:25.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:25.608+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:25.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:25.609+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:26.125 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:31:25.799+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63418 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:31:26.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:25.795+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:25.796799+0000) 2024-02-19T20:31:26.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:25.796+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:25.797885+0000) 2024-02-19T20:31:26.126 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:25.797+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:25.798050+0000) 2024-02-19T20:31:26.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:25.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:26.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:25.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:26.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:25.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:26.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:25.929+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:26.414 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:26.201+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:26.202894+0000) 2024-02-19T20:31:26.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:26.423+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:26.424596+0000) 2024-02-19T20:31:26.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:26.560+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:26.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:26.560+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:26.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:26.560+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:26.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:26.561+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:26.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:26.515+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:26.516007+0000) 2024-02-19T20:31:26.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:26.637+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:26.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:26.637+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:26.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:26.637+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:26.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:26.637+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:27.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:26.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:27.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:26.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:27.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:26.883+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:27.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:26.884+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:27.417 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:27.204+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:27.205449+0000) 2024-02-19T20:31:27.671 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:27.426+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:27.427638+0000) 2024-02-19T20:31:27.671 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:27.550+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:27.671 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:27.550+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:27.671 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:27.550+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:27.671 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:27.551+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:27.671 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:27.517+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:27.518370+0000) 2024-02-19T20:31:28.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:27.859+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:28.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:27.859+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:28.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:27.859+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:28.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:27.859+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:28.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:27.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:28.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:27.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:28.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:27.680+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:28.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:27.680+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:28.418 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:28.207+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:28.208125+0000) 2024-02-19T20:31:28.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:28.429+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:28.430159+0000) 2024-02-19T20:31:28.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:28.529+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:28.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:28.529+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:28.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:28.529+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:28.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:28.529+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:28.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:28.519+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:28.520628+0000) 2024-02-19T20:31:28.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:28.676+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:28.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:28.676+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:28.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:28.676+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:28.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:28.677+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:29.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:28.897+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:29.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:28.897+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:29.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:28.897+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:29.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:28.898+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:29.418 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:29.209+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:29.210604+0000) 2024-02-19T20:31:29.702 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:29.431+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:29.432503+0000) 2024-02-19T20:31:29.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:29.516+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:29.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:29.516+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:29.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:29.516+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:29.703 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:29.517+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:29.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:29.522+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:29.523183+0000) 2024-02-19T20:31:29.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:29.711+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:29.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:29.711+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:29.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:29.711+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:30.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:29.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:30.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:29.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:30.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:29.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:30.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:29.931+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:30.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:29.712+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:30.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:30.211+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:30.212228+0000) 2024-02-19T20:31:30.705 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:30.433+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:30.434616+0000) 2024-02-19T20:31:30.705 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:30.541+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:30.705 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:30.541+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:30.705 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:30.541+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:30.706 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:30.541+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:30.706 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:30.524+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:30.525078+0000) 2024-02-19T20:31:30.706 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:30.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:30.706 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:30.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:30.706 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:30.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:31.009 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:31:30.806+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63459 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:31:31.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:30.803+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:30.804296+0000) 2024-02-19T20:31:31.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:30.715+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:31.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:30.803+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:30.804866+0000) 2024-02-19T20:31:31.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:30.803+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:30.804350+0000) 2024-02-19T20:31:31.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:30.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:31.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:30.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:31.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:30.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:31.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:30.887+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:31.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:31.211+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:31.212445+0000) 2024-02-19T20:31:31.749 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:31.524+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:31.525276+0000) 2024-02-19T20:31:31.749 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:31.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:31.749 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:31.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:31.749 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:31.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:31.749 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:31.433+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:31.434876+0000) 2024-02-19T20:31:31.750 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:31.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:31.750 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:31.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:31.750 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:31.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:31.750 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:31.589+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:32.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:31.758+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:32.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:31.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:32.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:31.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:32.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:31.865+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:32.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:31.866+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:32.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:32.213+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:32.214415+0000) 2024-02-19T20:31:32.708 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:32.435+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:32.436876+0000) 2024-02-19T20:31:32.708 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:32.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:32.708 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:32.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:32.708 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:32.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:32.708 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:32.616+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:32.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:32.526+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:32.527260+0000) 2024-02-19T20:31:32.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:32.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:32.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:32.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:32.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:32.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:33.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:32.718+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:33.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:32.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:33.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:32.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:33.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:32.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:33.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:32.834+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:33.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:33.215+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:33.216832+0000) 2024-02-19T20:31:33.726 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:33.439+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:33.440133+0000) 2024-02-19T20:31:33.726 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:33.629+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:33.726 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:33.629+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:33.727 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:33.629+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:33.727 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:33.629+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:33.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:33.529+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:33.530831+0000) 2024-02-19T20:31:33.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:33.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:33.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:33.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:33.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:33.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:34.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:33.736+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:34.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:33.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:34.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:33.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:34.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:33.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:34.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:33.792+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:34.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:34.217+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:34.218948+0000) 2024-02-19T20:31:34.744 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:34.441+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:34.442582+0000) 2024-02-19T20:31:34.744 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:34.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:34.744 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:34.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:34.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:34.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:34.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:34.671+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:34.745 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:34.532+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:34.533270+0000) 2024-02-19T20:31:34.745 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:34.753+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:34.745 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:34.753+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:34.745 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:34.753+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:35.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:34.754+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:35.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:34.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:35.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:34.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:35.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:34.791+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:35.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:34.792+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:35.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:35.219+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:35.220922+0000) 2024-02-19T20:31:35.724 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:35.443+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:35.444516+0000) 2024-02-19T20:31:35.724 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:35.622+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:35.724 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:35.622+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:35.725 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:35.622+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:35.725 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:35.623+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:35.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:35.534+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:35.535589+0000) 2024-02-19T20:31:35.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:35.733+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:35.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:35.733+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:35.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:35.733+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:36.009 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:31:35.815+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63499 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:31:36.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:35.811+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:35.812753+0000) 2024-02-19T20:31:36.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:35.734+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:36.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:35.811+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:35.812883+0000) 2024-02-19T20:31:36.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:35.812+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:35.813234+0000) 2024-02-19T20:31:36.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:35.817+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:36.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:35.817+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:36.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:35.817+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:36.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:35.818+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:36.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:36.220+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:36.221088+0000) 2024-02-19T20:31:36.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:36.443+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:36.444717+0000) 2024-02-19T20:31:36.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:36.605+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:36.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:36.605+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:36.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:36.605+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:36.714 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:36.606+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:36.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:36.534+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:36.535756+0000) 2024-02-19T20:31:36.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:36.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:36.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:36.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:36.715 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:36.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:37.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:36.723+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:37.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:36.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:37.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:36.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:37.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:36.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:37.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:36.786+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:37.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:37.222+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:37.223551+0000) 2024-02-19T20:31:37.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:37.537+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:37.538145+0000) 2024-02-19T20:31:37.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:37.684+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:37.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:37.446+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:37.447915+0000) 2024-02-19T20:31:37.676 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:37.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:37.676 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:37.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:37.676 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:37.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:37.676 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:37.570+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:37.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:37.684+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:37.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:37.684+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:37.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:37.686+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:37.974 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:37.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:37.974 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:37.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:37.974 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:37.765+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:37.974 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:37.766+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:38.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:38.224+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:38.225647+0000) 2024-02-19T20:31:38.708 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:38.448+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:38.449941+0000) 2024-02-19T20:31:38.708 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:38.549+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:38.708 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:38.549+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:38.709 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:38.549+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:38.709 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:38.549+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:38.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:38.539+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:38.540433+0000) 2024-02-19T20:31:38.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:38.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:38.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:38.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:38.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:38.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:39.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:38.718+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:39.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:38.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:39.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:38.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:39.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:38.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:39.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:38.797+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:39.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:39.226+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:39.227847+0000) 2024-02-19T20:31:39.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:39.541+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:39.542637+0000) 2024-02-19T20:31:39.694 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:39.451+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:39.452526+0000) 2024-02-19T20:31:39.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:39.537+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:39.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:39.537+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:39.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:39.537+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:39.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:39.538+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:40.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:39.704+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:40.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:39.704+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:40.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:39.704+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:40.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:39.704+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:40.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:39.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:40.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:39.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:40.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:39.758+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:40.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:39.759+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:40.419 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:40.228+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:40.229614+0000) 2024-02-19T20:31:40.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:40.543+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:40.544725+0000) 2024-02-19T20:31:40.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:40.453+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:40.454557+0000) 2024-02-19T20:31:40.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:40.579+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:40.705 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:40.579+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:40.705 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:40.579+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:40.705 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:40.580+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:41.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:40.819+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:40.820161+0000) 2024-02-19T20:31:41.010 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:31:40.823+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63540 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:31:41.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:40.727+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:41.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:40.727+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:41.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:40.727+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:41.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:40.728+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:41.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:40.819+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:40.820251+0000) 2024-02-19T20:31:41.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:40.713+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:41.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:40.713+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:41.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:40.713+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:41.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:40.714+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:41.012 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:40.819+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:40.820774+0000) 2024-02-19T20:31:41.420 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:41.228+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:41.229785+0000) 2024-02-19T20:31:41.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:41.543+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:41.544910+0000) 2024-02-19T20:31:41.689 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:41.453+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:41.454747+0000) 2024-02-19T20:31:41.689 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:41.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:41.689 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:41.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:41.690 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:41.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:41.690 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:41.604+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:42.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:41.762+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:42.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:41.762+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:42.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:41.762+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:42.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:41.762+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:42.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:41.698+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:42.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:41.698+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:42.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:41.698+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:42.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:41.699+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:42.420 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:42.231+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:42.232225+0000) 2024-02-19T20:31:42.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:42.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:42.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:42.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:42.712 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:42.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:42.716 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:42.658+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:42.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:42.546+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:42.547558+0000) 2024-02-19T20:31:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:42.455+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:42.456794+0000) 2024-02-19T20:31:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:42.610+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:42.610+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:42.610+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:42.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:42.610+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:43.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:42.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:43.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:42.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:43.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:42.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:43.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:42.721+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:43.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:43.233+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:43.234559+0000) 2024-02-19T20:31:43.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:43.458+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:43.459397+0000) 2024-02-19T20:31:43.610 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:43.549+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:43.550130+0000) 2024-02-19T20:31:44.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:43.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:44.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:43.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:44.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:43.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:44.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:43.735+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:44.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:43.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:44.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:43.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:44.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:43.619+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:44.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:43.620+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:44.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:43.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:44.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:43.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:44.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:43.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:44.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:43.625+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:44.420 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:44.235+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:44.236750+0000) 2024-02-19T20:31:44.704 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:44.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:44.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:44.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:44.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:44.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:44.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:44.668+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:44.705 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:44.551+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:44.552369+0000) 2024-02-19T20:31:44.705 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:44.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:44.706 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:44.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:44.706 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:44.714+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:44.706 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:44.460+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:44.461520+0000) 2024-02-19T20:31:44.706 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:44.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:44.706 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:44.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:44.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:44.672+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:44.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:44.673+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:45.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:44.714+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:45.420 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:45.238+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:45.239620+0000) 2024-02-19T20:31:45.687 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:45.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:45.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:45.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:45.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:45.697+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:45.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:45.462+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:45.463602+0000) 2024-02-19T20:31:45.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:45.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:45.689 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:45.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:45.689 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:45.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:45.689 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:45.672+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:45.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:45.553+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:45.554025+0000) 2024-02-19T20:31:45.938 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:31:45.830+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63580 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:31:45.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:45.827+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:45.828206+0000) 2024-02-19T20:31:45.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:45.698+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:45.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:45.827+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:45.828916+0000) 2024-02-19T20:31:45.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:45.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:45.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:45.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:45.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:45.697+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:45.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:45.698+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:45.940 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:45.827+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:45.828221+0000) 2024-02-19T20:31:46.508 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:46.462+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:46.463779+0000) 2024-02-19T20:31:46.509 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:46.238+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:46.239848+0000) 2024-02-19T20:31:47.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:46.655+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:47.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:46.655+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:47.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:46.655+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:47.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:46.656+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:47.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:46.666+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:47.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:46.666+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:47.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:46.666+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:47.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:46.666+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:47.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:46.553+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:46.554202+0000) 2024-02-19T20:31:47.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:46.728+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:47.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:46.728+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:47.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:46.728+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:47.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:46.728+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:47.420 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:47.241+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:47.242265+0000) 2024-02-19T20:31:47.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:47.555+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:47.556289+0000) 2024-02-19T20:31:47.686 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:47.465+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:47.466176+0000) 2024-02-19T20:31:47.686 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:47.669+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:47.686 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:47.669+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:47.686 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:47.669+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:47.686 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:47.670+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:48.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:47.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:48.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:47.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:48.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:47.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:48.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:47.696+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:48.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:47.729+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:48.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:47.729+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:48.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:47.729+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:48.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:47.729+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:48.342 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:48.243+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:48.244832+0000) 2024-02-19T20:31:48.614 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:48.467+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:48.468212+0000) 2024-02-19T20:31:48.614 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:48.557+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:48.558881+0000) 2024-02-19T20:31:49.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:48.687+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:49.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:48.687+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:49.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:48.687+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:49.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:48.687+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:49.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:48.700+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:49.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:48.700+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:49.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:48.700+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:49.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:48.701+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:49.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:48.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:49.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:48.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:49.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:48.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:49.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:48.624+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:49.420 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:49.246+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:49.247395+0000) 2024-02-19T20:31:49.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:49.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:49.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:49.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:49.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:49.661+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:49.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:49.662+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:49.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:49.560+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:49.561162+0000) 2024-02-19T20:31:49.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:49.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:49.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:49.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:49.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:49.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:49.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:49.469+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:49.470157+0000) 2024-02-19T20:31:49.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:49.674+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:49.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:49.674+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:49.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:49.674+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:49.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:49.675+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:50.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:49.682+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:50.420 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:50.248+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:50.249429+0000) 2024-02-19T20:31:50.688 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:50.562+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:50.563196+0000) 2024-02-19T20:31:50.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:50.676+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:50.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:50.676+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:50.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:50.676+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:50.690 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:50.676+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:50.690 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:50.471+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:50.472232+0000) 2024-02-19T20:31:50.690 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:50.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:50.691 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:50.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:50.691 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:50.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:50.948 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:31:50.838+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63621 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:31:50.948 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:50.835+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:50.836552+0000) 2024-02-19T20:31:50.949 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:50.698+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:50.949 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:50.835+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:50.836344+0000) 2024-02-19T20:31:50.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:50.710+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:50.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:50.710+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:50.949 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:50.710+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:50.950 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:50.711+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:50.950 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:50.835+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:50.836377+0000) 2024-02-19T20:31:51.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:51.248+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:51.249633+0000) 2024-02-19T20:31:51.509 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:51.471+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:51.472447+0000) 2024-02-19T20:31:52.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:51.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:52.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:51.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:52.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:51.632+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:52.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:51.632+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:52.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:51.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:52.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:51.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:52.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:51.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:52.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:51.672+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:52.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:51.562+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:51.563356+0000) 2024-02-19T20:31:52.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:51.687+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:52.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:51.687+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:52.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:51.687+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:52.011 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:51.687+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:52.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:52.251+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:52.252154+0000) 2024-02-19T20:31:52.694 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:52.620+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:52.694 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:52.620+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:52.694 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:52.620+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:52.694 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:52.621+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:52.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:52.473+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:52.474881+0000) 2024-02-19T20:31:52.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:52.655+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:52.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:52.655+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:52.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:52.655+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:52.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:52.655+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:52.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:52.564+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:52.565677+0000) 2024-02-19T20:31:52.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:52.703+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:52.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:52.703+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:52.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:52.703+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:53.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:52.704+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:53.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:53.253+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:53.254725+0000) 2024-02-19T20:31:53.692 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:53.475+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:53.476883+0000) 2024-02-19T20:31:53.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:53.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:53.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:53.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:53.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:53.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:53.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:53.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:53.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:53.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:53.694 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:53.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:53.694 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:53.669+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:53.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:53.566+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:53.567963+0000) 2024-02-19T20:31:53.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:53.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:53.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:53.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:53.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:53.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:53.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:53.659+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:54.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:53.703+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:54.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:54.257+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:54.258061+0000) 2024-02-19T20:31:54.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:54.478+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:54.479001+0000) 2024-02-19T20:31:54.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:54.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:54.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:54.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:54.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:54.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:54.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:54.663+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:54.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:54.664+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:54.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:54.569+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:54.570810+0000) 2024-02-19T20:31:54.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:54.622+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:54.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:54.622+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:54.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:54.622+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:54.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:54.623+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:55.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:54.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:55.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:54.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:55.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:54.685+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:55.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:55.259+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:55.260948+0000) 2024-02-19T20:31:55.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:55.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:55.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:55.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:55.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:55.626+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:55.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:55.627+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:55.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:55.480+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:55.481047+0000) 2024-02-19T20:31:55.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:55.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:55.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:55.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:55.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:55.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:55.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:55.679+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:55.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:55.571+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:55.572658+0000) 2024-02-19T20:31:55.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:55.591+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:55.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:55.592+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:55.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:55.592+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:55.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:55.592+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:56.150 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:31:55.846+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63661 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:31:56.150 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:55.842+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:55.843738+0000) 2024-02-19T20:31:56.151 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:55.842+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:55.843139+0000) 2024-02-19T20:31:56.151 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:55.842+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:55.843077+0000) 2024-02-19T20:31:56.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:56.260+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:56.261128+0000) 2024-02-19T20:31:56.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:56.480+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:56.481241+0000) 2024-02-19T20:31:56.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:56.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:56.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:56.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:56.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:56.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:56.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:56.570+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:56.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:56.570+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:56.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:56.570+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:56.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:56.570+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:56.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:56.571+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:56.572834+0000) 2024-02-19T20:31:56.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:56.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:56.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:56.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:56.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:56.617+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:56.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:56.618+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:57.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:56.727+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:57.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:57.262+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:57.263210+0000) 2024-02-19T20:31:57.741 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:57.652+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:57.742 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:57.652+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:57.742 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:57.652+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:57.742 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:57.653+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:57.742 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:57.482+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:57.483592+0000) 2024-02-19T20:31:57.743 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:57.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:57.743 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:57.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:57.743 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:57.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:57.743 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:57.526+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:57.743 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:57.526+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:57.743 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:57.526+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:57.744 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:57.527+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:57.744 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:57.574+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:57.575370+0000) 2024-02-19T20:31:58.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:57.752+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:58.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:58.264+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:58.265486+0000) 2024-02-19T20:31:58.709 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:58.485+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:58.486778+0000) 2024-02-19T20:31:58.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:58.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:58.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:58.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:58.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:58.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:58.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:58.605+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:58.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:58.493+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:58.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:58.493+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:58.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:58.493+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:58.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:58.493+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:58.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:58.576+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:58.577613+0000) 2024-02-19T20:31:59.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:58.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:59.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:58.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:59.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:58.718+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:31:59.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:58.719+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:31:59.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:59.267+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:59.267710+0000) 2024-02-19T20:31:59.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:59.489+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:59.490343+0000) 2024-02-19T20:31:59.696 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:59.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:59.696 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:59.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:59.696 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:59.650+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:31:59.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:31:59.651+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:59.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:59.535+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:59.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:59.535+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:59.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:59.535+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:31:59.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:59.535+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:31:59.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:31:59.577+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:59.579697+0000) 2024-02-19T20:32:00.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:59.704+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:00.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:59.704+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:00.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:59.704+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:00.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:31:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:31:59.705+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:00.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:00.269+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:00.269998+0000) 2024-02-19T20:32:00.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:00.492+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:00.493531+0000) 2024-02-19T20:32:00.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:00.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:00.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:00.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:00.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:00.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:00.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:00.673+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:00.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:00.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:00.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:00.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:00.718 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:00.508+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:00.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:00.508+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:00.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:00.580+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:00.581419+0000) 2024-02-19T20:32:01.008 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:32:00.853+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63702 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:32:01.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:00.850+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:00.851702+0000) 2024-02-19T20:32:01.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:00.853+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:00.854698+0000) 2024-02-19T20:32:01.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:00.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:01.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:00.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:01.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:00.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:01.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:00.728+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:01.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:00.851+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:00.852164+0000) 2024-02-19T20:32:01.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:01.269+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:01.270227+0000) 2024-02-19T20:32:01.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:01.492+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:01.493795+0000) 2024-02-19T20:32:01.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:01.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:01.696 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:01.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:01.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:01.460+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:01.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:01.460+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:01.697 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:01.580+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:01.581690+0000) 2024-02-19T20:32:02.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:01.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:02.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:01.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:02.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:01.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:02.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:01.760+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:02.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:01.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:02.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:01.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:02.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:01.705+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:02.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:01.706+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:02.421 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:02.271+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:02.272848+0000) 2024-02-19T20:32:02.679 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:02.495+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:02.496292+0000) 2024-02-19T20:32:02.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:02.496+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:02.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:02.496+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:02.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:02.496+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:02.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:02.497+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:02.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:02.583+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:02.584044+0000) 2024-02-19T20:32:03.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:02.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:03.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:02.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:03.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:02.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:03.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:02.690+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:03.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:02.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:03.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:02.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:03.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:02.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:03.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:02.726+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:03.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:03.275+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:03.276342+0000) 2024-02-19T20:32:03.689 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:03.497+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:03.498591+0000) 2024-02-19T20:32:03.690 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:03.647+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:03.690 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:03.647+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:03.690 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:03.647+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:03.690 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:03.648+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:03.691 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:03.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:03.691 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:03.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:03.691 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:03.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:03.691 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:03.494+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:03.691 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:03.585+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:03.586221+0000) 2024-02-19T20:32:04.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:03.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:04.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:03.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:04.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:03.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:04.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:03.701+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:04.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:04.277+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:04.278554+0000) 2024-02-19T20:32:04.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:04.501+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:04.502176+0000) 2024-02-19T20:32:04.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:04.680+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:04.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:04.680+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:04.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:04.680+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:04.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:04.681+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:04.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:04.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:04.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:04.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:04.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:04.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:04.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:04.530+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:04.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:04.587+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:04.588272+0000) 2024-02-19T20:32:04.760 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:04.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:04.760 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:04.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:04.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:04.607+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:04.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:04.608+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:05.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:05.279+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:05.280390+0000) 2024-02-19T20:32:05.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:05.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:05.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:05.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:05.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:05.623+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:05.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:05.624+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:05.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:05.503+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:05.504020+0000) 2024-02-19T20:32:05.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:05.674+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:05.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:05.674+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:05.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:05.674+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:05.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:05.674+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:05.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:05.525+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:05.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:05.525+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:05.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:05.525+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:05.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:05.525+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:05.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:05.590+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:05.591227+0000) 2024-02-19T20:32:05.971 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:32:05.862+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63742 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:32:05.972 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:05.858+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:05.859408+0000) 2024-02-19T20:32:05.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:05.857+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:05.858612+0000) 2024-02-19T20:32:05.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:05.858+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:05.859126+0000) 2024-02-19T20:32:06.548 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:06.279+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:06.280584+0000) 2024-02-19T20:32:06.549 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:06.503+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:06.504197+0000) 2024-02-19T20:32:07.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:06.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:07.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:06.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:07.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:06.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:07.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:06.684+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:07.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:06.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:07.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:06.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:07.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:06.558+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:07.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:06.558+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:07.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:06.590+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:06.591402+0000) 2024-02-19T20:32:07.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:06.656+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:07.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:06.656+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:07.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:06.656+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:07.011 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:06.656+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:07.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:07.282+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:07.283747+0000) 2024-02-19T20:32:07.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:07.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:07.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:07.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:07.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:07.648+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:07.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:07.649+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:07.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:07.506+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:07.507554+0000) 2024-02-19T20:32:07.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:07.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:07.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:07.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:07.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:07.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:07.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:07.646+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:07.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:07.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:07.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:07.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:07.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:07.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:07.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:07.569+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:07.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:07.593+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:07.594659+0000) 2024-02-19T20:32:08.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:08.284+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:08.285800+0000) 2024-02-19T20:32:08.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:08.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:08.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:08.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:08.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:08.629+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:08.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:08.630+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:08.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:08.509+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:08.510877+0000) 2024-02-19T20:32:08.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:08.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:08.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:08.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:08.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:08.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:08.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:08.607+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:08.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:08.596+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:08.597402+0000) 2024-02-19T20:32:08.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:08.604+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:08.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:08.604+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:08.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:08.604+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:08.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:08.604+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:09.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:09.287+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:09.288254+0000) 2024-02-19T20:32:09.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:09.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:09.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:09.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:09.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:09.636+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:09.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:09.637+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:09.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:09.513+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:09.514341+0000) 2024-02-19T20:32:09.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:09.596+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:09.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:09.597+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:09.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:09.597+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:09.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:09.597+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:09.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:09.578+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:09.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:09.578+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:09.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:09.578+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:09.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:09.578+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:09.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:09.598+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:09.599510+0000) 2024-02-19T20:32:10.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:10.290+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:10.291202+0000) 2024-02-19T20:32:10.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:10.605+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:10.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:10.605+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:10.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:10.605+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:10.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:10.606+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:10.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:10.551+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:10.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:10.551+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:10.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:10.551+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:10.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:10.552+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:10.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:10.600+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:10.601788+0000) 2024-02-19T20:32:10.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:10.516+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:10.517228+0000) 2024-02-19T20:32:10.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:10.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:10.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:10.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:10.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:10.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:10.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:10.573+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:11.151 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:32:10.869+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63783 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:32:11.151 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:10.865+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:10.866968+0000) 2024-02-19T20:32:11.151 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:10.865+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:10.866771+0000) 2024-02-19T20:32:11.152 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:10.865+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:10.866916+0000) 2024-02-19T20:32:11.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:11.290+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:11.291398+0000) 2024-02-19T20:32:11.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:11.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:11.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:11.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:11.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:11.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:11.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:11.605+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:11.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:11.516+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:11.517450+0000) 2024-02-19T20:32:11.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:11.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:11.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:11.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:11.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:11.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:11.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:11.601+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:11.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:11.523+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:11.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:11.523+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:11.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:11.523+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:11.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:11.524+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:11.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:11.601+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:11.601995+0000) 2024-02-19T20:32:12.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:12.292+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:12.293517+0000) 2024-02-19T20:32:12.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:12.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:12.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:12.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:12.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:12.573+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:12.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:12.574+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:12.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:12.518+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:12.519369+0000) 2024-02-19T20:32:12.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:12.574+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:12.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:12.574+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:12.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:12.574+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:12.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:12.575+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:12.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:12.537+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:12.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:12.537+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:12.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:12.537+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:12.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:12.538+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:12.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:12.603+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:12.604379+0000) 2024-02-19T20:32:13.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:13.294+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:13.295675+0000) 2024-02-19T20:32:13.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:13.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:13.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:13.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:13.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:13.546+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:13.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:13.547+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:13.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:13.526+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:13.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:13.526+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:13.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:13.526+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:13.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:13.526+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:13.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:13.605+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:13.606348+0000) 2024-02-19T20:32:13.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:13.520+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:13.521465+0000) 2024-02-19T20:32:13.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:13.561+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:13.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:13.561+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:13.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:13.561+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:13.761 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:13.562+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:14.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:14.296+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:14.297758+0000) 2024-02-19T20:32:14.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:14.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:14.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:14.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:14.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:14.514+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:14.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:14.515+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:14.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:14.522+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:14.523623+0000) 2024-02-19T20:32:14.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:14.549+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:14.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:14.549+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:14.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:14.549+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:14.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:14.550+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:14.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:14.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:14.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:14.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:14.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:14.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:14.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:14.528+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:14.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:14.608+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:14.609477+0000) 2024-02-19T20:32:15.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:15.298+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:15.299428+0000) 2024-02-19T20:32:15.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:15.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:15.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:15.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:15.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:15.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:15.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:15.540+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:15.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:15.524+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:15.525672+0000) 2024-02-19T20:32:15.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:15.544+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:15.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:15.544+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:15.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:15.544+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:15.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:15.545+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:15.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:15.554+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:15.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:15.554+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:15.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:15.554+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:15.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:15.554+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:15.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:15.611+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:15.612141+0000) 2024-02-19T20:32:16.151 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:15.873+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:15.874635+0000) 2024-02-19T20:32:16.152 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:32:15.877+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63823 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:32:16.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:15.873+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:15.874318+0000) 2024-02-19T20:32:16.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:15.874+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:15.875377+0000) 2024-02-19T20:32:16.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:16.298+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:16.299614+0000) 2024-02-19T20:32:16.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:16.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:16.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:16.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:16.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:16.539+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:16.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:16.540+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:16.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:16.524+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:16.525883+0000) 2024-02-19T20:32:16.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:16.592+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:16.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:16.592+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:16.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:16.592+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:16.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:16.593+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:16.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:16.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:16.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:16.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:16.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:16.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:16.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:16.529+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:16.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:16.611+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:16.612310+0000) 2024-02-19T20:32:17.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:17.300+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:17.301779+0000) 2024-02-19T20:32:17.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:17.561+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:17.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:17.561+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:17.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:17.561+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:17.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:17.562+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:17.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:17.527+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:17.528790+0000) 2024-02-19T20:32:17.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:17.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:17.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:17.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:17.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:17.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:17.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:17.616+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:17.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:17.520+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:17.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:17.520+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:17.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:17.520+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:17.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:17.520+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:17.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:17.613+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:17.614295+0000) 2024-02-19T20:32:18.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:18.303+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:18.304220+0000) 2024-02-19T20:32:18.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:18.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:18.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:18.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:18.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:18.553+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:18.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:18.553+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:18.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:18.530+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:18.531091+0000) 2024-02-19T20:32:18.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:18.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:18.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:18.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:18.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:18.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:18.760 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:18.616+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:18.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:18.514+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:18.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:18.514+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:18.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:18.514+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:18.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:18.515+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:18.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:18.616+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:18.617902+0000) 2024-02-19T20:32:19.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:19.305+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:19.306576+0000) 2024-02-19T20:32:19.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:19.532+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:19.533200+0000) 2024-02-19T20:32:19.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:19.659+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:19.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:19.659+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:19.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:19.659+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:19.759 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:19.660+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:19.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:19.524+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:19.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:19.524+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:19.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:19.524+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:19.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:19.524+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:19.761 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:19.620+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:19.621480+0000) 2024-02-19T20:32:19.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:19.564+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:19.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:19.564+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:19.761 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:19.564+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:19.762 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:19.565+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:20.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:20.308+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:20.309590+0000) 2024-02-19T20:32:20.681 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:20.534+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:20.535331+0000) 2024-02-19T20:32:20.681 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:20.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:20.681 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:20.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:20.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:20.543+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:20.682 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:20.544+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:20.682 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:20.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:20.682 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:20.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:20.682 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:20.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:20.682 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:20.480+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:20.683 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:20.623+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:20.624356+0000) 2024-02-19T20:32:20.995 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:32:20.884+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63864 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:32:20.996 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:20.880+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:20.881928+0000) 2024-02-19T20:32:20.996 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:20.881+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:20.882628+0000) 2024-02-19T20:32:20.996 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:20.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:20.997 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:20.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:20.997 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:20.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:20.997 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:20.692+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:20.997 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:20.881+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:20.882108+0000) 2024-02-19T20:32:21.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:21.308+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:21.309817+0000) 2024-02-19T20:32:21.567 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:21.534+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:21.535538+0000) 2024-02-19T20:32:21.567 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:21.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:21.567 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:21.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:21.567 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:21.487+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:21.568 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:21.488+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:22.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:21.623+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:21.624554+0000) 2024-02-19T20:32:22.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:21.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:22.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:21.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:22.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:21.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:22.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:21.577+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:22.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:21.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:22.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:21.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:22.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:21.740+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:22.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:21.740+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:22.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:22.311+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:22.312277+0000) 2024-02-19T20:32:22.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:22.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:22.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:22.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:22.717 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:22.550+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:22.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:22.550+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:22.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:22.537+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:22.537976+0000) 2024-02-19T20:32:22.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:22.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:22.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:22.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:22.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:22.727+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:22.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:22.525+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:22.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:22.525+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:22.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:22.525+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:22.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:22.526+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:22.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:22.627+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:22.628144+0000) 2024-02-19T20:32:23.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:22.728+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:23.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:23.313+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:23.314950+0000) 2024-02-19T20:32:23.691 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:23.539+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:23.540056+0000) 2024-02-19T20:32:23.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:23.566+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:23.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:23.566+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:23.692 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:23.566+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:23.693 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:23.567+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:23.693 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:23.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:23.693 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:23.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:23.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:23.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:23.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:23.573+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:23.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:23.629+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:23.630233+0000) 2024-02-19T20:32:24.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:23.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:24.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:23.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:24.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:23.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:24.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:23.702+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:24.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:24.316+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:24.317688+0000) 2024-02-19T20:32:24.684 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:24.541+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:24.542180+0000) 2024-02-19T20:32:24.684 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:24.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:24.685 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:24.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:24.685 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:24.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:24.685 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:24.568+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:24.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:24.612+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:24.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:24.612+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:24.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:24.612+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:24.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:24.612+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:24.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:24.631+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:24.632393+0000) 2024-02-19T20:32:25.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:24.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:25.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:24.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:25.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:24.694+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:25.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:24.695+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:25.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:25.318+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:25.319897+0000) 2024-02-19T20:32:25.729 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:25.543+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:25.544003+0000) 2024-02-19T20:32:25.729 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:25.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:25.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:25.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:25.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:25.542+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:25.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:25.542+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:25.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:25.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:25.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:25.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:25.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:25.589+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:25.731 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:25.590+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:25.731 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:25.633+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:25.634083+0000) 2024-02-19T20:32:26.001 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:32:25.893+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63904 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:32:26.002 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:25.889+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:25.890703+0000) 2024-02-19T20:32:26.002 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:25.889+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:25.890498+0000) 2024-02-19T20:32:26.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:25.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:26.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:25.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:26.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:25.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:26.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:25.740+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:26.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:25.888+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:25.889960+0000) 2024-02-19T20:32:26.567 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:26.543+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:26.544191+0000) 2024-02-19T20:32:26.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:26.319+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:26.320080+0000) 2024-02-19T20:32:26.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:26.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:26.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:26.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:26.568 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:26.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:26.568 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:26.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:26.568 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:26.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:26.568 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:26.547+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:27.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:26.633+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:26.634250+0000) 2024-02-19T20:32:27.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:26.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:27.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:26.578+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:27.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:26.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:27.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:26.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:27.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:26.742+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:27.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:26.743+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:27.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:27.321+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:27.322143+0000) 2024-02-19T20:32:27.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:27.545+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:27.546600+0000) 2024-02-19T20:32:27.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:27.600+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:27.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:27.600+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:27.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:27.600+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:27.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:27.601+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:27.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:27.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:27.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:27.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:27.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:27.550+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:27.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:27.551+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:27.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:27.636+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:27.637587+0000) 2024-02-19T20:32:28.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:27.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:28.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:27.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:28.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:27.776+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:28.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:27.777+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:28.423 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:28.323+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:28.324871+0000) 2024-02-19T20:32:28.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:28.549+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:28.550118+0000) 2024-02-19T20:32:28.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:28.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:28.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:28.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:28.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:28.604+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:28.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:28.605+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:28.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:28.567+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:28.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:28.567+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:28.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:28.567+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:28.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:28.567+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:28.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:28.638+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:28.639741+0000) 2024-02-19T20:32:29.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:28.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:29.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:28.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:29.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:28.804+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:29.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:28.804+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:29.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:29.325+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:29.326877+0000) 2024-02-19T20:32:29.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:29.552+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:29.553506+0000) 2024-02-19T20:32:29.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:29.620+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:29.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:29.620+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:29.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:29.620+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:29.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:29.621+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:29.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:29.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:29.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:29.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:29.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:29.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:29.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:29.559+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:29.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:29.640+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:29.641854+0000) 2024-02-19T20:32:30.152 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:29.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:30.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:29.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:30.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:29.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:30.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:29.820+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:30.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:30.328+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:30.329016+0000) 2024-02-19T20:32:30.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:30.554+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:30.555205+0000) 2024-02-19T20:32:30.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:30.656+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:30.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:30.656+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:30.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:30.656+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:30.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:30.657+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:30.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:30.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:30.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:30.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:30.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:30.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:30.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:30.530+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:30.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:30.642+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:30.643721+0000) 2024-02-19T20:32:31.153 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:32:30.900+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63945 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:32:31.153 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:30.897+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:30.898764+0000) 2024-02-19T20:32:31.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:30.897+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:30.898636+0000) 2024-02-19T20:32:31.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:30.857+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:31.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:30.857+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:31.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:30.857+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:31.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:30.858+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:31.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:30.897+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:30.898219+0000) 2024-02-19T20:32:31.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:31.328+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:31.329196+0000) 2024-02-19T20:32:31.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:31.554+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:31.555411+0000) 2024-02-19T20:32:31.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:31.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:31.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:31.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:31.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:31.635+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:31.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:31.637+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:31.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:31.491+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:31.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:31.491+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:31.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:31.491+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:31.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:31.491+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:31.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:31.642+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:31.643956+0000) 2024-02-19T20:32:32.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:31.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:32.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:31.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:32.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:31.817+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:32.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:31.818+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:32.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:32.331+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:32.332564+0000) 2024-02-19T20:32:32.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:32.556+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:32.557435+0000) 2024-02-19T20:32:32.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:32.634+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:32.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:32.634+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:32.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:32.634+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:32.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:32.635+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:32.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:32.526+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:32.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:32.526+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:32.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:32.526+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:32.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:32.527+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:32.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:32.645+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:32.646001+0000) 2024-02-19T20:32:33.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:32.855+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:33.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:32.855+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:33.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:32.855+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:33.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:32.856+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:33.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:33.333+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:33.334808+0000) 2024-02-19T20:32:33.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:33.558+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:33.559796+0000) 2024-02-19T20:32:33.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:33.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:33.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:33.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:33.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:33.611+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:33.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:33.612+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:33.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:33.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:33.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:33.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:33.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:33.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:33.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:33.549+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:33.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:33.647+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:33.648043+0000) 2024-02-19T20:32:34.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:33.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:34.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:33.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:34.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:33.902+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:34.153 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:33.903+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:34.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:34.335+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:34.336940+0000) 2024-02-19T20:32:34.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:34.562+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:34.563296+0000) 2024-02-19T20:32:34.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:34.653+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:34.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:34.653+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:34.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:34.653+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:34.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:34.654+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:34.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:34.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:34.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:34.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:34.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:34.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:34.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:34.503+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:34.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:34.650+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:34.651662+0000) 2024-02-19T20:32:35.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:34.922+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:35.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:34.922+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:35.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:34.922+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:35.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:34.923+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:35.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:35.338+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:35.339166+0000) 2024-02-19T20:32:35.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:35.564+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:35.565473+0000) 2024-02-19T20:32:35.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:35.679+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:35.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:35.679+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:35.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:35.679+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:35.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:35.680+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:35.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:35.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:35.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:35.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:35.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:35.475+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:35.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:35.476+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:35.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:35.653+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:35.654506+0000) 2024-02-19T20:32:36.153 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:32:35.908+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 63985 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:32:36.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:35.906+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:35.907149+0000) 2024-02-19T20:32:36.154 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:35.905+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:35.906124+0000) 2024-02-19T20:32:36.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:35.905+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:35.906594+0000) 2024-02-19T20:32:36.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:35.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:36.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:35.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:36.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:35.949+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:36.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:35.950+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:36.424 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:36.338+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:36.339357+0000) 2024-02-19T20:32:36.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:36.564+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:36.565676+0000) 2024-02-19T20:32:36.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:36.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:36.675 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:36.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:36.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:36.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:36.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:36.490+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:36.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:36.653+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:36.654713+0000) 2024-02-19T20:32:37.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:36.685+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:37.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:36.685+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:37.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:36.685+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:37.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:36.686+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:37.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:36.934+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:37.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:36.934+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:37.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:36.934+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:37.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:36.935+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:37.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:37.340+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:37.341729+0000) 2024-02-19T20:32:37.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:37.566+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:37.567735+0000) 2024-02-19T20:32:37.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:37.649+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:37.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:37.649+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:37.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:37.649+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:37.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:37.650+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:37.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:37.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:37.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:37.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:37.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:37.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:37.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:37.530+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:37.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:37.656+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:37.657178+0000) 2024-02-19T20:32:38.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:37.955+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:38.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:37.955+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:38.259 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:37.955+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:38.259 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:37.956+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:38.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:38.342+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:38.343797+0000) 2024-02-19T20:32:38.585 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:38.569+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:38.570113+0000) 2024-02-19T20:32:38.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:38.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:38.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:38.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:38.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:38.569+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:38.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:38.570+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:38.931 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:38.658+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:38.659566+0000) 2024-02-19T20:32:38.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:38.641+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:38.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:38.641+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:38.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:38.641+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:38.932 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:38.642+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:39.192 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:38.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:39.192 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:38.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:39.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:38.941+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:39.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:38.942+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:39.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:39.346+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:39.347079+0000) 2024-02-19T20:32:39.906 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:39.572+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:39.573650+0000) 2024-02-19T20:32:39.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:39.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:39.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:39.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:39.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:39.624+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:39.907 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:39.624+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:39.908 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:39.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:39.908 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:39.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:39.908 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:39.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:39.908 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:39.557+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:39.908 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:39.661+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:39.662020+0000) 2024-02-19T20:32:40.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:39.917+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:40.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:39.917+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:40.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:39.917+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:40.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:39.917+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:40.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:40.347+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:40.348836+0000) 2024-02-19T20:32:40.860 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:40.664+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:40.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:40.664+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:40.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:40.664+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:40.861 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:40.665+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:40.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:40.574+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:40.575866+0000) 2024-02-19T20:32:40.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:40.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:40.861 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:40.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:40.862 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:40.871+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:40.862 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:40.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:40.862 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:40.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:40.862 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:40.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:40.862 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:40.588+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:40.862 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:40.662+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:40.663794+0000) 2024-02-19T20:32:41.153 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:32:40.916+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64026 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:32:41.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:40.912+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:40.913912+0000) 2024-02-19T20:32:41.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:40.871+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:41.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:40.912+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:40.913734+0000) 2024-02-19T20:32:41.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:40.912+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:40.913797+0000) 2024-02-19T20:32:41.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:41.348+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:41.349040+0000) 2024-02-19T20:32:41.684 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:41.575+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:41.576068+0000) 2024-02-19T20:32:41.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:41.611+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:41.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:41.611+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:41.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:41.611+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:41.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:41.611+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:41.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:41.662+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:41.663989+0000) 2024-02-19T20:32:42.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:41.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:42.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:41.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:42.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:41.695+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:42.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:41.695+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:42.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:41.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:42.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:41.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:42.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:41.833+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:42.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:41.834+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:42.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:42.351+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:42.352384+0000) 2024-02-19T20:32:42.684 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:42.577+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:42.578323+0000) 2024-02-19T20:32:42.684 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:42.629+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:42.684 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:42.629+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:42.684 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:42.629+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:42.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:42.629+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:42.685 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:42.665+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:42.666524+0000) 2024-02-19T20:32:43.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:42.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:43.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:42.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:43.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:42.694+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:43.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:42.695+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:43.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:42.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:43.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:42.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:43.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:42.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:43.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:42.860+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:43.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:43.353+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:43.354803+0000) 2024-02-19T20:32:43.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:43.580+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:43.581615+0000) 2024-02-19T20:32:43.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:43.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:43.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:43.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:43.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:43.658+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:43.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:43.659+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:43.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:43.582+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:43.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:43.582+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:43.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:43.582+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:43.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:43.583+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:43.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:43.668+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:43.669879+0000) 2024-02-19T20:32:44.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:43.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:44.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:43.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:44.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:43.813+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:44.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:43.814+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:44.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:44.356+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:44.357014+0000) 2024-02-19T20:32:44.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:44.583+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:44.583999+0000) 2024-02-19T20:32:44.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:44.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:44.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:44.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:44.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:44.668+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:44.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:44.669+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:44.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:44.594+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:44.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:44.594+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:44.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:44.594+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:44.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:44.594+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:44.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:44.671+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:44.672299+0000) 2024-02-19T20:32:45.154 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:44.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:45.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:44.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:45.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:44.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:45.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:44.799+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:45.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:45.357+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:45.358874+0000) 2024-02-19T20:32:45.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:45.584+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:45.585769+0000) 2024-02-19T20:32:45.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:45.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:45.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:45.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:45.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:45.674+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:45.759 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:45.674+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:45.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:45.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:45.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:45.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:45.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:45.555+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:45.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:45.556+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:45.760 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:45.674+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:45.675256+0000) 2024-02-19T20:32:46.034 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:32:45.924+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64066 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:32:46.035 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:45.920+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:45.921641+0000) 2024-02-19T20:32:46.035 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:45.921+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:45.922290+0000) 2024-02-19T20:32:46.036 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:45.772+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:46.036 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:45.772+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:46.036 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:45.772+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:46.036 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:45.773+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:46.036 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:45.920+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:45.921403+0000) 2024-02-19T20:32:46.664 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:46.358+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:46.359063+0000) 2024-02-19T20:32:46.664 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:46.585+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:46.586014+0000) 2024-02-19T20:32:46.664 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:46.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:46.665 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:46.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:46.665 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:46.513+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:46.665 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:46.514+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:46.665 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:46.674+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:46.675478+0000) 2024-02-19T20:32:47.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:46.685+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:47.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:46.685+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:47.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:46.685+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:47.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:46.686+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:47.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:46.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:47.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:46.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:47.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:46.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:47.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:46.822+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:47.425 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:47.360+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:47.361541+0000) 2024-02-19T20:32:47.725 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:47.586+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:47.588137+0000) 2024-02-19T20:32:47.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:47.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:47.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:47.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:47.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:47.479+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:47.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:47.479+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:47.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:47.676+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:47.677868+0000) 2024-02-19T20:32:48.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:47.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:48.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:47.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:48.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:47.735+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:48.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:47.735+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:48.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:47.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:48.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:47.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:48.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:47.852+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:48.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:47.853+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:48.420 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:48.362+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:48.363872+0000) 2024-02-19T20:32:48.681 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:48.589+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:48.590600+0000) 2024-02-19T20:32:48.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:48.430+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:48.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:48.430+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:48.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:48.430+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:48.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:48.431+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:48.682 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:48.679+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:48.680172+0000) 2024-02-19T20:32:49.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:48.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:49.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:48.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:49.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:48.783+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:49.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:48.784+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:49.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:48.851+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:49.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:48.851+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:49.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:48.851+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:49.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:48.852+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:49.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:49.365+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:49.366017+0000) 2024-02-19T20:32:49.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:49.591+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:49.592738+0000) 2024-02-19T20:32:49.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:49.469+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:49.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:49.469+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:49.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:49.469+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:49.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:49.469+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:49.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:49.682+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:49.683141+0000) 2024-02-19T20:32:50.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:49.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:50.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:49.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:50.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:49.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:50.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:49.781+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:50.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:49.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:50.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:49.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:50.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:49.860+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:50.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:49.861+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:50.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:50.367+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:50.368834+0000) 2024-02-19T20:32:50.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:50.593+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:50.594646+0000) 2024-02-19T20:32:50.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:50.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:50.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:50.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:50.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:50.436+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:50.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:50.437+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:50.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:50.683+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:50.684934+0000) 2024-02-19T20:32:51.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:50.928+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:50.929225+0000) 2024-02-19T20:32:51.155 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:32:50.931+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64107 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:32:51.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:50.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:51.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:50.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:51.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:50.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:51.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:50.820+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:51.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:50.928+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:50.929664+0000) 2024-02-19T20:32:51.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:50.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:51.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:50.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:51.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:50.829+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:51.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:50.830+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:51.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:50.928+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:50.929693+0000) 2024-02-19T20:32:51.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:51.368+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:51.369041+0000) 2024-02-19T20:32:51.426 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:51.406+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:51.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:51.406+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:51.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:51.406+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:51.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:51.406+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:51.751 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:51.593+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:51.594858+0000) 2024-02-19T20:32:51.752 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:51.684+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:51.685107+0000) 2024-02-19T20:32:52.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:51.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:52.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:51.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:52.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:51.796+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:52.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:51.796+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:52.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:51.789+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:52.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:51.789+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:52.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:51.789+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:52.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:51.789+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:52.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:52.370+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:52.371636+0000) 2024-02-19T20:32:52.749 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:52.596+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:52.597338+0000) 2024-02-19T20:32:52.749 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:52.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:52.750 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:52.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:52.750 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:52.440+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:52.750 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:52.441+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:52.750 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:52.686+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:52.687583+0000) 2024-02-19T20:32:53.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:52.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:53.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:52.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:53.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:52.760+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:53.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:52.761+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:53.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:52.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:53.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:52.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:53.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:52.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:53.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:52.838+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:53.422 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:53.372+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:53.373759+0000) 2024-02-19T20:32:53.678 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:53.598+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:53.599514+0000) 2024-02-19T20:32:53.679 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:53.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:53.679 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:53.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:53.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:53.432+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:53.680 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:53.433+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:54.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:53.689+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:53.690104+0000) 2024-02-19T20:32:54.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:53.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:54.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:53.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:54.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:53.786+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:54.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:53.787+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:54.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:53.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:54.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:53.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:54.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:53.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:54.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:53.843+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:54.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:54.375+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:54.376250+0000) 2024-02-19T20:32:54.681 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:54.602+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:54.603045+0000) 2024-02-19T20:32:54.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:54.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:54.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:54.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:54.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:54.476+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:54.682 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:54.476+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:55.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:54.691+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:54.692422+0000) 2024-02-19T20:32:55.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:54.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:55.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:54.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:55.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:54.832+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:55.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:54.833+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:55.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:54.747+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:55.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:54.747+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:55.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:54.747+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:55.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:54.748+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:55.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:55.377+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:55.378070+0000) 2024-02-19T20:32:55.683 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:55.603+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:55.604895+0000) 2024-02-19T20:32:55.683 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:55.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:55.683 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:55.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:55.683 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:55.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:55.684 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:55.474+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:55.933 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:32:55.939+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64147 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:32:55.934 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:55.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:55.934 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:55.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:55.934 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:55.825+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:55.934 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:55.826+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:55.934 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:55.935+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:55.936411+0000) 2024-02-19T20:32:55.935 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:55.693+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:55.694454+0000) 2024-02-19T20:32:55.935 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:55.935+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:55.936760+0000) 2024-02-19T20:32:55.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:55.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:55.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:55.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:55.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:55.753+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:55.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:55.754+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:55.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:55.936+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:55.937272+0000) 2024-02-19T20:32:56.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:56.377+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:56.378236+0000) 2024-02-19T20:32:56.508 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:56.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:56.508 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:56.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:56.508 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:56.457+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:56.509 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:56.457+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:57.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:56.693+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:56.694625+0000) 2024-02-19T20:32:57.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:56.722+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:57.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:56.722+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:57.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:56.722+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:57.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:56.722+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:57.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:56.604+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:56.605103+0000) 2024-02-19T20:32:57.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:56.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:57.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:56.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:57.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:56.843+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:57.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:56.844+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:57.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:57.379+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:57.380245+0000) 2024-02-19T20:32:57.686 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:57.606+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:57.607075+0000) 2024-02-19T20:32:57.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:57.503+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:57.686 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:57.503+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:57.687 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:57.503+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:57.687 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:57.504+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:58.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:57.696+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:57.697882+0000) 2024-02-19T20:32:58.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:57.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:58.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:57.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:58.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:57.767+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:58.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:57.768+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:58.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:57.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:58.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:57.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:58.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:57.819+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:58.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:57.819+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:58.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:58.381+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:58.382952+0000) 2024-02-19T20:32:58.689 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:58.608+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:58.609019+0000) 2024-02-19T20:32:58.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:58.545+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:58.690 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:58.545+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:58.690 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:58.545+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:58.690 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:58.546+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:59.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:58.700+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:58.701161+0000) 2024-02-19T20:32:59.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:58.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:59.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:58.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:59.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:58.848+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:32:59.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:58.849+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:32:59.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:58.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:59.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:58.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:59.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:58.803+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:32:59.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:58.804+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:32:59.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:59.385+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:59.386485+0000) 2024-02-19T20:32:59.691 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:59.611+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:59.612536+0000) 2024-02-19T20:32:59.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:59.557+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:59.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:59.557+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:59.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:59.557+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:32:59.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:59.558+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:00.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:32:59.702+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:59.703362+0000) 2024-02-19T20:33:00.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:59.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:00.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:59.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:00.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:59.810+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:00.010 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:32:59.811+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:00.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:59.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:00.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:59.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:00.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:59.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:00.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:32:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:32:59.807+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:00.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:00.387+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:00.388244+0000) 2024-02-19T20:33:00.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:00.614+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:00.615651+0000) 2024-02-19T20:33:00.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:00.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:00.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:00.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:00.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:00.556+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:00.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:00.557+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:00.952 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:33:00.947+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64188 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:33:00.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:00.704+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:00.705545+0000) 2024-02-19T20:33:00.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:00.943+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:00.944729+0000) 2024-02-19T20:33:00.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:00.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:00.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:00.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:00.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:00.791+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:00.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:00.792+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:00.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:00.943+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:00.944449+0000) 2024-02-19T20:33:00.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:00.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:00.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:00.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:00.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:00.762+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:00.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:00.763+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:00.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:00.943+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:00.944569+0000) 2024-02-19T20:33:01.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:01.387+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:01.388445+0000) 2024-02-19T20:33:01.811 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:01.614+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:01.615856+0000) 2024-02-19T20:33:01.811 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:01.715+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:01.811 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:01.715+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:01.812 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:01.715+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:01.812 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:01.716+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:01.812 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:01.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:01.812 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:01.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:01.812 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:01.527+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:01.813 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:01.528+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:01.813 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:01.704+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:01.705778+0000) 2024-02-19T20:33:02.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:01.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:02.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:01.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:02.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:01.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:02.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:01.822+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:02.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:02.389+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:02.390592+0000) 2024-02-19T20:33:02.680 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:02.617+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:02.618255+0000) 2024-02-19T20:33:02.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:02.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:02.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:02.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:02.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:02.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:02.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:02.547+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:03.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:02.706+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:02.707963+0000) 2024-02-19T20:33:03.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:02.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:03.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:02.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:03.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:02.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:03.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:02.863+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:03.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:02.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:03.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:02.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:03.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:02.691+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:03.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:02.692+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:03.427 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:03.391+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:03.392703+0000) 2024-02-19T20:33:03.691 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:03.619+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:03.620394+0000) 2024-02-19T20:33:03.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:03.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:03.692 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:03.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:03.693 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:03.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:03.693 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:03.587+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:04.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:03.709+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:03.710380+0000) 2024-02-19T20:33:04.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:03.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:04.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:03.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:04.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:03.702+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:04.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:03.703+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:04.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:03.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:04.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:03.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:04.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:03.884+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:04.011 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:03.885+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:04.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:04.393+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:04.394774+0000) 2024-02-19T20:33:04.612 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:04.601+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:04.612 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:04.601+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:04.612 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:04.601+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:04.613 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:04.602+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:04.909 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:04.622+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:04.623720+0000) 2024-02-19T20:33:04.910 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:04.711+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:04.712811+0000) 2024-02-19T20:33:04.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:04.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:04.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:04.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:04.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:04.672+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:04.910 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:04.673+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:05.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:04.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:05.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:04.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:05.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:04.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:05.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:04.921+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:05.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:05.396+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:05.397875+0000) 2024-02-19T20:33:05.941 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:05.625+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:05.626780+0000) 2024-02-19T20:33:05.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:05.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:05.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:05.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:05.941 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:05.689+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:05.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:05.689+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:05.942 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:05.633+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:05.942 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:05.633+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:05.942 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:05.633+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:05.942 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:05.634+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:05.942 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:05.715+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:05.716098+0000) 2024-02-19T20:33:06.215 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:33:05.955+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64228 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:33:06.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:05.951+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:05.952258+0000) 2024-02-19T20:33:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:05.951+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:05.952540+0000) 2024-02-19T20:33:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:05.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:05.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:05.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:06.216 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:05.958+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:06.217 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:05.951+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:05.952040+0000) 2024-02-19T20:33:06.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:06.397+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:06.398074+0000) 2024-02-19T20:33:06.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:06.626+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:06.627002+0000) 2024-02-19T20:33:06.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:06.726+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:06.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:06.726+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:06.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:06.726+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:06.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:06.726+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:06.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:06.605+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:06.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:06.605+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:06.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:06.605+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:06.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:06.606+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:06.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:06.715+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:06.716274+0000) 2024-02-19T20:33:07.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:06.993+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:07.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:06.993+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:07.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:06.993+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:07.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:06.994+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:07.616 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:07.399+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:07.400039+0000) 2024-02-19T20:33:07.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:07.628+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:07.629538+0000) 2024-02-19T20:33:07.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:07.747+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:07.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:07.747+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:07.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:07.747+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:07.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:07.748+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:07.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:07.626+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:07.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:07.626+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:07.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:07.626+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:37.309641+0000 (oldest deadline 2024-02-19T20:22:57.309641+0000) 2024-02-19T20:33:07.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:07.627+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:07.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:07.717+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:07.718806+0000) 2024-02-19T20:33:08.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:07.983+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:08.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:07.983+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:08.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:07.983+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:08.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:07.984+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:08.619 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:08.401+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:08.402462+0000) 2024-02-19T20:33:08.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:08.630+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:08.631540+0000) 2024-02-19T20:33:08.893 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:08.671+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:08.893 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:08.721+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:08.722380+0000) 2024-02-19T20:33:08.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:08.738+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:08.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:08.738+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:08.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:08.738+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:08.893 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:08.739+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:09.156 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:08.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:09.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:08.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:09.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:08.969+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:09.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:08.970+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:09.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:09.405+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:09.406000+0000) 2024-02-19T20:33:09.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:09.632+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:09.633793+0000) 2024-02-19T20:33:09.713 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:09.628+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:09.989 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:09.723+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:09.724881+0000) 2024-02-19T20:33:09.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:09.768+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:09.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:09.768+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:09.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:09.768+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:09.990 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:09.769+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:10.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:09.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:10.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:09.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:10.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:09.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:10.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:10.000+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:10.624 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:10.407+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:10.408186+0000) 2024-02-19T20:33:10.624 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:10.605+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:10.945 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:10.634+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:10.635812+0000) 2024-02-19T20:33:10.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:10.807+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:10.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:10.807+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:10.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:10.807+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:10.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:10.808+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:10.946 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:10.726+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:10.727926+0000) 2024-02-19T20:33:11.220 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:33:10.962+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64269 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:33:11.221 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:10.959+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:10.960615+0000) 2024-02-19T20:33:11.221 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:10.960+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:10.961400+0000) 2024-02-19T20:33:11.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:10.960+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:10.961027+0000) 2024-02-19T20:33:11.221 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:11.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:11.222 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:11.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:11.222 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:11.009+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:11.222 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:11.010+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:11.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:11.407+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:11.408366+0000) 2024-02-19T20:33:11.950 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:11.578+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:11.950 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:11.727+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:11.728106+0000) 2024-02-19T20:33:11.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:11.823+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:11.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:11.823+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:11.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:11.823+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:11.951 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:11.824+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:11.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:11.635+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:11.636028+0000) 2024-02-19T20:33:11.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:11.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:11.951 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:11.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:11.952 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:11.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:40.841656+0000 (oldest deadline 2024-02-19T20:23:00.841656+0000) 2024-02-19T20:33:12.223 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:11.961+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:12.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:12.409+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:12.410787+0000) 2024-02-19T20:33:12.843 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:12.637+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:12.638103+0000) 2024-02-19T20:33:12.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:12.537+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:12.844 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:12.729+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:12.730207+0000) 2024-02-19T20:33:13.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:12.957+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:13.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:12.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:13.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:12.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:13.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:12.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:13.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:12.855+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:13.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:13.412+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:13.413251+0000) 2024-02-19T20:33:13.720 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:13.639+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:13.640489+0000) 2024-02-19T20:33:13.720 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:13.556+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:14.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:13.978+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:14.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:13.731+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:13.732302+0000) 2024-02-19T20:33:14.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:13.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:14.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:13.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:14.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:13.873+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:14.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:13.874+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:14.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:14.414+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:14.415926+0000) 2024-02-19T20:33:14.722 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:14.641+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:14.642970+0000) 2024-02-19T20:33:14.723 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:14.572+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:15.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:14.980+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:15.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:14.733+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:14.734616+0000) 2024-02-19T20:33:15.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:14.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:15.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:14.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:15.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:14.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:15.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:14.861+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:15.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:15.416+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:15.417818+0000) 2024-02-19T20:33:15.726 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:15.644+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:15.645007+0000) 2024-02-19T20:33:15.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:15.555+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:15.976 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:33:15.970+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64309 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:33:15.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:15.946+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:15.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:15.966+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:15.967592+0000) 2024-02-19T20:33:15.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:15.736+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:15.737843+0000) 2024-02-19T20:33:15.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:15.966+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:15.967884+0000) 2024-02-19T20:33:15.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:15.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:15.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:15.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:15.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:15.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:15.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:15.863+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:15.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:15.967+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:15.968514+0000) 2024-02-19T20:33:16.633 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:16.573+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:16.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:16.417+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:16.418046+0000) 2024-02-19T20:33:16.893 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:16.737+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:16.738044+0000) 2024-02-19T20:33:16.893 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:16.644+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:16.645255+0000) 2024-02-19T20:33:16.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:16.904+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:16.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:16.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:16.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:16.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:16.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:16.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:16.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:16.842+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:17.728 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:17.647+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:17.648762+0000) 2024-02-19T20:33:17.728 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:17.420+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:17.421544+0000) 2024-02-19T20:33:17.729 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:17.570+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:17.729 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:17.739+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:17.740223+0000) 2024-02-19T20:33:18.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:17.927+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:18.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:17.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:18.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:17.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:18.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:17.888+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:22:43.464719+0000 (oldest deadline 2024-02-19T20:23:03.464719+0000) 2024-02-19T20:33:18.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:17.889+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:18.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:18.422+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:18.423970+0000) 2024-02-19T20:33:18.731 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:18.651+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:18.652099+0000) 2024-02-19T20:33:18.731 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:18.594+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:18.989 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:18.936+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:18.989 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:18.931+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:18.989 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:18.741+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:18.742542+0000) 2024-02-19T20:33:19.642 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:19.425+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:19.426026+0000) 2024-02-19T20:33:19.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:19.609+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:19.918 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:19.653+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:19.654092+0000) 2024-02-19T20:33:19.919 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:19.743+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:19.744786+0000) 2024-02-19T20:33:20.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:19.929+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:20.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:19.957+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:20.428 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:20.426+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:20.427896+0000) 2024-02-19T20:33:20.736 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:20.656+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:20.657061+0000) 2024-02-19T20:33:20.736 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:20.577+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:21.007 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:33:20.977+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64349 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:33:21.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:20.974+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:20.975822+0000) 2024-02-19T20:33:21.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:20.980+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:21.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:20.929+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:21.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:20.974+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:20.975377+0000) 2024-02-19T20:33:21.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:20.747+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:20.748016+0000) 2024-02-19T20:33:21.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:20.974+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:20.975301+0000) 2024-02-19T20:33:21.645 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:21.427+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:21.428073+0000) 2024-02-19T20:33:21.645 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:21.599+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:21.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:21.656+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:21.657244+0000) 2024-02-19T20:33:21.925 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:21.747+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:21.748189+0000) 2024-02-19T20:33:22.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:21.937+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:22.185 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:21.935+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:22.471 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:22.429+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:22.429999+0000) 2024-02-19T20:33:22.738 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:22.658+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:22.659363+0000) 2024-02-19T20:33:22.738 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:22.559+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:23.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:22.749+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:22.750227+0000) 2024-02-19T20:33:23.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:22.904+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:23.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:22.944+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:23.740 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:23.431+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:23.432158+0000) 2024-02-19T20:33:23.741 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:23.661+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:23.661964+0000) 2024-02-19T20:33:23.741 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:23.576+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:24.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:23.751+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:23.752448+0000) 2024-02-19T20:33:24.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:23.923+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:24.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:23.896+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:24.652 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:24.434+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:24.435719+0000) 2024-02-19T20:33:24.653 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:24.610+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:24.913 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:24.904+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:24.913 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:24.754+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:24.755078+0000) 2024-02-19T20:33:24.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:24.663+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:24.664436+0000) 2024-02-19T20:33:24.914 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:24.924+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:25.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:25.436+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:25.437611+0000) 2024-02-19T20:33:25.863 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:25.665+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:25.666398+0000) 2024-02-19T20:33:25.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:25.604+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:25.864 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:25.755+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:25.756857+0000) 2024-02-19T20:33:26.158 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:33:25.985+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64390 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:33:26.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:25.981+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:25.982897+0000) 2024-02-19T20:33:26.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:25.874+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:26.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:25.982+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:25.983067+0000) 2024-02-19T20:33:26.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:25.902+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:26.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:25.981+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:25.982144+0000) 2024-02-19T20:33:26.745 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:26.436+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:26.437791+0000) 2024-02-19T20:33:26.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:26.665+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:26.666622+0000) 2024-02-19T20:33:26.745 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:26.557+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:27.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:26.857+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:27.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:26.949+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:27.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:26.756+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:26.757015+0000) 2024-02-19T20:33:27.747 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:27.667+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:27.668696+0000) 2024-02-19T20:33:27.748 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:27.541+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:27.748 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:27.439+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:27.440251+0000) 2024-02-19T20:33:28.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:27.850+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:28.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:27.924+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:28.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:27.758+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:27.759496+0000) 2024-02-19T20:33:28.750 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:28.442+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:28.443745+0000) 2024-02-19T20:33:28.750 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:28.669+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:28.670904+0000) 2024-02-19T20:33:28.751 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:28.566+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:28.751 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:28.566+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:28.751 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:28.566+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:28.751 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:28.567+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:29.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:28.823+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:29.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:28.881+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:29.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:28.761+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:28.762041+0000) 2024-02-19T20:33:29.661 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:29.444+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:29.445824+0000) 2024-02-19T20:33:29.661 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:29.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:29.661 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:29.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:29.661 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:29.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:29.661 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:29.520+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:30.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:29.774+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:30.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:29.763+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:29.764381+0000) 2024-02-19T20:33:30.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:29.672+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:29.673018+0000) 2024-02-19T20:33:30.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:29.912+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:30.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:30.447+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:30.448101+0000) 2024-02-19T20:33:30.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:30.674+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:30.675266+0000) 2024-02-19T20:33:30.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:30.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:30.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:30.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:30.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:30.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:30.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:30.547+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:31.007 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:33:30.992+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64430 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:33:31.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:30.765+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:30.766631+0000) 2024-02-19T20:33:31.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:30.989+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:30.990561+0000) 2024-02-19T20:33:31.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:30.803+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:31.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:30.989+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:30.990140+0000) 2024-02-19T20:33:31.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:30.916+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:31.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:30.989+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:30.990759+0000) 2024-02-19T20:33:31.663 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:31.447+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:31.448298+0000) 2024-02-19T20:33:31.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:31.590+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:31.664 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:31.590+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:31.664 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:31.590+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:31.664 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:31.591+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:32.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:31.782+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:32.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:31.674+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:31.675434+0000) 2024-02-19T20:33:32.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:31.903+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:32.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:31.765+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:31.766847+0000) 2024-02-19T20:33:32.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:32.449+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:32.450316+0000) 2024-02-19T20:33:32.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:32.676+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:32.677707+0000) 2024-02-19T20:33:32.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:32.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:32.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:32.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:32.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:32.553+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:32.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:32.554+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:33.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:32.804+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:33.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:32.767+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:32.768898+0000) 2024-02-19T20:33:33.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:32.900+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:33.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:33.678+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:33.679761+0000) 2024-02-19T20:33:33.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:33.451+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:33.452388+0000) 2024-02-19T20:33:33.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:33.600+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:33.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:33.600+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:33.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:33.600+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:33.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:33.600+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:34.154 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:33.779+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:34.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:33.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:34.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:33.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:34.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:33.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:34.155 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:33.925+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:34.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:33.770+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:33.770999+0000) 2024-02-19T20:33:34.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:34.681+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:34.682274+0000) 2024-02-19T20:33:34.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:34.453+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:34.454458+0000) 2024-02-19T20:33:34.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:34.584+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:34.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:34.584+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:34.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:34.584+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:34.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:34.584+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:35.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:34.772+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:34.773107+0000) 2024-02-19T20:33:35.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:34.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:35.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:34.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:35.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:34.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:35.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:34.921+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:35.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:34.774+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:35.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:35.454+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:35.456389+0000) 2024-02-19T20:33:35.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:35.682+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:35.684716+0000) 2024-02-19T20:33:35.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:35.619+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:35.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:35.619+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:35.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:35.619+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:35.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:35.619+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:36.110 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:33:35.999+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64471 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:33:36.110 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:35.773+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:35.775130+0000) 2024-02-19T20:33:36.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:35.996+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:35.997719+0000) 2024-02-19T20:33:36.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:35.812+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:36.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:35.996+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:35.997391+0000) 2024-02-19T20:33:36.111 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:35.963+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:36.112 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:35.963+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:36.112 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:35.963+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:36.112 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:35.964+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:36.112 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:35.997+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:35.998943+0000) 2024-02-19T20:33:36.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:36.455+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:36.456563+0000) 2024-02-19T20:33:36.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:36.683+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:36.684941+0000) 2024-02-19T20:33:36.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:36.626+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:36.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:36.626+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:36.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:36.626+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:36.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:36.627+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:37.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:36.838+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:37.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:36.774+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:36.775318+0000) 2024-02-19T20:33:37.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:36.943+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:37.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:36.943+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:37.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:36.943+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:37.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:36.943+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:37.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:37.457+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:37.458600+0000) 2024-02-19T20:33:37.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:37.687+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:37.688363+0000) 2024-02-19T20:33:37.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:37.607+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:37.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:37.607+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:37.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:37.607+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:37.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:37.607+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:38.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:37.822+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:38.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:37.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:38.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:37.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:38.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:37.935+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:38.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:37.936+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:38.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:37.776+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:37.777359+0000) 2024-02-19T20:33:38.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:38.689+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:38.690569+0000) 2024-02-19T20:33:38.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:38.461+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:38.462137+0000) 2024-02-19T20:33:38.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:38.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:38.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:38.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:38.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:38.573+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:38.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:38.574+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:39.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:38.800+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:39.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:38.914+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:39.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:38.914+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:39.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:38.914+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:39.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:38.915+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:39.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:38.779+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:38.780075+0000) 2024-02-19T20:33:39.680 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:39.463+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:39.464164+0000) 2024-02-19T20:33:39.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:39.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:39.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:39.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:39.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:39.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:39.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:39.536+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:39.949 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:39.782+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:39.950 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:39.781+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:39.782828+0000) 2024-02-19T20:33:39.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:39.692+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:39.692996+0000) 2024-02-19T20:33:39.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:39.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:39.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:39.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:39.950 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:39.960+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:40.248 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:39.961+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:40.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:40.465+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:40.466190+0000) 2024-02-19T20:33:40.903 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:40.695+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:40.695975+0000) 2024-02-19T20:33:40.903 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:40.566+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:40.903 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:40.566+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:40.903 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:40.566+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:40.904 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:40.567+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:40.904 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:40.784+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:40.785914+0000) 2024-02-19T20:33:40.904 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:40.794+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:40.904 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:40.794+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:40.904 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:40.794+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:40.904 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:40.795+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:41.160 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:33:41.007+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64511 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:33:41.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:41.004+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:41.005452+0000) 2024-02-19T20:33:41.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:41.004+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:41.005498+0000) 2024-02-19T20:33:41.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:40.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:41.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:40.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:41.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:40.916+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:41.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:40.916+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:41.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:41.004+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:41.005117+0000) 2024-02-19T20:33:41.684 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:41.465+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:41.466405+0000) 2024-02-19T20:33:41.684 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:41.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:41.684 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:41.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:41.684 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:41.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:41.684 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:41.548+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:42.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:41.695+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:41.696190+0000) 2024-02-19T20:33:42.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:41.873+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:42.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:41.873+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:42.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:41.873+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:42.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:41.873+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:42.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:41.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:42.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:41.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:42.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:41.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:42.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:41.807+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:42.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:41.785+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:41.786110+0000) 2024-02-19T20:33:42.686 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:42.468+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:42.469901+0000) 2024-02-19T20:33:42.687 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:42.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:42.687 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:42.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:42.687 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:42.568+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:42.687 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:42.569+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:43.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:42.788+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:42.789685+0000) 2024-02-19T20:33:43.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:42.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:43.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:42.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:43.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:42.806+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:43.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:42.807+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:43.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:42.697+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:42.698728+0000) 2024-02-19T20:33:43.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:42.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:43.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:42.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:43.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:42.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:43.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:42.877+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:43.688 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:43.471+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:43.472461+0000) 2024-02-19T20:33:43.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:43.608+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:43.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:43.608+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:43.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:43.608+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:43.689 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:43.609+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:43.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:43.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:43.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:43.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:43.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:43.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:43.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:43.793+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:43.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:43.700+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:43.701025+0000) 2024-02-19T20:33:43.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:43.922+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:43.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:43.922+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:43.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:43.922+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:43.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:43.923+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:43.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:43.790+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:43.791839+0000) 2024-02-19T20:33:44.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:44.473+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:44.474462+0000) 2024-02-19T20:33:44.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:44.702+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:44.703429+0000) 2024-02-19T20:33:44.922 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:44.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:44.922 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:44.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:44.923 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:44.816+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:44.923 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:44.817+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:44.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:44.630+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:44.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:44.630+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:44.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:44.630+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:44.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:44.631+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:44.923 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:44.793+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:44.794434+0000) 2024-02-19T20:33:45.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:44.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:45.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:44.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:45.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:44.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:45.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:44.934+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:45.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:45.705+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:45.706483+0000) 2024-02-19T20:33:45.758 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:45.475+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:45.476761+0000) 2024-02-19T20:33:45.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:45.624+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:45.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:45.624+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:45.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:45.624+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:45.758 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:45.624+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:46.121 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:33:46.015+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64552 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:33:46.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:45.963+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:46.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:45.963+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:46.122 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:45.963+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:46.122 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:45.964+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:46.122 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:46.011+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:46.012165+0000) 2024-02-19T20:33:46.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:45.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:46.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:45.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:46.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:45.862+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:46.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:45.863+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:46.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:46.011+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:46.012249+0000) 2024-02-19T20:33:46.123 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:45.796+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:45.797036+0000) 2024-02-19T20:33:46.123 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:46.012+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:46.013187+0000) 2024-02-19T20:33:46.694 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:46.475+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:46.476971+0000) 2024-02-19T20:33:46.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:46.610+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:46.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:46.610+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:46.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:46.610+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:46.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:46.611+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:47.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:46.796+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:46.797203+0000) 2024-02-19T20:33:47.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:46.705+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:46.706709+0000) 2024-02-19T20:33:47.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:46.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:47.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:46.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:47.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:46.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:47.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:46.920+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:47.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:46.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:47.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:46.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:47.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:46.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:47.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:46.835+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:47.698 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:47.478+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:47.479080+0000) 2024-02-19T20:33:47.698 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:47.577+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:47.698 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:47.577+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:47.698 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:47.577+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:47.699 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:47.578+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:48.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:47.798+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:47.799147+0000) 2024-02-19T20:33:48.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:47.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:48.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:47.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:48.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:47.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:48.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:47.855+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:48.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:47.709+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:47.710176+0000) 2024-02-19T20:33:48.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:47.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:48.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:47.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:48.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:47.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:48.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:47.946+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:48.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:48.481+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:48.482317+0000) 2024-02-19T20:33:48.701 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:48.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:48.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:48.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:48.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:48.562+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:48.702 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:48.563+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:49.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:48.800+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:48.801312+0000) 2024-02-19T20:33:49.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:48.812+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:49.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:48.812+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:49.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:48.812+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:49.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:48.812+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:49.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:48.712+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:48.713481+0000) 2024-02-19T20:33:49.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:48.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:49.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:48.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:49.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:48.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:49.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:48.953+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:49.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:49.484+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:49.485625+0000) 2024-02-19T20:33:49.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:49.557+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:49.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:49.557+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:49.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:49.557+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:49.704 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:49.557+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:50.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:49.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:50.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:49.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:50.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:49.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:50.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:49.814+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:50.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:49.715+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:49.716068+0000) 2024-02-19T20:33:50.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:49.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:50.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:49.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:50.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:49.924+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:50.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:49.924+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:50.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:49.802+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:49.803876+0000) 2024-02-19T20:33:50.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:50.486+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:50.487922+0000) 2024-02-19T20:33:50.706 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:50.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:50.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:50.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:50.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:50.531+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:50.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:50.531+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:51.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:50.717+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:50.718447+0000) 2024-02-19T20:33:51.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:50.950+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:51.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:50.950+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:51.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:50.950+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:51.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:50.950+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:51.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:50.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:51.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:50.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:51.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:50.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:51.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:50.849+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:51.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:50.805+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:50.806216+0000) 2024-02-19T20:33:51.257 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:33:51.023+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64592 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:33:51.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:51.019+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:51.020764+0000) 2024-02-19T20:33:51.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:51.019+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:51.020353+0000) 2024-02-19T20:33:51.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:51.019+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:51.020242+0000) 2024-02-19T20:33:51.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:51.487+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:51.488111+0000) 2024-02-19T20:33:51.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:51.518+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:51.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:51.518+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:51.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:51.518+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:51.707 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:51.518+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:52.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:51.805+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:51.806378+0000) 2024-02-19T20:33:52.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:51.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:52.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:51.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:52.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:51.896+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:52.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:51.897+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:52.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:51.717+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:51.718665+0000) 2024-02-19T20:33:52.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:51.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:52.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:51.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:52.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:51.961+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:52.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:51.962+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:52.709 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:52.490+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:52.491643+0000) 2024-02-19T20:33:52.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:52.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:52.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:52.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:52.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:52.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:52.709 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:52.474+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:52.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:52.720+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:52.721174+0000) 2024-02-19T20:33:52.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:52.808+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:52.809125+0000) 2024-02-19T20:33:52.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:52.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:52.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:52.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:52.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:52.848+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:52.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:52.848+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:53.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:52.993+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:53.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:52.993+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:53.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:52.993+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:53.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:52.994+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:53.568 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:53.493+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:53.494050+0000) 2024-02-19T20:33:53.568 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:53.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:53.569 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:53.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:53.569 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:53.459+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:53.569 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:53.459+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:53.843 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:53.810+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:53.811113+0000) 2024-02-19T20:33:53.844 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:53.722+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:53.723717+0000) 2024-02-19T20:33:54.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:54.015+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:54.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:54.015+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:54.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:54.015+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:54.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:54.015+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:54.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:53.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:54.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:53.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:54.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:53.855+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:54.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:53.855+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:54.713 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:54.495+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:54.496149+0000) 2024-02-19T20:33:54.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:54.463+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:54.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:54.464+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:54.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:54.464+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:54.714 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:54.464+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:54.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:54.724+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:54.725800+0000) 2024-02-19T20:33:54.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:54.812+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:54.813323+0000) 2024-02-19T20:33:54.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:54.843+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:54.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:54.843+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:54.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:54.844+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:54.968 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:54.845+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:55.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:54.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:55.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:54.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:55.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:54.978+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:55.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:54.979+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:55.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:55.497+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:55.498449+0000) 2024-02-19T20:33:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:55.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:55.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:55.418+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:55.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:55.418+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:55.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:55.814+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:55.815212+0000) 2024-02-19T20:33:55.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:55.726+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:55.727953+0000) 2024-02-19T20:33:55.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:55.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:55.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:55.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:55.972 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:55.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:55.972 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:55.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:55.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:55.870+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:55.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:55.871+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:56.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:56.027+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:56.028733+0000) 2024-02-19T20:33:56.257 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:33:56.030+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64633 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:33:56.258 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:56.027+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:56.028918+0000) 2024-02-19T20:33:56.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:55.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:56.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:55.983+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:56.258 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:56.028+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:56.029346+0000) 2024-02-19T20:33:56.715 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:56.497+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:56.498734+0000) 2024-02-19T20:33:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:56.386+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:56.386+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:56.386+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:56.716 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:56.387+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:57.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:56.727+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:56.728164+0000) 2024-02-19T20:33:57.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:56.814+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:56.815444+0000) 2024-02-19T20:33:57.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:56.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:57.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:56.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:57.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:56.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:57.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:56.850+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:57.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:57.023+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:57.267 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:57.023+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:57.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:57.023+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:57.268 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:57.024+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:57.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:57.501+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:57.502230+0000) 2024-02-19T20:33:57.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:57.389+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:57.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:57.389+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:57.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:57.389+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:57.719 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:57.390+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:57.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:57.816+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:57.817787+0000) 2024-02-19T20:33:57.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:57.807+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:57.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:57.807+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:57.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:57.807+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:57.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:57.808+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:57.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:57.730+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:57.731067+0000) 2024-02-19T20:33:57.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:57.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:57.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:57.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:57.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:57.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:58.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:57.991+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:58.722 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:58.503+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:58.504408+0000) 2024-02-19T20:33:58.722 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:58.392+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:58.722 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:58.392+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:58.722 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:58.392+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:58.723 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:58.392+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:59.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:58.733+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:58.734445+0000) 2024-02-19T20:33:59.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:58.820+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:58.821203+0000) 2024-02-19T20:33:59.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:58.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:59.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:58.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:59.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:58.849+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:33:59.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:58.850+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:33:59.269 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:59.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:59.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:59.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:59.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:59.021+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:33:59.270 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:59.023+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:33:59.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:59.505+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:59.506473+0000) 2024-02-19T20:33:59.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:59.384+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:59.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:59.384+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:59.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:59.384+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:33:59.725 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:59.385+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:00.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:33:59.735+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:59.736776+0000) 2024-02-19T20:34:00.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:33:59.822+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:59.823223+0000) 2024-02-19T20:34:00.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:59.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:00.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:59.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:00.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:59.808+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:00.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:33:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:33:59.808+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:00.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:00.015+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:00.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:00.015+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:00.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:00.015+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:00.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:00.016+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:00.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:00.508+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:00.509525+0000) 2024-02-19T20:34:00.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:00.335+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:00.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:00.335+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:00.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:00.335+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:00.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:00.336+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:01.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:00.824+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:00.825508+0000) 2024-02-19T20:34:01.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:00.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:01.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:00.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:01.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:00.787+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:01.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:00.787+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:01.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:00.738+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:00.739285+0000) 2024-02-19T20:34:01.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:00.974+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:01.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:00.974+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:01.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:00.974+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:01.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:00.974+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:01.314 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:34:01.039+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64673 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:34:01.315 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:01.035+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:01.036421+0000) 2024-02-19T20:34:01.315 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:01.035+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:01.036416+0000) 2024-02-19T20:34:01.315 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:01.034+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:01.035688+0000) 2024-02-19T20:34:01.315 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:01.325+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:01.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:01.325+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:01.316 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:01.325+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:01.727 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:01.508+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:01.509754+0000) 2024-02-19T20:34:01.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:01.326+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:02.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:01.824+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:01.825727+0000) 2024-02-19T20:34:02.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:01.738+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:01.739484+0000) 2024-02-19T20:34:02.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:01.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:02.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:01.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:02.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:01.979+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:02.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:01.980+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:02.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:01.763+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:02.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:01.763+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:02.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:01.763+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:02.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:01.764+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:02.432 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:02.325+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:02.432 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:02.325+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:02.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:02.325+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:02.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:02.326+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:02.722 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:02.510+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:02.511673+0000) 2024-02-19T20:34:03.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:02.740+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:02.741479+0000) 2024-02-19T20:34:03.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:02.828+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:02.829281+0000) 2024-02-19T20:34:03.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:02.733+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:03.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:02.733+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:03.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:02.733+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:03.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:02.734+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:03.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:03.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:03.022+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:03.271 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:03.023+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:03.569 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:03.512+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:03.513844+0000) 2024-02-19T20:34:03.570 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:03.282+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:03.570 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:03.282+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:03.570 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:03.282+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:03.570 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:03.283+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:03.820 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:03.743+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:03.744768+0000) 2024-02-19T20:34:03.820 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:03.775+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:03.821 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:03.775+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:03.821 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:03.775+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:03.821 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:03.776+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:04.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:03.831+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:03.832617+0000) 2024-02-19T20:34:04.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:03.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:04.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:03.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:04.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:03.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:04.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:03.983+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:04.432 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:04.313+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:04.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:04.313+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:04.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:04.313+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:04.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:04.314+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:04.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:04.514+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:04.515815+0000) 2024-02-19T20:34:05.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:04.834+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:04.835874+0000) 2024-02-19T20:34:05.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:04.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:05.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:04.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:05.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:04.780+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:05.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:04.781+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:05.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:04.747+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:04.748089+0000) 2024-02-19T20:34:05.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:04.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:05.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:04.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:05.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:04.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:05.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:04.955+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:05.432 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:05.333+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:05.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:05.333+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:05.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:05.333+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:05.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:05.333+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:05.738 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:05.518+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:05.519466+0000) 2024-02-19T20:34:06.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:05.838+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:05.839028+0000) 2024-02-19T20:34:06.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:05.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:06.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:05.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:06.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:05.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:06.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:05.786+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:06.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:05.750+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:05.751111+0000) 2024-02-19T20:34:06.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:05.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:06.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:05.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:06.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:05.990+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:06.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:05.990+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:06.291 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:34:06.047+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64714 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:34:06.292 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:06.044+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:06.045724+0000) 2024-02-19T20:34:06.292 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:06.043+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:06.044756+0000) 2024-02-19T20:34:06.292 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:06.044+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:06.045036+0000) 2024-02-19T20:34:06.739 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:06.518+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:06.519655+0000) 2024-02-19T20:34:06.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:06.302+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:06.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:06.302+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:06.739 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:06.302+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:06.740 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:06.303+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:07.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:06.838+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:06.839218+0000) 2024-02-19T20:34:07.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:06.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:07.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:06.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:07.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:06.785+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:07.009 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:06.786+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:07.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:06.750+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:06.751300+0000) 2024-02-19T20:34:07.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:06.997+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:07.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:06.997+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:07.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:06.997+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:07.010 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:06.998+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:07.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:07.254+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:07.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:07.254+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:07.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:07.254+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:07.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:07.255+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:07.742 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:07.522+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:07.523181+0000) 2024-02-19T20:34:08.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:07.840+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:07.841159+0000) 2024-02-19T20:34:08.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:07.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:08.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:07.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:08.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:07.792+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:08.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:07.793+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:08.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:07.753+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:07.754661+0000) 2024-02-19T20:34:08.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:07.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:08.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:07.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:08.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:07.988+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:08.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:07.988+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:08.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:08.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:08.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:08.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:08.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:08.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:08.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:08.209+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:08.745 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:08.524+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:08.525899+0000) 2024-02-19T20:34:09.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:08.842+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:08.843292+0000) 2024-02-19T20:34:09.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:08.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:09.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:08.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:09.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:08.813+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:09.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:08.813+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:09.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:08.757+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:08.758116+0000) 2024-02-19T20:34:09.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:08.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:09.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:08.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:09.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:08.952+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:09.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:08.953+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:09.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:09.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:09.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:09.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:09.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:09.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:09.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:09.182+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:09.748 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:09.528+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:09.529460+0000) 2024-02-19T20:34:10.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:09.844+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:09.845374+0000) 2024-02-19T20:34:10.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:09.801+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:10.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:09.801+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:10.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:09.801+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:10.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:09.802+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:10.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:09.759+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:09.760761+0000) 2024-02-19T20:34:10.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:09.987+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:10.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:09.987+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:10.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:09.987+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:10.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:09.987+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:10.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:10.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:10.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:10.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:10.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:10.180+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:10.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:10.181+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:10.750 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:10.531+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:10.532820+0000) 2024-02-19T20:34:11.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:10.846+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:10.847395+0000) 2024-02-19T20:34:11.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:10.814+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:11.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:10.814+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:11.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:10.814+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:11.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:10.815+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:11.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:10.761+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:10.762955+0000) 2024-02-19T20:34:11.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:10.963+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:11.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:10.963+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:11.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:10.963+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:11.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:10.964+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:11.355 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:34:11.055+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64754 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:34:11.356 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:11.052+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:11.053837+0000) 2024-02-19T20:34:11.356 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:11.052+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:11.053318+0000) 2024-02-19T20:34:11.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:11.054+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:11.055536+0000) 2024-02-19T20:34:11.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:11.187+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:11.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:11.187+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:11.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:11.187+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:11.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:11.188+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:11.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:11.532+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:11.533016+0000) 2024-02-19T20:34:11.934 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:11.846+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:11.847591+0000) 2024-02-19T20:34:11.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:11.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:11.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:11.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:11.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:11.804+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:11.935 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:11.805+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:11.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:11.762+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:11.763158+0000) 2024-02-19T20:34:11.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:11.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:11.935 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:11.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:11.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:11.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:12.190 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:11.947+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:12.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:12.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:12.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:12.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:12.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:12.184+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:12.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:12.185+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:12.797 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:12.764+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:12.765058+0000) 2024-02-19T20:34:12.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:12.534+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:12.535435+0000) 2024-02-19T20:34:12.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:12.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:12.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:12.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:12.798 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:12.809+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:13.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:12.810+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:13.127 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:12.848+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:12.849622+0000) 2024-02-19T20:34:13.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:12.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:13.128 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:12.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:13.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:12.954+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:13.131 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:12.955+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:13.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:13.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:13.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:13.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:13.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:13.138+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:13.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:13.139+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:13.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:13.536+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:13.537467+0000) 2024-02-19T20:34:14.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:13.851+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:13.852122+0000) 2024-02-19T20:34:14.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:13.823+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:14.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:13.823+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:14.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:13.823+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:14.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:13.824+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:14.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:13.766+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:13.767085+0000) 2024-02-19T20:34:14.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:13.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:14.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:13.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:14.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:13.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:14.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:13.939+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:14.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:14.167+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:14.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:14.167+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:14.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:14.167+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:14.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:14.168+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:14.694 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:14.538+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:14.539475+0000) 2024-02-19T20:34:15.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:14.853+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:14.854533+0000) 2024-02-19T20:34:15.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:14.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:15.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:14.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:15.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:14.819+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:15.008 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:14.820+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:15.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:14.768+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:14.769681+0000) 2024-02-19T20:34:15.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:14.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:15.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:14.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:15.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:14.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:15.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:14.905+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:15.433 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:15.178+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:15.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:15.178+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:15.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:15.178+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:15.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:15.178+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:15.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:15.541+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:15.542801+0000) 2024-02-19T20:34:16.045 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:15.855+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:15.856872+0000) 2024-02-19T20:34:16.045 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:15.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:16.046 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:15.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:16.046 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:15.861+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:16.046 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:15.862+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:16.046 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:15.770+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:15.771776+0000) 2024-02-19T20:34:16.046 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:15.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:16.046 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:15.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:16.047 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:15.945+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:16.047 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:15.946+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:16.356 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:34:16.063+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64795 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:34:16.356 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:16.059+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:16.060929+0000) 2024-02-19T20:34:16.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:16.059+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:16.060498+0000) 2024-02-19T20:34:16.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:16.059+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:16.060651+0000) 2024-02-19T20:34:16.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:16.183+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:16.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:16.183+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:16.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:16.183+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:16.358 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:16.184+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:16.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:16.542+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:16.543002+0000) 2024-02-19T20:34:17.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:16.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:17.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:16.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:17.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:16.904+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:17.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:16.905+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:17.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:16.771+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:16.772072+0000) 2024-02-19T20:34:17.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:16.958+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:17.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:16.958+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:17.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:16.958+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:17.141 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:16.959+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:17.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:16.856+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:16.857070+0000) 2024-02-19T20:34:17.142 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:17.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:17.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:17.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:17.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:17.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:17.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:17.152+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:17.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:17.544+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:17.545354+0000) 2024-02-19T20:34:18.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:17.859+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:17.860008+0000) 2024-02-19T20:34:18.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:17.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:18.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:17.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:18.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:17.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:18.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:17.878+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:18.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:17.773+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:17.774161+0000) 2024-02-19T20:34:18.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:17.940+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:18.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:17.940+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:18.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:17.940+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:18.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:17.941+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:18.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:18.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:18.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:18.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:18.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:18.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:18.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:18.193+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:18.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:18.546+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:18.547717+0000) 2024-02-19T20:34:19.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:18.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:19.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:18.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:19.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:18.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:19.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:18.896+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:19.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:18.775+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:18.776561+0000) 2024-02-19T20:34:19.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:18.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:19.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:18.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:19.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:18.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:19.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:18.969+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:19.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:18.861+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:18.862588+0000) 2024-02-19T20:34:19.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:19.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:19.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:19.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:19.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:19.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:19.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:19.209+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:19.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:19.549+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:19.550114+0000) 2024-02-19T20:34:20.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:19.871+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:20.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:19.871+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:20.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:19.871+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:20.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:19.872+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:20.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:19.777+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:19.778653+0000) 2024-02-19T20:34:20.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:20.012+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:20.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:20.012+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:20.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:20.012+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:20.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:20.013+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:20.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:19.864+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:19.865155+0000) 2024-02-19T20:34:20.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:20.225+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:20.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:20.225+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:20.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:20.225+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:20.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:20.226+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:20.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:20.551+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:20.552527+0000) 2024-02-19T20:34:21.053 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:20.866+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:20.867168+0000) 2024-02-19T20:34:21.054 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:20.913+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:21.054 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:20.913+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:21.054 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:20.913+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:21.054 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:20.914+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:21.054 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:20.780+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:20.781118+0000) 2024-02-19T20:34:21.054 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:20.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:21.055 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:20.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:21.055 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:20.998+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:21.055 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:20.998+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:21.356 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:34:21.072+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64835 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:34:21.357 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:21.068+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:21.069888+0000) 2024-02-19T20:34:21.357 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:21.068+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:21.069421+0000) 2024-02-19T20:34:21.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:21.069+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:21.070449+0000) 2024-02-19T20:34:21.357 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:21.256+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:21.358 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:21.256+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:21.358 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:21.256+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:21.358 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:21.256+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:21.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:21.551+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:21.552749+0000) 2024-02-19T20:34:22.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:21.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:22.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:21.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:22.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:21.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:22.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:21.911+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:22.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:21.780+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:21.781316+0000) 2024-02-19T20:34:22.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:21.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:22.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:21.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:22.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:21.965+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:22.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:21.965+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:22.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:21.866+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:21.867368+0000) 2024-02-19T20:34:22.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:22.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:22.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:22.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:22.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:22.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:22.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:22.240+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:22.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:22.554+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:22.554659+0000) 2024-02-19T20:34:23.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:22.868+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:22.870498+0000) 2024-02-19T20:34:23.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:22.782+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:22.783788+0000) 2024-02-19T20:34:23.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:22.976+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:23.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:22.976+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:23.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:22.976+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:23.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:22.976+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:23.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:22.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:23.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:22.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:23.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:22.914+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:23.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:22.915+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:23.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:23.232+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:23.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:23.232+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:23.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:23.232+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:23.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:23.232+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:23.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:23.555+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:23.557126+0000) 2024-02-19T20:34:24.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:23.871+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:23.872937+0000) 2024-02-19T20:34:24.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:23.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:24.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:23.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:24.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:23.945+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:24.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:23.946+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:24.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:23.785+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:23.786302+0000) 2024-02-19T20:34:24.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:23.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:24.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:23.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:24.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:23.991+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:24.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:23.991+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:24.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:24.256+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:24.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:24.256+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:24.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:24.256+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:24.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:24.256+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:24.710 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:24.558+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:24.559211+0000) 2024-02-19T20:34:25.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:24.787+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:24.788541+0000) 2024-02-19T20:34:25.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:25.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:25.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:24.874+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:24.875014+0000) 2024-02-19T20:34:25.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:24.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:25.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:24.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:25.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:24.911+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:25.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:24.912+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:25.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:25.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:25.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:25.016+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:25.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:25.017+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:25.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:25.247+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:25.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:25.247+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:25.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:25.247+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:25.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:25.247+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:25.864 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:25.560+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:25.561450+0000) 2024-02-19T20:34:25.864 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:25.790+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:25.791079+0000) 2024-02-19T20:34:26.164 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:34:26.080+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64876 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:34:26.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:25.876+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:25.877019+0000) 2024-02-19T20:34:26.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:26.076+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:26.077656+0000) 2024-02-19T20:34:26.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:25.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:26.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:25.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:26.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:25.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:26.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:25.933+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:26.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:26.076+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:26.077274+0000) 2024-02-19T20:34:26.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:26.023+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:26.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:26.023+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:26.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:26.023+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:26.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:26.023+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:26.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:26.076+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:26.077120+0000) 2024-02-19T20:34:26.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:26.252+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:26.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:26.252+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:26.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:26.252+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:26.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:26.252+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:26.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:26.560+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:26.561621+0000) 2024-02-19T20:34:27.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:26.876+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:26.877219+0000) 2024-02-19T20:34:27.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:26.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:27.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:26.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:27.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:26.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:27.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:26.937+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:27.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:26.790+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:26.791250+0000) 2024-02-19T20:34:27.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:27.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:27.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:27.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:27.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:27.038+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:27.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:27.039+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:27.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:27.229+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:27.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:27.229+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:27.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:27.229+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:27.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:27.230+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:27.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:27.563+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:27.564143+0000) 2024-02-19T20:34:28.037 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:27.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:28.037 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:27.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:28.037 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:27.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:28.037 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:27.931+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:28.038 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:27.793+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:27.794770+0000) 2024-02-19T20:34:28.038 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:28.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:28.038 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:28.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:28.038 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:28.048+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:28.038 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:27.878+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:27.879785+0000) 2024-02-19T20:34:28.302 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:28.049+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:28.302 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:28.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:28.302 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:28.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:28.303 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:28.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:28.303 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:28.183+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:28.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:28.796+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:28.797207+0000) 2024-02-19T20:34:28.870 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:28.565+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:28.566620+0000) 2024-02-19T20:34:29.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:28.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:29.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:28.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:29.165 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:28.944+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:29.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:28.945+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:29.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:29.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:29.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:29.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:29.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:29.084+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:29.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:29.085+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:29.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:28.881+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:28.882369+0000) 2024-02-19T20:34:29.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:29.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:29.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:29.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:29.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:29.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:29.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:29.182+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:29.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:29.567+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:29.568808+0000) 2024-02-19T20:34:30.042 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:29.798+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:29.799441+0000) 2024-02-19T20:34:30.043 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:29.884+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:29.884974+0000) 2024-02-19T20:34:30.043 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:29.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:30.043 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:29.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:30.044 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:29.939+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:30.044 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:29.939+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:30.302 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:30.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:30.303 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:30.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:30.303 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:30.054+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:30.303 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:30.055+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:30.303 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:30.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:30.303 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:30.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:30.303 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:30.182+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:30.304 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:30.182+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:30.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:30.801+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:30.802634+0000) 2024-02-19T20:34:30.875 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:30.570+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:30.571108+0000) 2024-02-19T20:34:31.166 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:34:31.088+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64916 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:34:31.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:30.886+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:30.887679+0000) 2024-02-19T20:34:31.166 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:31.086+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:31.087482+0000) 2024-02-19T20:34:31.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:31.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:31.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:31.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:31.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:31.006+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:31.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:31.007+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:31.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:31.083+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:31.084709+0000) 2024-02-19T20:34:31.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:30.954+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:31.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:30.954+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:31.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:30.954+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:31.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:30.955+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:31.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:31.084+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:31.085347+0000) 2024-02-19T20:34:31.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:31.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:31.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:31.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:31.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:31.215+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:31.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:31.216+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:31.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:31.570+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:31.571331+0000) 2024-02-19T20:34:32.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:31.886+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:31.887894+0000) 2024-02-19T20:34:32.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:31.996+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:32.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:31.996+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:32.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:31.996+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:32.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:31.997+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:32.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:31.801+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:31.802842+0000) 2024-02-19T20:34:32.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:31.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:32.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:31.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:32.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:31.986+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:32.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:31.987+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:32.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:32.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:32.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:32.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:32.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:32.194+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:32.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:32.194+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:32.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:32.573+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:32.574875+0000) 2024-02-19T20:34:33.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:32.804+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:32.805302+0000) 2024-02-19T20:34:33.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:33.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:33.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:33.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:33.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:33.026+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:33.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:33.027+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:33.127 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:32.889+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:32.890376+0000) 2024-02-19T20:34:33.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:32.953+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:33.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:32.953+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:33.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:32.953+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:33.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:32.954+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:33.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:33.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:33.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:33.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:33.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:33.174+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:33.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:33.174+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:33.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:33.575+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:33.576905+0000) 2024-02-19T20:34:34.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:33.891+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:33.892953+0000) 2024-02-19T20:34:34.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:33.807+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:33.808667+0000) 2024-02-19T20:34:34.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:33.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:34.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:33.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:34.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:33.994+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:34.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:33.995+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:34.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:33.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:34.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:33.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:34.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:33.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:34.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:33.937+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:34.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:34.217+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:34.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:34.217+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:34.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:34.217+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:34.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:34.217+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:34.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:34.578+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:34.579428+0000) 2024-02-19T20:34:35.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:34.895+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:34.896589+0000) 2024-02-19T20:34:35.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:34.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:35.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:34.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:35.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:34.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:35.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:34.970+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:35.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:34.811+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:34.812025+0000) 2024-02-19T20:34:35.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:34.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:35.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:34.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:35.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:34.999+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:35.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:35.000+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:35.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:35.263+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:35.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:35.263+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:35.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:35.263+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:35.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:35.263+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:35.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:35.581+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:35.582065+0000) 2024-02-19T20:34:36.077 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:35.813+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:35.814694+0000) 2024-02-19T20:34:36.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:35.972+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:36.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:35.972+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:36.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:35.972+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:36.078 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:35.972+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:36.078 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:35.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:36.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:35.927+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:36.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:35.927+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:36.079 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:35.928+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:36.079 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:35.897+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:35.898550+0000) 2024-02-19T20:34:36.357 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:34:36.096+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64957 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:34:36.358 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:36.091+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:36.092855+0000) 2024-02-19T20:34:36.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:36.091+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:36.092689+0000) 2024-02-19T20:34:36.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:36.092+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:36.093580+0000) 2024-02-19T20:34:36.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:36.251+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:36.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:36.251+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:36.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:36.251+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:36.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:36.251+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:36.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:36.581+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:36.582278+0000) 2024-02-19T20:34:37.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:36.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:37.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:36.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:37.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:36.922+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:37.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:36.923+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:37.166 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:36.813+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:36.814899+0000) 2024-02-19T20:34:37.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:36.997+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:37.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:36.997+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:37.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:36.997+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:37.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:36.998+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:37.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:36.897+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:36.898722+0000) 2024-02-19T20:34:37.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:37.287+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:37.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:37.287+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:37.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:37.287+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:37.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:37.288+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:37.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:37.583+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:37.584402+0000) 2024-02-19T20:34:38.165 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:37.901+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:37.901972+0000) 2024-02-19T20:34:38.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:37.884+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:38.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:37.884+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:38.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:37.884+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:38.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:37.885+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:38.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:37.817+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:37.818288+0000) 2024-02-19T20:34:38.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:38.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:38.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:38.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:38.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:38.013+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:38.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:38.014+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:38.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:38.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:38.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:38.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:38.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:38.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:38.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:38.312+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:38.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:38.586+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:38.587346+0000) 2024-02-19T20:34:39.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:38.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:39.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:38.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:39.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:38.841+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:39.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:38.842+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:39.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:38.820+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:38.821486+0000) 2024-02-19T20:34:39.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:39.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:39.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:39.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:39.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:39.043+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:39.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:39.044+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:39.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:38.904+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:38.905388+0000) 2024-02-19T20:34:39.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:39.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:39.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:39.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:39.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:39.324+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:39.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:39.324+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:39.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:39.589+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:39.590682+0000) 2024-02-19T20:34:40.070 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:39.822+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:39.823557+0000) 2024-02-19T20:34:40.070 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:39.907+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:39.908817+0000) 2024-02-19T20:34:40.070 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:39.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:40.071 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:39.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:40.071 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:39.831+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:40.071 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:39.832+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:40.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:40.282+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:40.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:40.282+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:40.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:40.282+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:40.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:40.282+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:40.360 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:40.081+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:40.360 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:40.081+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:40.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:40.081+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:40.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:40.082+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:40.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:40.593+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:40.593997+0000) 2024-02-19T20:34:41.085 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:40.824+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:40.825659+0000) 2024-02-19T20:34:41.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:40.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:41.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:40.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:41.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:40.833+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:41.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:40.834+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:41.086 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:40.911+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:40.912112+0000) 2024-02-19T20:34:41.358 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:34:41.104+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 64997 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:34:41.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:41.100+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:41.101067+0000) 2024-02-19T20:34:41.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:41.100+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:41.101555+0000) 2024-02-19T20:34:41.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:41.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:41.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:41.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:41.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:41.113+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:41.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:41.114+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:41.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:41.101+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:41.102889+0000) 2024-02-19T20:34:41.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:41.260+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:41.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:41.260+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:41.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:41.260+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:41.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:41.261+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:41.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:41.593+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:41.594188+0000) 2024-02-19T20:34:42.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:41.824+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:41.825863+0000) 2024-02-19T20:34:42.124 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:41.911+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:41.912303+0000) 2024-02-19T20:34:42.124 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:41.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:42.124 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:41.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:42.125 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:41.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:42.125 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:41.882+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:42.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:42.259+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:42.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:42.259+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:42.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:42.259+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:42.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:42.260+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:42.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:42.135+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:42.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:42.135+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:42.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:42.135+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:42.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:42.136+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:42.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:42.595+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:42.596311+0000) 2024-02-19T20:34:43.116 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:42.827+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:42.828311+0000) 2024-02-19T20:34:43.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:42.914+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:42.915553+0000) 2024-02-19T20:34:43.117 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:42.927+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:43.117 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:42.927+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:43.117 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:42.927+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:43.117 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:42.928+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:43.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:43.264+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:43.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:43.264+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:43.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:43.264+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:43.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:43.264+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:43.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:43.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:43.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:43.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:43.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:43.128+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:43.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:43.128+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:43.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:43.598+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:43.599049+0000) 2024-02-19T20:34:44.144 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:43.917+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:43.918882+0000) 2024-02-19T20:34:44.145 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:43.830+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:43.831802+0000) 2024-02-19T20:34:44.145 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:44.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:44.145 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:44.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:44.145 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:44.156+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:44.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:43.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:44.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:43.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:44.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:43.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:44.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:43.880+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:44.436 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:44.157+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:44.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:44.305+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:44.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:44.305+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:44.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:44.305+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:44.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:44.306+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:44.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:44.600+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:44.601218+0000) 2024-02-19T20:34:45.099 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:44.833+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:44.834557+0000) 2024-02-19T20:34:45.100 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:44.920+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:44.921104+0000) 2024-02-19T20:34:45.100 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:44.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:45.100 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:44.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:45.100 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:44.835+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:45.101 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:44.836+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:45.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:45.111+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:45.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:45.111+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:45.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:45.111+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:45.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:45.112+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:45.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:45.287+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:45.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:45.287+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:45.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:45.287+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:45.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:45.288+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:45.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:45.603+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:45.604702+0000) 2024-02-19T20:34:46.086 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:45.835+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:45.836898+0000) 2024-02-19T20:34:46.087 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:45.922+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:45.923836+0000) 2024-02-19T20:34:46.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:45.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:46.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:45.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:46.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:45.825+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:46.087 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:45.826+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:46.358 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:34:46.112+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65038 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:34:46.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:46.109+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:46.110251+0000) 2024-02-19T20:34:46.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:46.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:46.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:46.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:46.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:46.098+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:46.360 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:46.099+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:46.360 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:46.108+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:46.109548+0000) 2024-02-19T20:34:46.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:46.109+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:46.110519+0000) 2024-02-19T20:34:46.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:46.256+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:46.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:46.256+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:46.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:46.256+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:46.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:46.256+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:46.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:46.603+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:46.604913+0000) 2024-02-19T20:34:47.137 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:46.923+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:46.924067+0000) 2024-02-19T20:34:47.137 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:46.836+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:46.837128+0000) 2024-02-19T20:34:47.137 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:47.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:47.138 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:47.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:47.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:46.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:47.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:46.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:47.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:46.793+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:47.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:46.793+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:47.436 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:47.148+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:47.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:47.150+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:47.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:47.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:47.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:47.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:47.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:47.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:47.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:47.296+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:47.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:47.607+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:47.608441+0000) 2024-02-19T20:34:48.151 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:47.838+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:47.839225+0000) 2024-02-19T20:34:48.151 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:47.925+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:47.926147+0000) 2024-02-19T20:34:48.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:47.843+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:48.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:47.843+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:48.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:47.843+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:48.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:47.844+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:48.436 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:48.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:48.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:48.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:48.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:48.162+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:48.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:48.164+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:48.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:48.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:48.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:48.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:48.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:48.312+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:48.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:48.312+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:48.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:48.610+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:48.611092+0000) 2024-02-19T20:34:49.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:48.841+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:48.842738+0000) 2024-02-19T20:34:49.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:48.927+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:48.928146+0000) 2024-02-19T20:34:49.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:48.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:49.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:48.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:49.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:48.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:49.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:48.882+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:49.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:49.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:49.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:49.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:49.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:49.187+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:49.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:49.188+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:49.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:49.309+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:49.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:49.309+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:49.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:49.309+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:49.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:49.310+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:49.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:49.613+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:49.614715+0000) 2024-02-19T20:34:50.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:49.844+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:49.845248+0000) 2024-02-19T20:34:50.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:49.929+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:49.930181+0000) 2024-02-19T20:34:50.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:49.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:50.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:49.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:50.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:49.930+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:50.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:49.931+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:50.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:50.229+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:50.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:50.229+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:50.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:50.229+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:50.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:50.230+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:50.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:50.356+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:50.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:50.356+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:50.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:50.356+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:50.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:50.356+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:50.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:50.617+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:50.618216+0000) 2024-02-19T20:34:51.102 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:50.847+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:50.848716+0000) 2024-02-19T20:34:51.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:50.890+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:51.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:50.890+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:51.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:50.890+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:51.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:50.890+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:51.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:50.931+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:50.932511+0000) 2024-02-19T20:34:51.360 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:34:51.121+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65078 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:34:51.360 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:51.118+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:51.119047+0000) 2024-02-19T20:34:51.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:51.116+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:51.117705+0000) 2024-02-19T20:34:51.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:51.214+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:51.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:51.214+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:51.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:51.214+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:51.361 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:51.214+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:51.361 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:51.116+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:51.117477+0000) 2024-02-19T20:34:51.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:51.309+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:51.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:51.309+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:51.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:51.309+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:51.362 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:51.309+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:51.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:51.617+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:51.618417+0000) 2024-02-19T20:34:52.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:51.847+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:51.848908+0000) 2024-02-19T20:34:52.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:51.931+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:51.932734+0000) 2024-02-19T20:34:52.167 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:51.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:52.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:51.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:52.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:51.898+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:52.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:51.899+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:52.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:52.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:52.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:52.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:52.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:52.228+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:52.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:52.229+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:52.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:52.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:52.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:52.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:52.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:52.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:52.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:52.297+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:52.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:52.619+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:52.620400+0000) 2024-02-19T20:34:53.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:52.851+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:52.852326+0000) 2024-02-19T20:34:53.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:52.933+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:52.934928+0000) 2024-02-19T20:34:53.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:52.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:53.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:52.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:53.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:52.854+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:53.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:52.855+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:53.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:53.250+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:53.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:53.250+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:53.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:53.250+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:53.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:53.251+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:53.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:53.329+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:53.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:53.329+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:53.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:53.329+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:53.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:53.330+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:53.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:53.622+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:53.623023+0000) 2024-02-19T20:34:54.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:53.854+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:53.855718+0000) 2024-02-19T20:34:54.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:53.936+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:53.937003+0000) 2024-02-19T20:34:54.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:53.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:54.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:53.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:54.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:53.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:54.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:53.860+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:54.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:54.291+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:54.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:54.291+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:54.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:54.291+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:54.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:54.291+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:54.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:54.237+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:54.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:54.237+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:54.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:54.237+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:54.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:54.238+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:54.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:54.624+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:54.625368+0000) 2024-02-19T20:34:55.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:54.856+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:54.857800+0000) 2024-02-19T20:34:55.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:54.938+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:54.939455+0000) 2024-02-19T20:34:55.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:54.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:55.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:54.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:55.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:54.900+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:55.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:54.901+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:55.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:55.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:55.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:55.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:55.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:55.275+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:55.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:55.276+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:55.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:55.315+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:55.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:55.315+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:55.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:55.315+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:55.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:55.315+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:55.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:55.626+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:55.627305+0000) 2024-02-19T20:34:56.111 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:55.859+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:55.860236+0000) 2024-02-19T20:34:56.111 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:55.942+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:55.943019+0000) 2024-02-19T20:34:56.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:55.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:56.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:55.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:56.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:55.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:56.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:55.852+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:56.437 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:34:56.129+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65119 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:34:56.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:56.127+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:56.128438+0000) 2024-02-19T20:34:56.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:56.127+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:56.127984+0000) 2024-02-19T20:34:56.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:56.325+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:56.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:56.325+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:56.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:56.325+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:56.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:56.326+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:56.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:56.125+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:56.126474+0000) 2024-02-19T20:34:56.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:56.301+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:56.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:56.301+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:56.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:56.301+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:56.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:56.301+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:56.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:56.626+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:56.627511+0000) 2024-02-19T20:34:57.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:56.859+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:56.860411+0000) 2024-02-19T20:34:57.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:56.942+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:56.943204+0000) 2024-02-19T20:34:57.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:56.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:57.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:56.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:57.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:56.852+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:57.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:56.853+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:57.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:57.340+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:57.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:57.340+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:57.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:57.340+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:57.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:57.341+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:57.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:57.278+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:57.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:57.278+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:57.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:57.278+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:57.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:57.280+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:57.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:57.630+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:57.631028+0000) 2024-02-19T20:34:58.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:57.861+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:57.862407+0000) 2024-02-19T20:34:58.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:57.944+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:57.945324+0000) 2024-02-19T20:34:58.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:57.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:58.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:57.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:58.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:57.826+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:58.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:57.827+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:58.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:58.301+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:58.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:58.301+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:58.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:58.301+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:58.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:58.302+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:58.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:58.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:58.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:58.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:58.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:58.288+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:58.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:58.289+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:58.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:58.632+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:58.633181+0000) 2024-02-19T20:34:59.167 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:58.863+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:58.864343+0000) 2024-02-19T20:34:59.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:58.946+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:58.947564+0000) 2024-02-19T20:34:59.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:58.857+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:59.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:58.857+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:59.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:58.857+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:34:59.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:58.857+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:59.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:59.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:59.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:59.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:59.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:59.309+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:34:59.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:59.309+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:34:59.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:59.331+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:59.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:59.331+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:59.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:59.331+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:34:59.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:59.331+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:34:59.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:59.634+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:59.635755+0000) 2024-02-19T20:35:00.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:34:59.865+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:59.866276+0000) 2024-02-19T20:35:00.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:59.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:00.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:59.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:00.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:59.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:00.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:34:59.865+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:00.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:34:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:34:59.948+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:59.949588+0000) 2024-02-19T20:35:00.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:00.276+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:00.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:00.276+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:00.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:00.276+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:00.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:00.276+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:00.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:00.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:00.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:00.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:00.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:00.296+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:00.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:00.296+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:00.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:00.637+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:00.638243+0000) 2024-02-19T20:35:01.118 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:00.867+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:00.868645+0000) 2024-02-19T20:35:01.119 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:00.952+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:00.953164+0000) 2024-02-19T20:35:01.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:00.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:01.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:00.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:01.119 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:00.878+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:01.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:00.879+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:01.438 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:35:01.137+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65159 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:35:01.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:01.133+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:01.134502+0000) 2024-02-19T20:35:01.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:01.134+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:01.135298+0000) 2024-02-19T20:35:01.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:01.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:01.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:01.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:01.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:01.251+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:01.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:01.252+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:01.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:01.133+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:01.134919+0000) 2024-02-19T20:35:01.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:01.310+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:01.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:01.310+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:01.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:01.310+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:01.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:01.311+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:01.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:01.637+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:01.638424+0000) 2024-02-19T20:35:02.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:01.867+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:01.868846+0000) 2024-02-19T20:35:02.168 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:01.952+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:01.953364+0000) 2024-02-19T20:35:02.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:01.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:02.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:01.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:02.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:01.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:02.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:01.927+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:02.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:02.351+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:02.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:02.351+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:02.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:02.351+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:02.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:02.352+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:02.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:02.286+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:02.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:02.286+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:02.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:02.286+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:02.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:02.287+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:02.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:02.639+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:02.640672+0000) 2024-02-19T20:35:03.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:02.870+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:02.871411+0000) 2024-02-19T20:35:03.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:02.954+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:02.955431+0000) 2024-02-19T20:35:03.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:02.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:03.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:02.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:03.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:02.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:03.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:02.888+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:03.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:03.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:03.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:03.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:03.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:03.326+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:03.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:03.326+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:03.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:03.395+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:03.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:03.395+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:03.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:03.395+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:03.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:03.396+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:03.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:03.641+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:03.642804+0000) 2024-02-19T20:35:04.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:03.872+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:03.873960+0000) 2024-02-19T20:35:04.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:03.956+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:03.957543+0000) 2024-02-19T20:35:04.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:03.897+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:04.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:03.897+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:04.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:03.897+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:04.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:03.897+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:04.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:04.344+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:04.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:04.344+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:04.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:04.344+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:04.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:04.345+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:04.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:04.391+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:04.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:04.391+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:04.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:04.391+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:04.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:04.391+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:04.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:04.644+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:04.645045+0000) 2024-02-19T20:35:05.127 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:04.875+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:04.876412+0000) 2024-02-19T20:35:05.128 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:04.958+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:04.959860+0000) 2024-02-19T20:35:05.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:04.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:05.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:04.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:05.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:04.895+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:05.129 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:04.896+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:05.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:05.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:05.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:05.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:05.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:05.332+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:05.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:05.333+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:05.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:05.646+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:05.647059+0000) 2024-02-19T20:35:05.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:05.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:05.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:05.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:05.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:05.401+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:05.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:05.401+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:06.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:05.878+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:05.879835+0000) 2024-02-19T20:35:06.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:05.890+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:06.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:05.890+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:06.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:05.890+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:06.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:05.891+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:06.127 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:05.961+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:05.962128+0000) 2024-02-19T20:35:06.438 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:35:06.145+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65200 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:35:06.439 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:06.143+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:06.144277+0000) 2024-02-19T20:35:06.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:06.143+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:06.144067+0000) 2024-02-19T20:35:06.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:06.378+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:06.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:06.378+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:06.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:06.378+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:06.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:06.379+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:06.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:06.142+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:06.143317+0000) 2024-02-19T20:35:06.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:06.358+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:06.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:06.358+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:06.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:06.358+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:06.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:06.358+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:06.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:06.646+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:06.647232+0000) 2024-02-19T20:35:07.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:06.879+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:06.880023+0000) 2024-02-19T20:35:07.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:06.961+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:06.962308+0000) 2024-02-19T20:35:07.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:06.872+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:07.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:06.872+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:07.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:06.872+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:07.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:06.872+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:07.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:07.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:07.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:07.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:07.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:07.367+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:07.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:07.368+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:07.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:07.335+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:07.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:07.335+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:07.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:07.335+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:07.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:07.335+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:07.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:07.648+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:07.649423+0000) 2024-02-19T20:35:08.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:07.881+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:07.882601+0000) 2024-02-19T20:35:08.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:07.963+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:07.964844+0000) 2024-02-19T20:35:08.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:07.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:08.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:07.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:08.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:07.918+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:08.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:07.919+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:08.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:08.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:08.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:08.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:08.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:08.382+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:08.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:08.382+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:08.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:08.321+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:08.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:08.321+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:08.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:08.321+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:08.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:08.321+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:08.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:08.650+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:08.651863+0000) 2024-02-19T20:35:09.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:08.883+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:08.884800+0000) 2024-02-19T20:35:09.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:08.965+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:08.966830+0000) 2024-02-19T20:35:09.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:08.953+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:09.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:08.953+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:09.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:08.953+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:09.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:08.954+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:09.419 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:09.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:09.420 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:09.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:09.423 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:09.284+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:09.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:09.284+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:09.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:09.654+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:09.655365+0000) 2024-02-19T20:35:09.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:09.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:09.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:09.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:09.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:09.432+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:09.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:09.432+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:10.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:09.886+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:09.887308+0000) 2024-02-19T20:35:10.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:09.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:10.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:09.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:10.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:09.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:10.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:09.935+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:10.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:09.969+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:09.970149+0000) 2024-02-19T20:35:10.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:10.401+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:10.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:10.401+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:10.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:10.401+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:10.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:10.402+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:10.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:10.251+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:10.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:10.251+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:10.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:10.251+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:10.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:10.251+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:10.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:10.656+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:10.657858+0000) 2024-02-19T20:35:11.135 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:10.888+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:10.889849+0000) 2024-02-19T20:35:11.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:10.972+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:10.973435+0000) 2024-02-19T20:35:11.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:10.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:11.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:10.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:11.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:10.972+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:11.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:10.973+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:11.397 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:35:11.153+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65240 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:35:11.398 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:11.149+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:11.151483+0000) 2024-02-19T20:35:11.398 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:11.149+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:11.151253+0000) 2024-02-19T20:35:11.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:11.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:11.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:11.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:11.399 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:11.408+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:11.399 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:11.149+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:11.151040+0000) 2024-02-19T20:35:11.399 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:11.221+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:11.399 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:11.221+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:11.399 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:11.221+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:11.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:11.222+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:11.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:11.657+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:11.658091+0000) 2024-02-19T20:35:11.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:11.409+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:12.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:11.889+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:11.890103+0000) 2024-02-19T20:35:12.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:11.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:12.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:11.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:12.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:11.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:12.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:11.935+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:12.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:11.972+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:11.973660+0000) 2024-02-19T20:35:12.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:12.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:12.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:12.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:12.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:12.414+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:12.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:12.416+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:12.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:12.183+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:12.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:12.183+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:12.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:12.183+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:12.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:12.184+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:12.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:12.659+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:12.660786+0000) 2024-02-19T20:35:13.138 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:12.891+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:12.892123+0000) 2024-02-19T20:35:13.138 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:12.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:13.139 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:12.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:13.139 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:12.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:13.139 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:12.939+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:13.139 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:12.975+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:12.976089+0000) 2024-02-19T20:35:13.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:13.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:13.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:13.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:13.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:13.150+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:13.438 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:13.151+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:13.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:13.662+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:13.663481+0000) 2024-02-19T20:35:13.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:13.450+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:13.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:13.450+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:13.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:13.450+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:13.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:13.451+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:14.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:13.893+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:13.894275+0000) 2024-02-19T20:35:14.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:13.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:14.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:13.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:14.142 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:13.903+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:14.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:13.904+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:14.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:13.978+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:13.979658+0000) 2024-02-19T20:35:14.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:14.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:14.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:14.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:14.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:14.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:14.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:14.154+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:14.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:14.666+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:14.667088+0000) 2024-02-19T20:35:14.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:14.476+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:14.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:14.476+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:14.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:14.476+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:14.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:14.477+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:15.130 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:14.896+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:14.897094+0000) 2024-02-19T20:35:15.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:14.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:15.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:14.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:15.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:14.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:15.131 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:14.885+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:15.132 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:14.982+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:14.983265+0000) 2024-02-19T20:35:15.132 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:15.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:15.132 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:15.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:15.132 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:15.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:15.432 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:15.143+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:15.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:15.668+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:15.669344+0000) 2024-02-19T20:35:15.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:15.444+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:15.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:15.444+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:15.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:15.444+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:15.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:15.445+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:16.134 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:15.898+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:15.899793+0000) 2024-02-19T20:35:16.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:15.984+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:15.985329+0000) 2024-02-19T20:35:16.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:15.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:16.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:15.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:16.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:15.860+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:16.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:15.861+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:16.399 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:35:16.162+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65281 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:35:16.400 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:16.158+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:16.159871+0000) 2024-02-19T20:35:16.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:16.159+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:16.160289+0000) 2024-02-19T20:35:16.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:16.157+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:16.158973+0000) 2024-02-19T20:35:16.400 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:16.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:16.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:16.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:16.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:16.158+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:16.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:16.158+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:16.656 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:16.411+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:16.656 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:16.411+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:16.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:16.411+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:16.657 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:16.412+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:16.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:16.899+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:16.900100+0000) 2024-02-19T20:35:16.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:16.668+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:16.669608+0000) 2024-02-19T20:35:16.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:16.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:16.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:16.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:16.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:16.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:16.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:16.893+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:17.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:16.984+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:16.985568+0000) 2024-02-19T20:35:17.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:17.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:17.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:17.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:17.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:17.181+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:17.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:17.181+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:17.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:17.418+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:17.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:17.418+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:17.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:17.418+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:17.659 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:17.418+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:17.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:17.901+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:17.902772+0000) 2024-02-19T20:35:17.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:17.671+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:17.672065+0000) 2024-02-19T20:35:17.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:17.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:17.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:17.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:17.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:17.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:17.975 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:17.886+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:18.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:17.987+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:17.988078+0000) 2024-02-19T20:35:18.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:18.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:18.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:18.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:18.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:18.161+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:18.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:18.162+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:18.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:18.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:18.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:18.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:18.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:18.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:18.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:18.461+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:18.912 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:18.904+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:18.905162+0000) 2024-02-19T20:35:18.913 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:18.673+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:18.674346+0000) 2024-02-19T20:35:18.913 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:18.923+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:18.913 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:18.923+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:18.913 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:18.923+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:18.913 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:18.924+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:19.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:18.989+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:18.990682+0000) 2024-02-19T20:35:19.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:19.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:19.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:19.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:19.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:19.113+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:19.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:19.114+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:19.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:19.675+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:19.676517+0000) 2024-02-19T20:35:19.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:19.508+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:19.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:19.508+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:19.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:19.508+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:19.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:19.508+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:20.170 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:19.906+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:19.907575+0000) 2024-02-19T20:35:20.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:19.966+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:20.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:19.966+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:20.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:19.966+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:20.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:19.967+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:20.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:19.992+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:19.993215+0000) 2024-02-19T20:35:20.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:20.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:20.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:20.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:20.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:20.131+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:20.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:20.132+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:20.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:20.678+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:20.679023+0000) 2024-02-19T20:35:20.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:20.471+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:20.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:20.471+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:20.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:20.471+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:20.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:20.472+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:21.151 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:20.908+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:20.909725+0000) 2024-02-19T20:35:21.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:20.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:21.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:20.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:21.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:20.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:21.152 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:20.996+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:21.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:20.994+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:20.995897+0000) 2024-02-19T20:35:21.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:21.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:21.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:21.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:21.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:21.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:21.153 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:21.154+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:21.439 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:35:21.171+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65321 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:35:21.440 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:21.167+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:21.168396+0000) 2024-02-19T20:35:21.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:21.167+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:21.168060+0000) 2024-02-19T20:35:21.441 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:21.167+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:21.168380+0000) 2024-02-19T20:35:21.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:21.678+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:21.679220+0000) 2024-02-19T20:35:21.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:21.472+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:21.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:21.472+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:21.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:21.472+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:21.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:21.472+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:22.170 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:21.909+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:21.909991+0000) 2024-02-19T20:35:22.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:21.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:22.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:21.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:22.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:21.962+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:22.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:21.963+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:22.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:21.995+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:21.996160+0000) 2024-02-19T20:35:22.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:22.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:22.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:22.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:22.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:22.142+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:22.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:22.143+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:22.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:22.680+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:22.681196+0000) 2024-02-19T20:35:22.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:22.493+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:22.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:22.493+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:22.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:22.493+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:22.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:22.494+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:23.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:22.912+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:22.913511+0000) 2024-02-19T20:35:23.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:22.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:23.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:22.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:23.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:22.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:23.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:22.985+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:23.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:22.997+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:22.998670+0000) 2024-02-19T20:35:23.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:23.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:23.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:23.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:23.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:23.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:23.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:23.130+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:23.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:23.683+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:23.684662+0000) 2024-02-19T20:35:23.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:23.468+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:23.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:23.468+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:23.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:23.468+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:23.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:23.468+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:24.170 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:23.914+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:23.915584+0000) 2024-02-19T20:35:24.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:23.999+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:24.000846+0000) 2024-02-19T20:35:24.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:24.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:24.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:24.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:24.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:24.112+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:24.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:24.112+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:24.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:24.017+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:24.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:24.017+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:24.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:24.017+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:24.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:24.018+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:24.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:24.685+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:24.686636+0000) 2024-02-19T20:35:24.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:24.448+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:24.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:24.448+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:24.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:24.448+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:24.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:24.449+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:25.170 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:24.916+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:24.917644+0000) 2024-02-19T20:35:25.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:25.041+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:25.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:25.041+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:25.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:25.041+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:25.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:25.042+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:25.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:25.001+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:25.002903+0000) 2024-02-19T20:35:25.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:25.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:25.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:25.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:25.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:25.085+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:25.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:25.086+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:25.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:25.425+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:25.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:25.425+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:25.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:25.425+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:25.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:25.426+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:25.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:25.689+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:25.690098+0000) 2024-02-19T20:35:26.160 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:25.920+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:25.921169+0000) 2024-02-19T20:35:26.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:26.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:26.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:26.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:26.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:26.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:26.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:26.007+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:26.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:26.004+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:26.005026+0000) 2024-02-19T20:35:26.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:26.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:26.161 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:26.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:26.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:26.066+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:26.162 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:26.066+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:26.440 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:35:26.179+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65362 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:35:26.440 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:26.175+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:26.176823+0000) 2024-02-19T20:35:26.441 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:26.175+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:26.176582+0000) 2024-02-19T20:35:26.441 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:26.175+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:26.176282+0000) 2024-02-19T20:35:26.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:26.689+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:26.690346+0000) 2024-02-19T20:35:26.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:26.466+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:26.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:26.466+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:26.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:26.466+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:26.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:26.467+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:27.170 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:26.920+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:26.921472+0000) 2024-02-19T20:35:27.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:26.967+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:27.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:26.967+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:27.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:26.967+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:27.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:26.968+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:27.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:27.004+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:27.005300+0000) 2024-02-19T20:35:27.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:27.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:27.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:27.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:27.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:27.074+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:27.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:27.075+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:27.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:27.691+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:27.692779+0000) 2024-02-19T20:35:27.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:27.464+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:27.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:27.464+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:27.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:27.464+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:27.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:27.465+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:28.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:27.922+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:27.923463+0000) 2024-02-19T20:35:28.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:27.967+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:28.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:27.967+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:28.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:27.967+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:28.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:27.968+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:28.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:28.007+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:28.008614+0000) 2024-02-19T20:35:28.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:28.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:28.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:28.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:28.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:28.065+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:28.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:28.065+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:28.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:28.420+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:28.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:28.420+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:28.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:28.420+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:28.441 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:28.421+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:28.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:28.693+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:28.694896+0000) 2024-02-19T20:35:29.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:28.924+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:28.925475+0000) 2024-02-19T20:35:29.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:28.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:29.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:28.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:29.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:28.984+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:29.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:28.985+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:29.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:29.010+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:29.011028+0000) 2024-02-19T20:35:29.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:29.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:29.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:29.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:29.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:29.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:29.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:29.031+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:29.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:29.427+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:29.440 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:29.427+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:29.441 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:29.427+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:29.441 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:29.427+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:29.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:29.697+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:29.698203+0000) 2024-02-19T20:35:30.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:29.926+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:29.927590+0000) 2024-02-19T20:35:30.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:30.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:30.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:30.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:30.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:30.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:30.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:30.020+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:30.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:30.012+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:30.013072+0000) 2024-02-19T20:35:30.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:30.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:30.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:30.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:30.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:30.019+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:30.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:30.020+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:30.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:30.700+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:30.701758+0000) 2024-02-19T20:35:30.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:30.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:30.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:30.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:30.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:30.460+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:30.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:30.461+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:31.168 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:30.928+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:30.929635+0000) 2024-02-19T20:35:31.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:30.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:31.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:30.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:31.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:30.995+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:31.169 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:30.996+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:31.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:31.014+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:31.015601+0000) 2024-02-19T20:35:31.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:31.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:31.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:31.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:31.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:31.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:31.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:31.053+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:31.435 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:35:31.187+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65402 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:35:31.436 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:31.183+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:31.184183+0000) 2024-02-19T20:35:31.436 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:31.183+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:31.184896+0000) 2024-02-19T20:35:31.436 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:31.183+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:31.184308+0000) 2024-02-19T20:35:31.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:31.447+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:31.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:31.447+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:31.437 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:31.447+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:31.688 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:31.448+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:32.002 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:31.928+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:31.929856+0000) 2024-02-19T20:35:32.003 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:31.700+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:31.701968+0000) 2024-02-19T20:35:32.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:32.014+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:32.015889+0000) 2024-02-19T20:35:32.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:32.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:32.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:32.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:32.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:32.052+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:32.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:32.052+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:32.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:32.017+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:32.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:32.017+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:32.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:32.017+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:32.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:32.018+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:32.692 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:32.479+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:32.692 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:32.479+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:32.692 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:32.479+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:32.692 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:32.479+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:33.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:32.704+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:32.705389+0000) 2024-02-19T20:35:33.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:32.930+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:32.931880+0000) 2024-02-19T20:35:33.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:33.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:33.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:33.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:33.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:33.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:33.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:33.057+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:33.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:33.017+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:33.018671+0000) 2024-02-19T20:35:33.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:33.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:33.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:33.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:33.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:33.079+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:33.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:33.079+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:33.694 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:33.527+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:33.694 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:33.527+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:33.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:33.527+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:33.695 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:33.527+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:34.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:33.706+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:33.707677+0000) 2024-02-19T20:35:34.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:33.933+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:33.934139+0000) 2024-02-19T20:35:34.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:34.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:34.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:34.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:34.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:34.056+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:34.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:34.056+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:34.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:34.021+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:34.022146+0000) 2024-02-19T20:35:34.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:34.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:34.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:34.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:34.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:34.030+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:34.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:34.030+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:34.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:34.490+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:34.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:34.490+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:34.697 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:34.490+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:34.697 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:34.490+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:35.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:34.708+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:34.709843+0000) 2024-02-19T20:35:35.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:34.936+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:34.937706+0000) 2024-02-19T20:35:35.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:35.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:35.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:35.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:35.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:35.028+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:35.363 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:35.029+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:35.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:35.024+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:35.025757+0000) 2024-02-19T20:35:35.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:35.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:35.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:35.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:35.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:35.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:35.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:35.069+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:35.699 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:35.535+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:35.699 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:35.535+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:35.699 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:35.535+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:35.699 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:35.536+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:35.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:35.938+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:35.939808+0000) 2024-02-19T20:35:35.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:35.711+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:35.712277+0000) 2024-02-19T20:35:35.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:35.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:35.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:35.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:35.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:35.989+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:36.256 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:35:36.196+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65443 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:35:36.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:35.990+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:36.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:36.192+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:36.193487+0000) 2024-02-19T20:35:36.257 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:36.191+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:36.192969+0000) 2024-02-19T20:35:36.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:36.027+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:36.027975+0000) 2024-02-19T20:35:36.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:36.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:36.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:36.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:36.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:36.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:36.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:36.074+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:36.258 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:36.191+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:36.191978+0000) 2024-02-19T20:35:36.926 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:36.711+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:36.712562+0000) 2024-02-19T20:35:36.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:36.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:36.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:36.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:36.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:36.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:36.927 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:36.570+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:37.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:36.939+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:36.940101+0000) 2024-02-19T20:35:37.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:36.952+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:37.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:36.952+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:37.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:36.952+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:37.201 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:36.953+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:37.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:37.027+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:37.028255+0000) 2024-02-19T20:35:37.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:37.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:37.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:37.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:37.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:37.033+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:37.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:37.033+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:37.894 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:37.714+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:37.715014+0000) 2024-02-19T20:35:37.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:37.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:37.894 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:37.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:37.895 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:37.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:37.895 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:37.611+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:38.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:37.942+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:37.943617+0000) 2024-02-19T20:35:38.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:37.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:38.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:37.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:38.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:37.906+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:38.172 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:37.907+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:38.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:38.030+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:38.031675+0000) 2024-02-19T20:35:38.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:38.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:38.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:38.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:38.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:38.055+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:38.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:38.055+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:38.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:38.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:38.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:38.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:38.704 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:38.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:38.705 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:38.625+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:39.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:38.946+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:38.947156+0000) 2024-02-19T20:35:39.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:38.716+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:38.717755+0000) 2024-02-19T20:35:39.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:38.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:39.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:38.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:39.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:38.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:39.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:38.927+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:39.363 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:39.034+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:39.035139+0000) 2024-02-19T20:35:39.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:39.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:39.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:39.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:39.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:39.073+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:39.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:39.074+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:39.923 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:39.719+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:39.720053+0000) 2024-02-19T20:35:39.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:39.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:39.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:39.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:39.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:39.935+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:39.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:39.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:39.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:39.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:39.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:39.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:39.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:39.644+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:40.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:39.937+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:40.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:39.948+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:39.949308+0000) 2024-02-19T20:35:40.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:40.036+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:40.037210+0000) 2024-02-19T20:35:40.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:40.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:40.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:40.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:40.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:40.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:40.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:40.060+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:40.708 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:40.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:40.708 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:40.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:40.709 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:40.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:40.709 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:40.674+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:41.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:40.950+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:40.951583+0000) 2024-02-19T20:35:41.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:40.721+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:40.722096+0000) 2024-02-19T20:35:41.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:40.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:41.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:40.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:41.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:40.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:41.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:40.911+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:41.309 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:35:41.204+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65483 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:35:41.310 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:41.201+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:41.202027+0000) 2024-02-19T20:35:41.310 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:41.199+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:41.200179+0000) 2024-02-19T20:35:41.310 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:41.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:41.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:41.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:41.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:41.026+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:41.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:41.026+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:41.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:41.038+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:41.039221+0000) 2024-02-19T20:35:41.311 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:41.200+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:41.201125+0000) 2024-02-19T20:35:41.938 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:41.721+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:41.722394+0000) 2024-02-19T20:35:41.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:41.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:41.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:41.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:41.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:41.881+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:41.939 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:41.883+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:41.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:41.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:41.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:41.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:41.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:41.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:41.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:41.661+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:41.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:41.950+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:41.951923+0000) 2024-02-19T20:35:42.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:42.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:42.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:42.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:42.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:42.029+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:42.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:42.030+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:42.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:42.038+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:42.039519+0000) 2024-02-19T20:35:42.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:42.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:42.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:42.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:42.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:42.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:42.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:42.644+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:42.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:42.723+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:42.724606+0000) 2024-02-19T20:35:42.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:42.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:42.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:42.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:42.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:42.879+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:42.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:42.880+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:43.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:42.954+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:42.955282+0000) 2024-02-19T20:35:43.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:43.041+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:43.042115+0000) 2024-02-19T20:35:43.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:43.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:43.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:43.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:43.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:43.062+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:43.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:43.062+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:43.856 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:43.726+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:43.727186+0000) 2024-02-19T20:35:43.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:43.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:43.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:43.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:43.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:43.668+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:43.857 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:43.669+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:44.172 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:43.957+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:43.958156+0000) 2024-02-19T20:35:44.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:43.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:44.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:43.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:44.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:43.910+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:44.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:43.911+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:44.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:44.043+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:44.044247+0000) 2024-02-19T20:35:44.173 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:44.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:44.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:44.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:44.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:44.054+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:44.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:44.055+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:44.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:44.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:44.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:44.716 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:44.641+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:45.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:44.960+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:44.961463+0000) 2024-02-19T20:35:45.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:44.728+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:44.729211+0000) 2024-02-19T20:35:45.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:44.927+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:45.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:44.927+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:45.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:44.927+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:45.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:44.928+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:45.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:45.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:45.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:45.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:45.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:45.041+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:45.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:45.042+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:45.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:45.045+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:45.046281+0000) 2024-02-19T20:35:45.924 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:45.731+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:45.732762+0000) 2024-02-19T20:35:45.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:45.669+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:45.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:45.669+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:45.924 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:45.669+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:45.925 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:45.670+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:46.193 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:45.963+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:45.964785+0000) 2024-02-19T20:35:46.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:45.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:46.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:45.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:46.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:45.936+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:46.194 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:45.937+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:46.194 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:46.047+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:46.048697+0000) 2024-02-19T20:35:46.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:46.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:46.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:46.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:46.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:46.068+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:46.195 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:46.069+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:46.506 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:35:46.213+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65524 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:35:46.506 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:46.208+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:46.209731+0000) 2024-02-19T20:35:46.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:46.210+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:46.211336+0000) 2024-02-19T20:35:46.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:46.208+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:46.209393+0000) 2024-02-19T20:35:46.912 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:46.732+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:46.733097+0000) 2024-02-19T20:35:46.913 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:46.651+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:46.913 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:46.651+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:46.913 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:46.651+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:46.913 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:46.652+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:47.172 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:46.964+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:46.965116+0000) 2024-02-19T20:35:47.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:46.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:47.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:46.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:47.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:46.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:47.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:46.925+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:47.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:47.047+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:47.048955+0000) 2024-02-19T20:35:47.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:47.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:47.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:47.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:47.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:47.104+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:47.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:47.105+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:47.723 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:47.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:47.723 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:47.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:47.723 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:47.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:47.723 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:47.662+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:48.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:47.966+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:47.967095+0000) 2024-02-19T20:35:48.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:47.735+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:47.736448+0000) 2024-02-19T20:35:48.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:47.947+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:48.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:47.947+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:48.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:47.947+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:48.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:47.948+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:48.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:48.050+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:48.051061+0000) 2024-02-19T20:35:48.364 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:48.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:48.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:48.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:48.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:48.081+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:48.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:48.082+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:48.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:48.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:48.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:48.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:48.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:48.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:48.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:48.703+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:48.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:48.738+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:48.739028+0000) 2024-02-19T20:35:48.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:48.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:48.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:48.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:48.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:48.943+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:48.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:48.944+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:49.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:48.968+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:48.969707+0000) 2024-02-19T20:35:49.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:49.052+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:49.053636+0000) 2024-02-19T20:35:49.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:49.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:49.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:49.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:49.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:49.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:49.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:49.127+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:49.958 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:49.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:49.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:49.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:49.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:49.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:49.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:49.692+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:49.959 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:49.741+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:49.742422+0000) 2024-02-19T20:35:49.959 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:49.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:49.959 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:49.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:49.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:49.934+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:49.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:49.935+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:50.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:49.971+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:49.971997+0000) 2024-02-19T20:35:50.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:50.056+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:50.057194+0000) 2024-02-19T20:35:50.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:50.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:50.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:50.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:50.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:50.129+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:50.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:50.130+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:50.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:50.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:50.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:50.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:50.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:50.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:50.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:50.719+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:50.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:50.743+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:50.744914+0000) 2024-02-19T20:35:50.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:50.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:50.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:50.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:50.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:50.892+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:50.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:50.892+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:51.214 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:35:51.221+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65564 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:35:51.214 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:51.216+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:51.217873+0000) 2024-02-19T20:35:51.214 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:50.973+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:50.974611+0000) 2024-02-19T20:35:51.215 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:51.217+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:51.218772+0000) 2024-02-19T20:35:51.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:51.059+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:51.060787+0000) 2024-02-19T20:35:51.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:51.177+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:51.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:51.177+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:51.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:51.177+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:51.215 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:51.178+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:51.216 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:51.217+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:51.218867+0000) 2024-02-19T20:35:51.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:51.707+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:51.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:51.707+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:51.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:51.707+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:51.962 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:51.708+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:51.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:51.744+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:51.745194+0000) 2024-02-19T20:35:51.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:51.867+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:51.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:51.867+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:51.962 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:51.867+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:51.963 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:51.868+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:52.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:51.973+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:51.974927+0000) 2024-02-19T20:35:52.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:52.060+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:52.061106+0000) 2024-02-19T20:35:52.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:52.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:52.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:52.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:52.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:52.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:52.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:52.192+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:52.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:52.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:52.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:52.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:52.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:52.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:52.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:52.672+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:52.964 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:52.746+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:52.747349+0000) 2024-02-19T20:35:52.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:52.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:52.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:52.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:52.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:52.864+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:52.965 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:52.865+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:53.227 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:52.976+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:52.977292+0000) 2024-02-19T20:35:53.227 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:53.063+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:53.064668+0000) 2024-02-19T20:35:53.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:53.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:53.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:53.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:53.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:53.239+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:53.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:53.240+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:53.822 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:53.677+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:53.822 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:53.677+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:53.822 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:53.677+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:53.822 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:53.678+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:53.822 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:53.748+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:53.749647+0000) 2024-02-19T20:35:53.823 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:53.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:53.823 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:53.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:53.823 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:53.834+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:54.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:53.978+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:53.979387+0000) 2024-02-19T20:35:54.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:54.065+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:54.066741+0000) 2024-02-19T20:35:54.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:53.835+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:54.442 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:54.225+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:54.443 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:54.225+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:54.443 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:54.225+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:54.443 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:54.225+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:54.738 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:54.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:54.738 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:54.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:54.738 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:54.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:54.739 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:54.661+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:55.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:54.981+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:54.982334+0000) 2024-02-19T20:35:55.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:54.750+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:54.751858+0000) 2024-02-19T20:35:55.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:54.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:55.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:54.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:55.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:54.856+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:55.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:54.857+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:55.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:55.067+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:55.068857+0000) 2024-02-19T20:35:55.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:55.266+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:55.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:55.266+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:55.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:55.266+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:55.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:55.266+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:55.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:55.753+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:55.754103+0000) 2024-02-19T20:35:55.875 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:55.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:55.875 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:55.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:55.875 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:55.887+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:55.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:55.627+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:55.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:55.627+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:55.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:55.627+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:55.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:55.628+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:56.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:55.888+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:56.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:55.983+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:55.984567+0000) 2024-02-19T20:35:56.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:56.069+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:56.070868+0000) 2024-02-19T20:35:56.442 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:35:56.229+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65605 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:35:56.443 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:56.225+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:56.226878+0000) 2024-02-19T20:35:56.443 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:56.225+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:56.226911+0000) 2024-02-19T20:35:56.444 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:56.226+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:56.227066+0000) 2024-02-19T20:35:56.444 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:56.268+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:56.444 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:56.268+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:56.444 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:56.268+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:56.444 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:56.268+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:56.741 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:56.622+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:56.741 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:56.622+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:56.741 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:56.622+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:56.741 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:56.623+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:57.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:56.984+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:56.984999+0000) 2024-02-19T20:35:57.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:56.753+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:56.754461+0000) 2024-02-19T20:35:57.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:56.913+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:57.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:56.913+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:57.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:56.913+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:57.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:56.914+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:57.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:57.070+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:57.071193+0000) 2024-02-19T20:35:57.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:57.254+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:57.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:57.254+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:57.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:57.254+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:57.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:57.254+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:57.743 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:57.621+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:57.743 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:57.621+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:57.743 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:57.621+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:57.743 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:57.622+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:58.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:57.986+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:57.987018+0000) 2024-02-19T20:35:58.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:57.755+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:57.756428+0000) 2024-02-19T20:35:58.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:57.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:58.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:57.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:58.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:57.928+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:58.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:57.928+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:58.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:58.072+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:58.073149+0000) 2024-02-19T20:35:58.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:58.224+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:58.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:58.224+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:58.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:58.224+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:58.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:58.224+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:58.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:58.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:58.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:58.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:58.745 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:58.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:58.746 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:58.601+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:35:59.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:58.988+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:58.990552+0000) 2024-02-19T20:35:59.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:58.756+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:58.758725+0000) 2024-02-19T20:35:59.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:58.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:59.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:58.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:59.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:58.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:35:59.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:58.942+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:59.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:59.075+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:59.076063+0000) 2024-02-19T20:35:59.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:59.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:59.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:59.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:59.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:59.204+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:35:59.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:35:59.204+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:35:59.748 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:59.582+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:59.749 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:59.582+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:59.749 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:59.582+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:35:59.749 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:59.583+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:00.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:35:59.991+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:59.992880+0000) 2024-02-19T20:36:00.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:59.761+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:59.762066+0000) 2024-02-19T20:36:00.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:59.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:00.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:59.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:00.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:59.970+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:00.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:35:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:35:59.971+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:00.365 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:00.078+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:00.079014+0000) 2024-02-19T20:36:00.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:00.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:00.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:00.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:00.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:00.210+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:00.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:00.211+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:00.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:00.572+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:00.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:00.572+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:00.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:00.572+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:00.693 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:00.573+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:00.952 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:00.764+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:00.765558+0000) 2024-02-19T20:36:01.218 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:00.995+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:00.996454+0000) 2024-02-19T20:36:01.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:00.976+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:01.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:00.976+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:01.218 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:00.976+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:01.219 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:00.977+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:01.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:01.080+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:01.080996+0000) 2024-02-19T20:36:01.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:01.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:01.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:01.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:01.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:01.185+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:01.219 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:01.185+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:01.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:01.233+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:01.234174+0000) 2024-02-19T20:36:01.506 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:36:01.237+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65645 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:36:01.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:01.233+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:01.234914+0000) 2024-02-19T20:36:01.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:01.234+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:01.235112+0000) 2024-02-19T20:36:01.943 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:01.764+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:01.765845+0000) 2024-02-19T20:36:01.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:01.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:01.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:01.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:01.944 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:01.956+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:01.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:01.539+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:01.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:01.539+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:01.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:01.539+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:01.944 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:01.540+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:02.195 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:01.995+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:01.996764+0000) 2024-02-19T20:36:02.196 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:02.080+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:02.081280+0000) 2024-02-19T20:36:02.196 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:01.956+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:02.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:02.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:02.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:02.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:02.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:02.208+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:02.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:02.208+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:02.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:02.768+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:02.769224+0000) 2024-02-19T20:36:02.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:02.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:02.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:02.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:02.946 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:02.958+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:02.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:02.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:02.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:02.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:02.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:02.581+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:02.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:02.581+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:03.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:02.997+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:02.998771+0000) 2024-02-19T20:36:03.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:02.959+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:03.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:03.082+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:03.083276+0000) 2024-02-19T20:36:03.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:03.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:03.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:03.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:03.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:03.168+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:03.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:03.169+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:03.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:03.535+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:03.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:03.535+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:03.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:03.535+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:03.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:03.536+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:04.071 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:04.000+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:04.001235+0000) 2024-02-19T20:36:04.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:03.771+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:03.772576+0000) 2024-02-19T20:36:04.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:03.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:04.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:03.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:04.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:03.975+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:04.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:03.976+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:04.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:04.084+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:04.085358+0000) 2024-02-19T20:36:04.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:04.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:04.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:04.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:04.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:04.139+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:04.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:04.139+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:04.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:04.563+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:04.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:04.563+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:04.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:04.563+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:04.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:04.564+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:05.074 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:05.002+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:05.003609+0000) 2024-02-19T20:36:05.074 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:04.774+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:04.775890+0000) 2024-02-19T20:36:05.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:04.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:05.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:04.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:05.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:04.938+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:05.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:04.938+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:05.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:05.087+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:05.088020+0000) 2024-02-19T20:36:05.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:05.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:05.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:05.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:05.367 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:05.116+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:05.367 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:05.116+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:05.744 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:05.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:05.744 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:05.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:05.744 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:05.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:05.744 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:05.604+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:06.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:06.004+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:06.005610+0000) 2024-02-19T20:36:06.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:05.778+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:05.779325+0000) 2024-02-19T20:36:06.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:05.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:06.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:05.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:06.006 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:05.941+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:06.007 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:05.942+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:06.351 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:36:06.246+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65686 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:36:06.351 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:06.242+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:06.243915+0000) 2024-02-19T20:36:06.352 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:06.241+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:06.242448+0000) 2024-02-19T20:36:06.352 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:06.089+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:06.090404+0000) 2024-02-19T20:36:06.352 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:06.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:06.353 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:06.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:06.353 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:06.103+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:06.353 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:06.104+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:06.353 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:06.241+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:06.242646+0000) 2024-02-19T20:36:06.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:06.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:06.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:06.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:06.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:06.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:06.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:06.611+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:07.058 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:07.005+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:07.005998+0000) 2024-02-19T20:36:07.058 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:06.778+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:06.779734+0000) 2024-02-19T20:36:07.058 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:06.894+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:07.058 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:06.894+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:07.059 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:06.894+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:07.059 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:06.895+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:07.368 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:07.070+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:07.368 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:07.070+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:07.368 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:07.070+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:07.368 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:07.071+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:07.368 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:07.089+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:07.090727+0000) 2024-02-19T20:36:07.953 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:07.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:07.953 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:07.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:07.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:07.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:07.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:07.647+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:07.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:07.780+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:07.781835+0000) 2024-02-19T20:36:07.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:07.859+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:07.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:07.859+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:07.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:07.859+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:07.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:07.860+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:08.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:08.007+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:08.008098+0000) 2024-02-19T20:36:08.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:08.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:08.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:08.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:08.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:08.035+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:08.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:08.036+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:08.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:08.092+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:08.093113+0000) 2024-02-19T20:36:08.953 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:08.683+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:08.953 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:08.683+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:08.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:08.683+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:08.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:08.684+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:08.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:08.783+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:08.784229+0000) 2024-02-19T20:36:08.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:08.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:08.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:08.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:08.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:08.885+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:08.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:08.886+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:09.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:09.009+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:09.010328+0000) 2024-02-19T20:36:09.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:09.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:09.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:09.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:09.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:09.002+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:09.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:09.002+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:09.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:09.094+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:09.095792+0000) 2024-02-19T20:36:09.953 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:09.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:09.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:09.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:09.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:09.686+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:09.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:09.687+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:09.963 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:09.785+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:09.786276+0000) 2024-02-19T20:36:09.964 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:09.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:09.964 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:09.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:09.964 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:09.932+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:09.964 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:09.933+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:10.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:10.011+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:10.012761+0000) 2024-02-19T20:36:10.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:10.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:10.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:10.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:10.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:10.028+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:10.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:10.028+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:10.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:10.098+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:10.099214+0000) 2024-02-19T20:36:10.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:10.788+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:10.789849+0000) 2024-02-19T20:36:10.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:10.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:10.945 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:10.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:10.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:10.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:10.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:10.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:10.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:10.673+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:10.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:10.674+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:11.234 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:11.013+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:11.014849+0000) 2024-02-19T20:36:11.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:10.957+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:11.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:10.959+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:11.235 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:11.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:11.235 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:11.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:11.235 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:11.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:11.235 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:11.012+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:11.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:11.100+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:11.101274+0000) 2024-02-19T20:36:11.506 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:36:11.255+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65726 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:36:11.506 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:11.251+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:11.252226+0000) 2024-02-19T20:36:11.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:11.249+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:11.250919+0000) 2024-02-19T20:36:11.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:11.250+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:11.251437+0000) 2024-02-19T20:36:11.942 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:11.789+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:11.790261+0000) 2024-02-19T20:36:11.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:11.629+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:11.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:11.629+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:11.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:11.629+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:11.942 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:11.630+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:12.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:12.014+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:12.015179+0000) 2024-02-19T20:36:12.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:11.954+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:12.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:11.954+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:12.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:11.954+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:12.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:11.955+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:12.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:11.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:12.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:11.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:12.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:11.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:12.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:11.991+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:12.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:12.100+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:12.101575+0000) 2024-02-19T20:36:12.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:12.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:12.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:12.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:12.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:12.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:12.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:12.594+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:13.091 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:13.017+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:13.018723+0000) 2024-02-19T20:36:13.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:12.791+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:12.792362+0000) 2024-02-19T20:36:13.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:12.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:13.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:12.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:13.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:12.926+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:13.092 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:12.927+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:13.092 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:12.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:13.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:12.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:13.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:12.993+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:13.093 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:12.993+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:13.366 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:13.104+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:13.105021+0000) 2024-02-19T20:36:13.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:13.793+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:13.794557+0000) 2024-02-19T20:36:13.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:13.637+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:13.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:13.637+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:13.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:13.637+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:13.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:13.638+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:14.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:14.020+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:14.021165+0000) 2024-02-19T20:36:14.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:13.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:14.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:13.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:14.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:13.973+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:14.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:13.974+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:14.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:13.974+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:14.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:13.974+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:14.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:13.974+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:14.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:13.975+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:14.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:14.107+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:14.108661+0000) 2024-02-19T20:36:14.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:14.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:14.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:14.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:14.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:14.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:14.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:14.686+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:14.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:14.795+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:14.796565+0000) 2024-02-19T20:36:14.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:14.927+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:14.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:14.927+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:14.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:14.927+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:14.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:14.927+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:15.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:15.023+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:15.024568+0000) 2024-02-19T20:36:15.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:15.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:15.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:15.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:15.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:15.022+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:15.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:15.022+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:15.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:15.109+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:15.110862+0000) 2024-02-19T20:36:15.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:15.799+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:15.800020+0000) 2024-02-19T20:36:15.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:15.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:15.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:15.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:15.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:15.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:15.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:15.692+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:16.243 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:16.027+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:16.027997+0000) 2024-02-19T20:36:16.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:15.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:16.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:15.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:16.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:15.948+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:16.245 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:15.949+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:16.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:16.011+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:16.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:16.011+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:16.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:16.011+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:16.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:16.012+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:16.245 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:16.112+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:16.113448+0000) 2024-02-19T20:36:16.505 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:36:16.263+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65767 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:36:16.506 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:16.259+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:16.260293+0000) 2024-02-19T20:36:16.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:16.259+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:16.260655+0000) 2024-02-19T20:36:16.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:16.261+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:16.262027+0000) 2024-02-19T20:36:16.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:16.799+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:16.800281+0000) 2024-02-19T20:36:16.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:16.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:16.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:16.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:16.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:16.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:16.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:16.645+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:17.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:17.027+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:17.028450+0000) 2024-02-19T20:36:17.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:17.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:17.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:17.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:17.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:17.060+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:17.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:17.060+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:17.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:17.112+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:17.113799+0000) 2024-02-19T20:36:17.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:16.999+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:17.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:16.999+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:17.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:16.999+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:17.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:17.000+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:17.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:17.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:17.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:17.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:17.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:17.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:17.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:17.609+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:18.102 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:18.029+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:18.030395+0000) 2024-02-19T20:36:18.102 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:17.802+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:17.803788+0000) 2024-02-19T20:36:18.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:17.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:18.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:17.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:18.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:17.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:18.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:17.978+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:18.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:18.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:18.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:18.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:18.104 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:18.012+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:18.104 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:18.012+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:18.104 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:18.114+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:18.115960+0000) 2024-02-19T20:36:18.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:18.804+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:18.805869+0000) 2024-02-19T20:36:18.954 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:18.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:18.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:18.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:18.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:18.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:18.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:18.655+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:19.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:19.031+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:19.032630+0000) 2024-02-19T20:36:19.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:19.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:19.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:19.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:19.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:19.003+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:19.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:19.004+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:19.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:19.117+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:19.118440+0000) 2024-02-19T20:36:19.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:19.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:19.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:19.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:19.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:19.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:19.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:19.007+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:19.954 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:19.807+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:19.808403+0000) 2024-02-19T20:36:19.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:19.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:19.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:19.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:19.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:19.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:19.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:19.637+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:20.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:20.035+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:20.036175+0000) 2024-02-19T20:36:20.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:20.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:20.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:20.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:20.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:20.038+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:20.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:20.039+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:20.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:20.005+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:20.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:20.005+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:20.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:20.005+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:20.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:20.005+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:20.257 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:20.120+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:20.121760+0000) 2024-02-19T20:36:20.899 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:20.810+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:20.811946+0000) 2024-02-19T20:36:20.899 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:20.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:20.899 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:20.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:20.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:20.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:20.900 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:20.650+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:21.176 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:21.038+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:21.039696+0000) 2024-02-19T20:36:21.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:21.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:21.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:21.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:21.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:21.007+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:21.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:21.008+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:21.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:20.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:21.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:20.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:21.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:20.980+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:21.177 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:20.981+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:21.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:21.123+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:21.124209+0000) 2024-02-19T20:36:21.445 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:36:21.271+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65807 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:36:21.445 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:21.267+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:21.268487+0000) 2024-02-19T20:36:21.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:21.267+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:21.268395+0000) 2024-02-19T20:36:21.446 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:21.267+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:21.268825+0000) 2024-02-19T20:36:21.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:21.622+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:21.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:21.622+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:21.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:21.622+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:21.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:21.623+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:22.111 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:22.039+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:22.040171+0000) 2024-02-19T20:36:22.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:21.811+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:21.812282+0000) 2024-02-19T20:36:22.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:22.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:22.111 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:22.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:22.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:22.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:22.112 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:22.007+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:22.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:21.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:22.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:21.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:22.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:21.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:22.112 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:21.935+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:22.367 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:22.123+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:22.124704+0000) 2024-02-19T20:36:22.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:22.814+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:22.815807+0000) 2024-02-19T20:36:22.955 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:22.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:22.955 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:22.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:22.955 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:22.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:22.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:22.888+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:22.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:22.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:22.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:22.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:22.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:22.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:22.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:22.649+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:23.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:23.126+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:23.127578+0000) 2024-02-19T20:36:23.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:23.041+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:23.042172+0000) 2024-02-19T20:36:23.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:23.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:23.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:23.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:23.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:23.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:23.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:23.032+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:23.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:23.816+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:23.817864+0000) 2024-02-19T20:36:23.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:23.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:23.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:23.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:23.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:23.687+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:23.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:23.687+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:23.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:23.881+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:23.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:23.881+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:23.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:23.881+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:23.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:23.881+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:24.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:24.043+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:24.044206+0000) 2024-02-19T20:36:24.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:24.130+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:24.131146+0000) 2024-02-19T20:36:24.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:24.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:24.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:24.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:24.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:24.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:24.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:24.007+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:24.955 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:24.818+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:24.819946+0000) 2024-02-19T20:36:24.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:24.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:24.955 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:24.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:24.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:24.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:24.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:24.661+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:24.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:24.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:24.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:24.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:24.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:24.885+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:24.956 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:24.886+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:25.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:25.132+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:25.133532+0000) 2024-02-19T20:36:25.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:25.045+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:25.046218+0000) 2024-02-19T20:36:25.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:24.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:25.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:24.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:25.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:24.980+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:25.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:24.981+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:25.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:25.620+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:25.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:25.620+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:25.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:25.620+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:25.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:25.621+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:26.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:26.047+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:26.048420+0000) 2024-02-19T20:36:26.123 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:25.822+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:25.823126+0000) 2024-02-19T20:36:26.124 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:25.996+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:26.124 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:25.996+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:26.124 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:25.996+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:26.124 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:25.997+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:26.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:25.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:26.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:25.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:26.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:25.912+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:26.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:25.912+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:26.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:26.135+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:26.136843+0000) 2024-02-19T20:36:26.383 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:36:26.279+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65848 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:36:26.384 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:26.276+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:26.277428+0000) 2024-02-19T20:36:26.384 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:26.274+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:26.275912+0000) 2024-02-19T20:36:26.385 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:26.275+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:26.276349+0000) 2024-02-19T20:36:26.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:26.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:26.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:26.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:26.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:26.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:26.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:26.601+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:27.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:27.047+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:27.048803+0000) 2024-02-19T20:36:27.124 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:26.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:27.124 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:26.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:27.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:26.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:27.125 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:26.923+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:27.125 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:26.822+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:26.823443+0000) 2024-02-19T20:36:27.126 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:26.985+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:27.126 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:26.985+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:27.126 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:26.985+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:27.126 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:26.986+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:27.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:27.136+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:27.137238+0000) 2024-02-19T20:36:27.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:27.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:27.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:27.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:27.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:27.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:27.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:27.632+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:28.126 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:28.051+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:28.052167+0000) 2024-02-19T20:36:28.126 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:27.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:28.127 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:27.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:28.127 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:27.909+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:28.127 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:27.910+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:28.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:27.825+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:27.826115+0000) 2024-02-19T20:36:28.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:28.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:28.127 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:28.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:28.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:28.020+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:28.128 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:28.020+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:28.417 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:28.139+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:28.140144+0000) 2024-02-19T20:36:28.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:28.618+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:28.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:28.618+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:28.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:28.618+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:28.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:28.618+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:29.129 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:29.053+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:29.054671+0000) 2024-02-19T20:36:29.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:28.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:29.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:28.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:29.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:28.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:29.129 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:28.873+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:29.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:28.827+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:28.828484+0000) 2024-02-19T20:36:29.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:29.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:29.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:29.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:29.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:29.040+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:29.130 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:29.041+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:29.417 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:29.141+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:29.142797+0000) 2024-02-19T20:36:29.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:29.578+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:29.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:29.578+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:29.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:29.578+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:29.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:29.578+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:30.132 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:30.055+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:30.056721+0000) 2024-02-19T20:36:30.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:29.831+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:29.832000+0000) 2024-02-19T20:36:30.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:30.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:30.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:30.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:30.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:30.019+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:30.133 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:30.019+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:30.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:29.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:30.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:29.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:30.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:29.903+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:30.134 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:29.904+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:30.417 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:30.145+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:30.146286+0000) 2024-02-19T20:36:30.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:30.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:30.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:30.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:30.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:30.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:30.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:30.605+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:31.135 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:31.057+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:31.058792+0000) 2024-02-19T20:36:31.135 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:30.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:31.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:30.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:31.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:30.854+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:31.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:30.855+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:31.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:30.832+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:30.833968+0000) 2024-02-19T20:36:31.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:31.030+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:31.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:31.030+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:31.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:31.030+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:31.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:31.031+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:31.394 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:36:31.286+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65888 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:36:31.395 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:31.284+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:31.285681+0000) 2024-02-19T20:36:31.395 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:31.283+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:31.284856+0000) 2024-02-19T20:36:31.396 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:31.148+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:31.149175+0000) 2024-02-19T20:36:31.396 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:31.282+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:31.283940+0000) 2024-02-19T20:36:31.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:31.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:31.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:31.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:31.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:31.611+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:31.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:31.612+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:32.135 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:32.058+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:32.059268+0000) 2024-02-19T20:36:32.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:31.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:32.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:31.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:32.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:31.901+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:32.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:31.901+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:32.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:31.833+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:31.834359+0000) 2024-02-19T20:36:32.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:32.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:32.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:32.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:32.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:32.079+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:32.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:32.079+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:32.419 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:32.148+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:32.149588+0000) 2024-02-19T20:36:32.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:32.584+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:32.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:32.584+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:32.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:32.584+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:32.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:32.585+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:33.115 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:33.060+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:33.061492+0000) 2024-02-19T20:36:33.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:32.835+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:32.836323+0000) 2024-02-19T20:36:33.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:33.128+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:33.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:33.128+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:33.116 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:33.128+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:33.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:32.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:33.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:32.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:33.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:32.886+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:33.117 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:32.886+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:33.368 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:33.129+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:33.369 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:33.150+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:33.151791+0000) 2024-02-19T20:36:33.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:33.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:33.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:33.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:33.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:33.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:33.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:33.620+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:34.086 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:33.837+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:33.838847+0000) 2024-02-19T20:36:34.087 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:34.062+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:34.063633+0000) 2024-02-19T20:36:34.087 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:33.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:34.087 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:33.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:34.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:33.895+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:34.088 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:33.895+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:34.368 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:34.153+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:34.154357+0000) 2024-02-19T20:36:34.369 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:34.099+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:34.369 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:34.099+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:34.369 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:34.099+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:34.369 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:34.100+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:34.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:34.571+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:34.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:34.571+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:34.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:34.571+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:34.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:34.572+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:35.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:35.065+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:35.066957+0000) 2024-02-19T20:36:35.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:34.840+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:34.841313+0000) 2024-02-19T20:36:35.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:35.087+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:35.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:35.087+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:35.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:35.087+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:35.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:35.087+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:35.143 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:34.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:35.144 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:34.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:35.144 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:34.871+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:35.144 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:34.871+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:35.421 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:35.155+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:35.156381+0000) 2024-02-19T20:36:35.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:35.583+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:35.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:35.583+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:35.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:35.583+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:35.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:35.584+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:36.145 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:36.068+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:36.069227+0000) 2024-02-19T20:36:36.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:35.842+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:35.843770+0000) 2024-02-19T20:36:36.145 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:36.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:36.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:36.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:36.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:36.054+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:36.146 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:36.055+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:36.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:35.840+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:36.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:35.840+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:36.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:35.840+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:36.147 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:35.840+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:36.400 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:36:36.294+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65929 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:36:36.400 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:36.290+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:36.291669+0000) 2024-02-19T20:36:36.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:36.157+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:36.158879+0000) 2024-02-19T20:36:36.401 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:36.291+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:36.292543+0000) 2024-02-19T20:36:36.401 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:36.291+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:36.292261+0000) 2024-02-19T20:36:36.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:36.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:36.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:36.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:36.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:36.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:36.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:36.617+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:37.145 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:37.068+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:37.069778+0000) 2024-02-19T20:36:37.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:36.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:37.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:36.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:37.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:36.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:37.146 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:36.822+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:37.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:36.843+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:36.844132+0000) 2024-02-19T20:36:37.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:37.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:37.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:37.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:37.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:37.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:37.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:37.034+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:37.424 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:37.158+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:37.159534+0000) 2024-02-19T20:36:37.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:37.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:37.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:37.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:37.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:37.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:37.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:37.624+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:38.058 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:37.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:38.058 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:37.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:38.058 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:37.810+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:38.058 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:37.810+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:38.059 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:37.846+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:37.847668+0000) 2024-02-19T20:36:38.059 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:38.044+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:38.059 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:38.044+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:38.059 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:38.044+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:38.059 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:38.045+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:38.369 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:38.070+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:38.071830+0000) 2024-02-19T20:36:38.369 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:38.160+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:38.161701+0000) 2024-02-19T20:36:38.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:38.849+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:38.850028+0000) 2024-02-19T20:36:38.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:38.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:38.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:38.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:38.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:38.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:38.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:38.650+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:38.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:38.815+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:38.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:38.815+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:38.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:38.815+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:38.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:38.815+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:39.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:39.073+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:39.074389+0000) 2024-02-19T20:36:39.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:39.163+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:39.164094+0000) 2024-02-19T20:36:39.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:39.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:39.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:39.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:39.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:39.048+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:39.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:39.049+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:39.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:39.851+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:39.852449+0000) 2024-02-19T20:36:39.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:39.666+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:39.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:39.666+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:39.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:39.666+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:39.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:39.667+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:39.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:39.849+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:39.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:39.849+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:39.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:39.849+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:39.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:39.849+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:40.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:40.076+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:40.077947+0000) 2024-02-19T20:36:40.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:40.165+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:40.166124+0000) 2024-02-19T20:36:40.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:40.032+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:40.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:40.032+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:40.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:40.032+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:40.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:40.033+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:40.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:40.618+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:40.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:40.618+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:40.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:40.618+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:40.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:40.619+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:41.102 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:41.079+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:41.080026+0000) 2024-02-19T20:36:41.102 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:40.890+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:41.102 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:40.890+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:41.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:40.890+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:41.103 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:40.890+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:41.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:40.853+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:40.854527+0000) 2024-02-19T20:36:41.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:41.031+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:41.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:41.032+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:41.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:41.032+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:41.103 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:41.032+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:41.369 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:36:41.302+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 65969 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:36:41.369 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:41.298+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:41.299720+0000) 2024-02-19T20:36:41.370 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:41.298+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:41.299466+0000) 2024-02-19T20:36:41.370 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:41.167+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:41.168814+0000) 2024-02-19T20:36:41.370 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:41.299+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:41.300229+0000) 2024-02-19T20:36:41.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:41.854+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:41.855013+0000) 2024-02-19T20:36:41.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:41.632+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:41.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:41.632+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:41.958 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:41.632+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:41.958 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:41.633+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:41.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:41.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:41.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:41.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:41.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:41.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:41.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:41.867+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:42.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:42.079+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:42.080529+0000) 2024-02-19T20:36:42.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:42.168+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:42.169221+0000) 2024-02-19T20:36:42.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:41.991+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:42.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:41.991+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:42.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:41.991+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:42.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:41.992+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:42.897 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:42.857+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:42.858242+0000) 2024-02-19T20:36:42.898 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:42.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:42.898 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:42.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:42.898 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:42.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:42.898 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:42.649+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:43.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:43.082+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:43.083129+0000) 2024-02-19T20:36:43.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:42.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:43.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:42.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:43.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:42.987+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:43.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:42.988+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:43.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:42.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:43.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:42.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:43.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:42.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:43.158 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:42.911+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:43.431 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:43.170+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:43.171199+0000) 2024-02-19T20:36:43.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:43.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:43.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:43.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:43.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:43.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:43.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:43.678+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:44.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:44.085+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:44.086396+0000) 2024-02-19T20:36:44.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:43.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:44.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:43.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:44.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:43.873+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:44.160 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:43.874+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:44.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:43.860+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:43.861493+0000) 2024-02-19T20:36:44.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:43.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:44.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:43.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:44.160 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:43.977+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:44.161 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:43.978+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:44.431 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:44.172+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:44.173222+0000) 2024-02-19T20:36:44.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:44.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:44.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:44.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:44.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:44.692+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:44.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:44.693+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:45.161 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:45.087+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:45.088551+0000) 2024-02-19T20:36:45.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:44.863+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:44.864192+0000) 2024-02-19T20:36:45.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:44.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:45.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:44.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:45.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:44.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:45.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:44.970+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:45.163 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:44.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:45.163 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:44.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:45.163 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:44.867+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:45.163 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:44.867+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:45.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:45.174+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:45.175869+0000) 2024-02-19T20:36:45.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:45.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:45.958 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:45.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:45.958 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:45.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:45.958 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:45.703+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:45.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:45.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:45.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:45.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:45.959 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:45.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:45.959 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:45.867+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:45.959 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:45.865+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:45.866916+0000) 2024-02-19T20:36:45.959 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:45.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:45.959 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:45.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:45.959 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:45.925+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:45.959 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:45.926+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:46.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:46.090+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:46.091230+0000) 2024-02-19T20:36:46.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:46.177+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:46.178486+0000) 2024-02-19T20:36:46.649 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:36:46.310+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66010 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:36:46.649 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:46.307+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:46.308512+0000) 2024-02-19T20:36:46.650 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:46.307+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:46.308897+0000) 2024-02-19T20:36:46.651 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:46.306+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:46.307885+0000) 2024-02-19T20:36:46.651 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:46.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:46.651 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:46.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:46.651 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:46.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:46.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:46.866+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:46.867415+0000) 2024-02-19T20:36:46.958 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:46.662+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:46.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:46.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:46.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:46.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:46.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:46.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:46.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:46.879+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:47.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:47.090+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:47.091846+0000) 2024-02-19T20:36:47.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:47.176+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:47.178911+0000) 2024-02-19T20:36:47.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:46.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:47.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:46.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:47.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:46.969+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:47.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:46.971+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:47.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:47.870+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:47.871196+0000) 2024-02-19T20:36:47.937 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:47.949+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:47.937 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:47.949+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:47.937 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:47.949+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:47.937 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:47.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:47.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:47.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:47.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:47.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:47.938 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:47.642+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:47.938 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:47.887+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:47.938 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:47.887+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:47.938 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:47.887+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:47.938 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:47.887+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:48.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:47.950+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:48.256 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:48.093+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:48.094089+0000) 2024-02-19T20:36:48.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:48.180+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:48.181081+0000) 2024-02-19T20:36:48.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:48.607+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:48.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:48.607+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:48.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:48.607+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:48.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:48.608+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:49.169 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:49.095+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:49.096775+0000) 2024-02-19T20:36:49.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:48.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:49.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:48.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:49.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:48.889+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:49.170 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:48.890+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:49.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:48.872+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:48.873575+0000) 2024-02-19T20:36:49.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:48.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:49.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:48.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:49.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:48.946+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:49.170 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:48.947+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:49.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:49.182+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:49.183000+0000) 2024-02-19T20:36:49.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:49.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:49.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:49.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:49.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:49.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:49.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:49.603+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:50.171 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:50.098+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:50.099180+0000) 2024-02-19T20:36:50.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:49.875+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:50.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:49.875+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:50.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:49.875+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:50.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:49.875+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:50.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:49.874+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:49.875837+0000) 2024-02-19T20:36:50.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:49.961+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:50.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:49.961+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:50.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:49.961+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:50.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:49.962+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:50.435 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:50.184+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:50.185723+0000) 2024-02-19T20:36:50.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:50.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:50.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:50.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:50.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:50.570+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:50.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:50.571+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:51.174 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:51.100+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:51.101474+0000) 2024-02-19T20:36:51.174 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:50.878+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:50.879384+0000) 2024-02-19T20:36:51.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:51.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:51.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:51.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:51.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:51.006+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:51.175 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:51.007+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:51.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:50.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:51.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:50.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:51.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:50.894+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:51.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:50.894+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:51.425 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:36:51.319+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66050 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:36:51.426 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:51.315+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:51.316276+0000) 2024-02-19T20:36:51.426 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:51.315+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:51.316609+0000) 2024-02-19T20:36:51.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:51.187+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:51.188380+0000) 2024-02-19T20:36:51.427 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:51.315+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:51.316342+0000) 2024-02-19T20:36:51.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:51.574+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:51.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:51.574+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:51.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:51.574+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:51.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:51.575+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:52.175 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:52.101+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:52.101996+0000) 2024-02-19T20:36:52.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:51.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:52.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:51.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:52.175 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:51.922+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:52.176 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:51.923+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:52.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:51.878+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:51.879906+0000) 2024-02-19T20:36:52.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:52.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:52.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:52.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:52.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:52.010+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:52.176 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:52.010+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:52.439 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:52.187+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:52.188860+0000) 2024-02-19T20:36:52.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:52.548+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:52.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:52.548+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:52.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:52.548+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:52.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:52.549+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:53.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:53.103+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:53.104417+0000) 2024-02-19T20:36:53.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:52.882+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:52.883368+0000) 2024-02-19T20:36:53.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:53.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:53.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:53.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:53.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:53.034+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:53.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:53.034+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:53.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:52.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:53.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:52.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:53.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:52.898+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:53.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:52.899+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:53.441 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:53.191+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:53.192010+0000) 2024-02-19T20:36:53.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:53.585+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:53.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:53.585+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:53.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:53.585+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:53.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:53.586+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:54.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:54.105+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:54.106488+0000) 2024-02-19T20:36:54.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:53.885+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:53.886003+0000) 2024-02-19T20:36:54.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:54.076+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:54.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:54.076+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:54.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:54.076+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:54.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:54.076+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:54.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:53.936+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:54.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:53.936+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:54.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:53.936+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:54.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:53.936+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:54.437 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:54.193+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:54.194678+0000) 2024-02-19T20:36:54.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:54.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:54.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:54.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:54.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:54.591+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:54.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:54.592+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:55.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:55.107+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:55.108666+0000) 2024-02-19T20:36:55.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:54.887+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:54.888090+0000) 2024-02-19T20:36:55.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:55.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:55.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:55.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:55.180 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:55.113+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:55.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:55.114+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:55.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:54.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:55.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:54.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:55.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:54.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:55.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:54.954+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:55.441 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:55.195+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:55.196873+0000) 2024-02-19T20:36:55.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:55.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:55.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:55.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:55.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:55.619+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:55.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:55.620+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:56.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:56.110+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:56.111162+0000) 2024-02-19T20:36:56.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:55.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:56.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:55.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:56.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:55.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:56.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:55.921+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:56.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:55.889+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:55.890314+0000) 2024-02-19T20:36:56.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:56.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:56.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:56.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:56.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:56.088+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:56.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:56.088+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:56.432 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:36:56.327+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66091 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:36:56.433 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:56.323+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:56.324733+0000) 2024-02-19T20:36:56.433 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:56.323+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:56.324507+0000) 2024-02-19T20:36:56.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:56.198+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:56.198979+0000) 2024-02-19T20:36:56.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:56.325+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:56.326150+0000) 2024-02-19T20:36:56.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:56.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:56.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:56.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:56.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:56.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:56.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:56.658+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:57.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:57.110+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:57.111760+0000) 2024-02-19T20:36:57.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:56.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:57.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:56.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:57.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:56.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:57.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:56.889+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:57.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:56.889+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:56.890864+0000) 2024-02-19T20:36:57.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:57.124+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:57.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:57.124+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:57.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:57.124+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:57.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:57.125+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:57.445 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:57.198+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:57.199399+0000) 2024-02-19T20:36:57.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:57.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:57.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:57.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:57.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:57.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:57.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:57.697+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:58.157 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:58.114+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:58.115239+0000) 2024-02-19T20:36:58.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:57.891+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:57.892967+0000) 2024-02-19T20:36:58.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:58.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:58.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:58.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:58.158 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:58.170+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:58.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:57.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:58.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:57.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:58.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:57.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:58.159 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:57.897+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:58.447 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:58.171+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:58.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:58.200+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:58.201559+0000) 2024-02-19T20:36:58.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:58.656+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:58.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:58.656+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:58.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:58.656+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:58.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:58.657+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:36:59.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:59.117+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:59.118808+0000) 2024-02-19T20:36:59.180 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:58.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:59.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:58.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:59.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:58.943+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:36:59.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:58.943+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:59.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:58.893+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:58.894956+0000) 2024-02-19T20:36:59.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:59.134+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:59.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:59.134+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:59.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:59.134+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:36:59.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:59.135+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:36:59.447 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:59.202+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:59.203572+0000) 2024-02-19T20:36:59.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:59.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:59.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:59.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:59.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:59.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:36:59.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:36:59.702+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:00.135 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:00.120+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:00.121004+0000) 2024-02-19T20:37:00.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:36:59.897+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:59.898514+0000) 2024-02-19T20:37:00.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:00.148+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:00.135 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:00.148+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:00.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:00.148+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:00.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:59.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:00.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:59.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:00.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:59.906+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:00.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:36:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:36:59.907+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:00.447 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:00.149+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:00.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:00.204+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:00.205523+0000) 2024-02-19T20:37:00.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:00.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:00.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:00.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:00.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:00.702+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:00.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:00.703+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:01.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:01.122+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:01.122995+0000) 2024-02-19T20:37:01.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:00.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:01.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:00.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:01.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:00.955+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:01.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:00.955+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:01.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:00.899+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:00.900737+0000) 2024-02-19T20:37:01.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:01.138+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:01.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:01.138+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:01.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:01.138+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:01.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:01.139+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:01.443 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:37:01.336+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66131 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:37:01.443 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:01.332+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:01.333291+0000) 2024-02-19T20:37:01.444 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:01.332+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:01.333030+0000) 2024-02-19T20:37:01.444 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:01.206+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:01.207700+0000) 2024-02-19T20:37:01.444 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:01.332+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:01.333250+0000) 2024-02-19T20:37:01.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:01.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:01.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:01.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:01.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:01.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:01.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:01.671+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:02.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:02.122+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:02.123524+0000) 2024-02-19T20:37:02.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:01.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:02.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:01.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:02.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:01.950+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:02.181 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:01.950+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:02.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:01.900+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:01.901168+0000) 2024-02-19T20:37:02.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:02.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:02.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:02.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:02.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:02.145+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:02.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:02.146+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:02.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:02.207+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:02.208218+0000) 2024-02-19T20:37:02.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:02.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:02.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:02.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:02.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:02.689+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:02.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:02.690+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:03.181 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:03.124+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:03.125610+0000) 2024-02-19T20:37:03.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:02.903+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:02.904704+0000) 2024-02-19T20:37:03.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:03.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:03.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:03.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:03.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:03.119+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:03.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:03.120+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:03.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:02.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:03.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:02.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:03.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:02.960+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:03.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:02.960+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:03.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:03.209+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:03.210412+0000) 2024-02-19T20:37:03.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:03.663+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:03.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:03.663+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:03.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:03.663+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:03.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:03.664+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:04.114 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:03.905+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:03.906921+0000) 2024-02-19T20:37:04.115 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:03.939+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:04.115 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:03.939+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:04.115 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:03.939+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:04.115 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:03.940+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:04.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:04.127+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:04.128682+0000) 2024-02-19T20:37:04.372 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:04.211+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:04.212482+0000) 2024-02-19T20:37:04.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:04.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:04.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:04.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:04.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:04.162+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:04.372 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:04.163+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:04.936 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:04.909+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:04.910487+0000) 2024-02-19T20:37:04.937 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:04.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:04.937 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:04.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:04.937 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:04.696+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:04.937 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:04.697+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:05.198 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:05.130+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:05.130995+0000) 2024-02-19T20:37:05.198 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:04.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:05.198 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:04.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:05.198 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:04.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:05.199 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:04.950+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:05.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:05.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:05.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:05.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:05.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:05.210+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:05.449 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:05.212+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:05.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:05.213+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:05.214828+0000) 2024-02-19T20:37:06.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:05.911+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:05.912781+0000) 2024-02-19T20:37:06.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:05.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:06.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:05.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:06.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:05.910+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:06.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:05.911+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:06.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:05.743+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:06.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:05.743+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:06.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:05.743+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:06.007 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:05.744+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:06.372 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:37:06.344+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66172 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:37:06.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:06.132+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:06.133123+0000) 2024-02-19T20:37:06.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:06.341+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:06.342169+0000) 2024-02-19T20:37:06.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:06.216+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:06.217018+0000) 2024-02-19T20:37:06.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:06.341+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:06.342347+0000) 2024-02-19T20:37:06.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:06.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:06.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:06.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:06.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:06.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:06.373 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:06.242+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:06.374 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:06.340+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:06.341833+0000) 2024-02-19T20:37:06.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:06.912+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:06.913309+0000) 2024-02-19T20:37:06.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:06.697+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:06.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:06.697+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:06.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:06.697+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:06.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:06.698+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:06.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:06.878+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:06.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:06.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:06.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:06.879+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:06.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:06.879+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:07.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:07.132+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:07.133776+0000) 2024-02-19T20:37:07.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:07.216+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:07.217563+0000) 2024-02-19T20:37:07.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:07.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:07.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:07.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:07.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:07.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:07.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:07.194+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:07.960 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:07.915+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:07.916523+0000) 2024-02-19T20:37:07.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:07.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:07.960 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:07.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:07.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:07.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:07.961 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:07.685+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:07.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:07.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:07.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:07.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:07.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:07.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:07.961 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:07.897+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:08.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:08.134+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:08.135889+0000) 2024-02-19T20:37:08.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:08.218+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:08.219709+0000) 2024-02-19T20:37:08.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:08.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:08.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:08.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:08.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:08.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:08.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:08.201+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:08.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:08.650+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:08.904 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:08.650+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:08.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:08.650+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:08.905 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:08.651+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:09.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:08.917+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:08.918596+0000) 2024-02-19T20:37:09.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:09.138+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:09.139404+0000) 2024-02-19T20:37:09.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:08.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:09.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:08.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:09.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:08.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:09.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:08.938+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:09.449 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:09.220+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:09.221781+0000) 2024-02-19T20:37:09.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:09.196+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:09.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:09.196+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:09.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:09.196+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:09.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:09.197+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:09.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:09.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:09.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:09.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:09.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:09.630+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:09.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:09.630+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:10.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:09.919+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:09.920762+0000) 2024-02-19T20:37:10.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:10.140+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:10.141796+0000) 2024-02-19T20:37:10.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:09.951+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:10.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:09.951+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:10.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:09.951+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:10.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:09.952+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:10.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:10.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:10.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:10.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:10.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:10.202+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:10.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:10.203+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:10.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:10.223+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:10.224234+0000) 2024-02-19T20:37:10.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:10.587+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:10.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:10.587+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:10.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:10.587+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:10.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:10.588+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:11.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:10.922+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:10.923291+0000) 2024-02-19T20:37:11.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:11.143+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:11.144356+0000) 2024-02-19T20:37:11.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:10.951+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:11.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:10.951+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:11.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:10.951+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:11.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:10.951+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:11.450 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:37:11.352+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66212 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:37:11.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:11.349+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:11.350350+0000) 2024-02-19T20:37:11.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:11.226+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:11.227385+0000) 2024-02-19T20:37:11.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:11.348+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:11.349821+0000) 2024-02-19T20:37:11.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:11.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:11.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:11.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:11.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:11.198+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:11.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:11.199+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:11.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:11.349+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:11.350129+0000) 2024-02-19T20:37:11.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:11.568+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:11.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:11.568+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:11.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:11.568+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:11.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:11.569+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:12.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:11.922+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:11.923856+0000) 2024-02-19T20:37:12.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:12.143+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:12.144955+0000) 2024-02-19T20:37:12.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:11.961+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:12.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:11.961+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:12.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:11.961+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:12.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:11.961+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:12.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:12.226+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:12.227934+0000) 2024-02-19T20:37:12.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:12.234+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:12.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:12.234+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:12.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:12.234+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:12.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:12.235+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:12.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:12.594+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:12.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:12.594+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:12.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:12.594+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:12.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:12.595+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:13.132 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:12.926+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:12.927088+0000) 2024-02-19T20:37:13.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:12.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:13.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:12.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:13.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:12.954+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:13.133 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:12.954+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:13.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:13.145+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:13.146932+0000) 2024-02-19T20:37:13.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:13.230+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:13.231227+0000) 2024-02-19T20:37:13.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:13.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:13.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:13.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:13.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:13.273+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:13.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:13.274+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:13.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:13.609+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:13.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:13.609+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:13.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:13.609+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:13.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:13.610+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:14.136 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:13.928+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:13.929139+0000) 2024-02-19T20:37:14.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:13.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:14.136 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:13.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:14.137 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:13.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:14.137 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:13.975+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:14.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:14.149+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:14.150444+0000) 2024-02-19T20:37:14.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:14.232+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:14.233769+0000) 2024-02-19T20:37:14.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:14.228+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:14.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:14.228+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:14.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:14.228+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:14.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:14.229+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:14.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:14.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:14.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:14.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:14.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:14.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:14.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:14.634+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:15.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:15.152+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:15.153181+0000) 2024-02-19T20:37:15.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:14.930+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:14.931879+0000) 2024-02-19T20:37:15.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:14.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:15.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:14.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:15.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:14.975+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:15.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:14.976+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:15.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:15.234+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:15.235886+0000) 2024-02-19T20:37:15.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:15.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:15.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:15.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:15.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:15.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:15.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:15.225+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:15.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:15.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:15.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:15.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:15.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:15.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:15.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:15.624+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:16.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:15.934+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:15.935446+0000) 2024-02-19T20:37:16.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:16.186+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:16.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:16.186+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:16.173 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:16.186+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:16.173 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:16.155+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:16.156718+0000) 2024-02-19T20:37:16.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:15.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:16.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:15.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:16.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:15.937+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:16.174 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:15.937+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:16.450 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:37:16.360+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66252 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:37:16.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:16.356+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:16.357905+0000) 2024-02-19T20:37:16.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:16.238+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:16.239213+0000) 2024-02-19T20:37:16.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:16.357+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:16.358180+0000) 2024-02-19T20:37:16.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:16.187+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:16.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:16.358+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:16.359046+0000) 2024-02-19T20:37:16.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:16.637+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:16.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:16.637+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:16.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:16.637+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:16.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:16.637+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:17.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:16.935+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:16.936079+0000) 2024-02-19T20:37:17.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:17.156+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:17.157336+0000) 2024-02-19T20:37:17.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:16.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:17.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:16.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:17.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:16.938+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:17.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:16.938+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:17.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:17.214+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:17.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:17.214+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:17.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:17.214+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:17.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:17.215+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:17.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:17.238+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:17.239873+0000) 2024-02-19T20:37:17.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:17.605+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:17.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:17.605+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:17.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:17.605+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:17.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:17.606+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:18.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:18.158+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:18.159501+0000) 2024-02-19T20:37:18.182 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:17.937+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:17.938169+0000) 2024-02-19T20:37:18.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:17.947+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:18.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:17.947+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:18.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:17.947+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:18.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:17.947+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:18.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:18.241+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:18.242602+0000) 2024-02-19T20:37:18.450 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:18.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:18.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:18.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:18.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:18.201+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:18.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:18.202+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:18.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:18.610+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:18.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:18.610+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:18.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:18.610+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:18.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:18.610+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:19.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:19.160+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:19.161884+0000) 2024-02-19T20:37:19.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:18.939+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:18.940733+0000) 2024-02-19T20:37:19.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:18.914+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:19.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:18.914+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:19.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:18.914+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:19.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:18.914+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:19.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:19.244+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:19.245187+0000) 2024-02-19T20:37:19.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:19.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:19.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:19.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:19.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:19.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:19.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:19.200+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:19.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:19.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:19.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:19.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:19.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:19.608+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:19.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:19.608+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:20.150 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:19.941+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:19.942952+0000) 2024-02-19T20:37:20.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:19.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:20.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:19.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:20.150 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:19.888+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:20.151 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:19.889+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:20.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:20.163+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:20.164474+0000) 2024-02-19T20:37:20.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:20.246+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:20.247906+0000) 2024-02-19T20:37:20.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:20.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:20.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:20.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:20.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:20.178+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:20.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:20.178+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:20.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:20.566+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:20.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:20.566+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:20.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:20.566+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:20.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:20.566+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:21.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:21.166+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:21.167022+0000) 2024-02-19T20:37:21.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:20.944+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:20.945019+0000) 2024-02-19T20:37:21.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:20.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:21.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:20.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:21.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:20.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:21.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:20.920+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:21.450 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:37:21.369+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66293 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:37:21.451 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:21.365+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:21.366749+0000) 2024-02-19T20:37:21.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:21.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:21.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:21.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:21.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:21.199+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:21.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:21.200+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:21.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:21.365+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:21.366723+0000) 2024-02-19T20:37:21.452 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:21.249+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:21.250610+0000) 2024-02-19T20:37:21.452 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:21.365+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:21.366714+0000) 2024-02-19T20:37:21.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:21.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:21.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:21.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:21.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:21.534+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:21.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:21.535+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:22.182 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:22.166+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:22.167450+0000) 2024-02-19T20:37:22.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:21.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:22.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:21.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:22.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:21.934+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:22.183 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:21.934+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:22.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:21.944+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:21.945657+0000) 2024-02-19T20:37:22.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:22.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:22.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:22.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:22.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:22.173+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:22.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:22.174+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:22.450 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:22.250+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:22.251249+0000) 2024-02-19T20:37:22.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:22.507+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:22.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:22.507+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:22.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:22.507+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:22.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:22.508+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:23.155 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:22.948+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:22.949173+0000) 2024-02-19T20:37:23.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:22.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:23.155 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:22.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:23.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:22.897+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:23.156 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:22.898+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:23.450 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:23.168+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:23.169425+0000) 2024-02-19T20:37:23.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:23.253+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:23.254699+0000) 2024-02-19T20:37:23.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:23.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:23.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:23.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:23.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:23.174+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:23.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:23.175+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:23.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:23.555+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:23.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:23.555+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:23.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:23.555+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:23.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:23.555+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:24.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:23.950+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:23.951246+0000) 2024-02-19T20:37:24.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:23.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:24.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:23.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:24.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:23.920+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:24.141 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:23.920+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:24.451 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:24.171+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:24.172681+0000) 2024-02-19T20:37:24.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:24.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:24.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:24.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:24.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:24.154+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:24.451 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:24.155+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:24.452 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:24.257+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:24.258237+0000) 2024-02-19T20:37:24.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:24.586+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:24.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:24.586+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:24.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:24.586+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:24.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:24.586+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:25.246 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:25.175+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:25.176162+0000) 2024-02-19T20:37:25.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:24.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:25.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:24.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:25.246 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:24.962+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:25.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:24.963+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:25.247 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:24.952+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:24.953836+0000) 2024-02-19T20:37:25.247 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:25.142+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:25.247 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:25.142+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:25.247 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:25.142+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:25.247 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:25.143+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:25.505 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:25.259+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:25.260307+0000) 2024-02-19T20:37:25.939 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:25.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:25.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:25.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:25.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:25.603+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:25.940 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:25.604+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:26.248 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:26.177+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:26.178292+0000) 2024-02-19T20:37:26.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:25.956+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:25.957327+0000) 2024-02-19T20:37:26.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:26.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:26.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:26.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:26.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:26.158+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:26.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:26.159+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:26.249 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:25.953+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:26.250 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:25.953+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:26.250 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:25.953+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:26.250 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:25.953+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:26.250 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:26.261+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:26.262863+0000) 2024-02-19T20:37:26.505 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:26.373+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:26.374423+0000) 2024-02-19T20:37:26.505 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:37:26.377+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66334 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:37:26.506 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:26.374+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:26.375099+0000) 2024-02-19T20:37:26.506 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:26.373+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:26.374550+0000) 2024-02-19T20:37:26.891 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:26.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:26.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:26.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:26.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:26.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:26.892 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:26.617+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:27.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:26.957+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:26.958015+0000) 2024-02-19T20:37:27.156 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:27.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:27.157 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:27.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:27.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:26.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:27.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:26.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:27.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:26.905+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:27.157 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:26.905+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:27.451 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:27.177+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:27.178744+0000) 2024-02-19T20:37:27.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:27.262+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:27.263440+0000) 2024-02-19T20:37:27.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:27.169+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:27.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:27.170+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:27.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:27.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:27.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:27.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:27.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:27.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:27.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:27.617+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:28.166 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:27.959+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:27.960256+0000) 2024-02-19T20:37:28.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:27.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:28.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:27.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:28.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:27.925+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:28.167 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:27.926+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:28.451 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:28.179+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:28.180817+0000) 2024-02-19T20:37:28.451 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:28.265+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:28.265975+0000) 2024-02-19T20:37:28.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:28.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:28.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:28.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:28.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:28.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:28.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:28.200+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:28.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:28.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:28.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:28.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:28.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:28.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:28.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:28.605+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:29.168 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:28.962+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:28.963021+0000) 2024-02-19T20:37:29.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:28.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:29.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:28.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:29.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:28.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:29.169 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:28.916+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:29.451 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:29.182+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:29.183019+0000) 2024-02-19T20:37:29.452 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:29.267+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:29.268015+0000) 2024-02-19T20:37:29.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:29.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:29.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:29.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:29.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:29.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:29.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:29.206+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:29.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:29.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:29.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:29.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:29.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:29.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:29.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:29.647+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:30.171 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:29.964+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:29.965005+0000) 2024-02-19T20:37:30.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:29.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:30.171 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:29.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:30.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:29.926+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:30.172 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:29.927+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:30.451 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:30.184+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:30.185434+0000) 2024-02-19T20:37:30.452 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:30.269+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:30.270145+0000) 2024-02-19T20:37:30.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:30.244+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:30.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:30.244+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:30.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:30.244+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:30.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:30.245+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:30.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:30.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:30.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:30.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:30.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:30.684+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:30.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:30.685+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:31.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:31.186+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:31.187824+0000) 2024-02-19T20:37:31.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:30.966+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:30.967395+0000) 2024-02-19T20:37:31.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:30.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:31.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:30.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:31.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:30.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:31.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:30.949+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:31.626 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:37:31.385+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66375 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:37:31.626 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:31.382+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:31.383547+0000) 2024-02-19T20:37:31.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:31.271+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:31.272127+0000) 2024-02-19T20:37:31.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:31.381+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:31.382898+0000) 2024-02-19T20:37:31.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:31.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:31.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:31.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:31.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:31.271+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:31.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:31.272+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:31.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:31.381+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:31.382802+0000) 2024-02-19T20:37:31.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:31.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:31.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:31.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:31.936 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:31.691+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:31.937 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:31.692+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:32.247 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:32.187+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:32.188302+0000) 2024-02-19T20:37:32.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:31.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:32.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:31.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:32.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:31.949+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:32.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:31.949+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:32.248 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:31.966+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:31.967864+0000) 2024-02-19T20:37:32.248 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:32.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:32.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:32.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:32.249 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:32.260+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:32.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:32.261+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:32.505 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:32.271+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:32.272821+0000) 2024-02-19T20:37:32.963 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:32.969+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:32.970408+0000) 2024-02-19T20:37:32.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:32.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:32.963 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:32.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:32.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:32.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:32.964 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:32.720+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:32.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:32.965+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:32.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:32.965+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:32.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:32.965+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:32.964 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:32.966+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:33.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:33.189+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:33.190455+0000) 2024-02-19T20:37:33.567 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:33.274+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:33.274969+0000) 2024-02-19T20:37:33.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:33.292+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:33.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:33.292+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:33.568 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:33.292+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:33.568 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:33.293+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:33.864 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:33.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:33.864 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:33.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:33.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:33.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:33.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:33.671+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:34.178 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:33.971+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:33.972925+0000) 2024-02-19T20:37:34.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:33.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:34.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:33.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:34.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:33.921+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:34.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:33.921+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:34.451 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:34.192+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:34.192495+0000) 2024-02-19T20:37:34.452 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:34.276+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:34.277065+0000) 2024-02-19T20:37:34.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:34.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:34.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:34.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:34.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:34.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:34.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:34.277+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:34.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:34.665+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:34.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:34.665+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:34.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:34.665+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:34.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:34.666+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:35.181 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:34.974+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:34.976221+0000) 2024-02-19T20:37:35.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:34.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:35.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:34.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:35.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:34.930+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:35.182 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:34.930+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:35.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:35.195+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:35.196000+0000) 2024-02-19T20:37:35.452 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:35.278+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:35.279822+0000) 2024-02-19T20:37:35.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:35.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:35.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:35.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:35.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:35.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:35.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:35.292+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:35.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:35.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:35.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:35.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:35.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:35.705+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:35.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:35.705+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:36.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:35.978+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:35.979093+0000) 2024-02-19T20:37:36.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:36.197+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:36.198538+0000) 2024-02-19T20:37:36.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:35.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:36.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:35.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:36.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:35.948+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:36.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:35.948+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:36.452 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:37:36.393+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66417 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:37:36.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:36.390+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:36.391241+0000) 2024-02-19T20:37:36.453 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:36.280+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:36.281970+0000) 2024-02-19T20:37:36.453 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:36.390+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:36.391099+0000) 2024-02-19T20:37:36.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:36.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:36.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:36.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:36.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:36.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:36.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:36.311+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:36.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:36.390+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:36.391702+0000) 2024-02-19T20:37:37.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:36.978+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:36.979649+0000) 2024-02-19T20:37:37.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:36.737+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:37.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:36.737+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:37.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:36.737+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:37.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:36.738+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:37.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:36.978+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:37.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:36.978+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:37.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:36.978+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:37.007 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:36.979+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:37.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:37.198+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:37.199360+0000) 2024-02-19T20:37:37.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:37.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:37.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:37.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:37.452 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:37.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:37.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:37.302+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:37.453 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:37.281+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:37.282439+0000) 2024-02-19T20:37:38.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:37.981+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:37.982044+0000) 2024-02-19T20:37:38.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:37.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:38.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:37.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:38.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:37.780+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:38.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:37.781+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:38.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:38.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:38.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:38.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:38.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:38.016+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:38.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:38.017+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:38.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:38.200+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:38.201715+0000) 2024-02-19T20:37:38.453 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:38.283+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:38.284855+0000) 2024-02-19T20:37:38.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:38.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:38.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:38.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:38.454 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:38.276+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:38.454 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:38.276+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:39.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:38.983+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:38.984645+0000) 2024-02-19T20:37:39.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:38.802+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:39.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:38.802+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:39.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:38.802+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:39.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:38.803+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:39.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:39.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:39.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:39.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:39.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:39.001+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:39.187 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:39.002+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:39.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:39.203+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:39.204203+0000) 2024-02-19T20:37:39.452 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:39.286+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:39.287255+0000) 2024-02-19T20:37:39.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:39.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:39.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:39.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:39.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:39.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:39.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:39.253+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:40.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:39.986+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:39.987176+0000) 2024-02-19T20:37:40.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:39.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:40.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:39.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:40.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:39.991+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:40.185 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:39.991+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:40.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:39.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:40.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:39.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:40.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:39.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:40.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:39.780+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:40.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:40.205+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:40.206429+0000) 2024-02-19T20:37:40.452 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:40.288+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:40.289641+0000) 2024-02-19T20:37:40.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:40.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:40.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:40.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:40.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:40.252+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:40.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:40.253+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:41.184 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:40.989+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:40.990728+0000) 2024-02-19T20:37:41.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:40.782+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:41.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:40.782+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:41.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:40.782+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:41.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:40.782+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:41.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:41.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:41.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:41.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:41.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:41.020+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:41.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:41.021+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:41.452 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:37:41.401+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66458 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:37:41.453 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:41.208+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:41.209020+0000) 2024-02-19T20:37:41.453 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:41.397+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:41.398906+0000) 2024-02-19T20:37:41.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:41.291+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:41.292105+0000) 2024-02-19T20:37:41.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:41.398+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:41.399155+0000) 2024-02-19T20:37:41.455 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:41.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:41.455 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:41.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:41.455 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:41.281+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:41.455 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:41.282+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:41.456 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:41.398+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:41.399299+0000) 2024-02-19T20:37:42.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:41.990+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:41.991223+0000) 2024-02-19T20:37:42.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:41.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:42.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:41.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:42.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:41.831+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:42.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:41.832+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:42.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:42.208+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:42.209508+0000) 2024-02-19T20:37:42.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:42.314+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:42.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:42.314+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:42.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:42.314+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:42.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:42.314+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:42.376 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:42.070+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:42.376 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:42.070+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:42.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:42.070+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:42.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:42.071+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:42.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:42.291+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:42.292891+0000) 2024-02-19T20:37:43.185 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:42.993+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:42.994570+0000) 2024-02-19T20:37:43.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:42.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:43.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:42.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:43.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:42.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:43.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:42.862+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:43.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:43.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:43.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:43.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:43.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:43.089+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:43.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:43.089+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:43.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:43.212+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:43.213013+0000) 2024-02-19T20:37:43.453 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:43.294+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:43.295034+0000) 2024-02-19T20:37:43.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:43.299+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:43.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:43.299+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:43.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:43.299+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:43.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:43.299+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:44.185 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:43.995+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:43.996547+0000) 2024-02-19T20:37:44.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:43.845+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:44.185 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:43.845+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:44.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:43.845+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:44.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:43.846+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:44.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:44.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:44.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:44.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:44.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:44.090+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:44.186 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:44.091+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:44.452 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:44.215+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:44.216288+0000) 2024-02-19T20:37:44.453 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:44.296+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:44.297567+0000) 2024-02-19T20:37:44.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:44.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:44.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:44.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:44.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:44.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:44.453 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:44.345+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:45.075 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:44.997+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:44.998669+0000) 2024-02-19T20:37:45.075 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:44.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:45.076 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:44.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:45.076 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:44.812+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:45.076 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:44.813+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:45.347 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:45.217+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:45.218415+0000) 2024-02-19T20:37:45.347 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:45.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:45.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:45.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:45.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:45.088+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:45.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:45.089+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:45.348 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:45.298+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:45.299716+0000) 2024-02-19T20:37:45.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:45.360+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:45.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:45.360+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:45.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:45.360+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:45.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:45.361+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:46.113 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:46.001+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:46.002117+0000) 2024-02-19T20:37:46.113 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:45.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:46.114 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:45.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:46.114 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:45.821+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:46.114 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:45.822+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:46.375 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:46.220+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:46.221761+0000) 2024-02-19T20:37:46.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:46.346+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:46.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:46.346+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:46.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:46.346+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:46.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:46.347+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:46.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:46.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:46.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:46.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:46.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:46.126+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:46.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:46.127+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:46.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:46.302+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:46.303202+0000) 2024-02-19T20:37:46.661 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:37:46.409+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66498 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:37:46.661 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:46.405+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:46.406840+0000) 2024-02-19T20:37:46.662 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:46.405+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:46.406751+0000) 2024-02-19T20:37:46.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:46.405+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:46.406414+0000) 2024-02-19T20:37:46.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:46.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:46.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:46.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:46.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:46.777+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:46.966 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:46.778+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:47.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:47.001+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:47.002832+0000) 2024-02-19T20:37:47.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:47.221+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:47.222514+0000) 2024-02-19T20:37:47.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:47.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:47.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:47.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:47.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:47.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:47.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:47.172+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:47.733 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:47.302+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:47.303951+0000) 2024-02-19T20:37:47.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:47.352+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:47.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:47.352+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:47.734 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:47.352+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:47.734 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:47.353+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:47.991 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:47.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:47.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:47.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:47.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:47.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:47.992 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:47.747+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:48.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:48.005+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:48.006340+0000) 2024-02-19T20:37:48.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:48.223+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:48.224580+0000) 2024-02-19T20:37:48.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:48.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:48.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:48.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:48.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:48.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:48.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:48.172+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:48.694 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:48.305+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:48.306080+0000) 2024-02-19T20:37:48.694 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:48.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:48.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:48.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:48.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:48.378+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:48.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:48.379+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:48.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:48.707+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:48.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:48.707+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:48.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:48.707+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:48.965 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:48.708+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:49.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:49.007+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:49.008447+0000) 2024-02-19T20:37:49.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:49.225+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:49.226607+0000) 2024-02-19T20:37:49.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:49.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:49.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:49.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:49.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:49.149+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:49.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:49.150+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:49.721 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:49.307+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:49.308832+0000) 2024-02-19T20:37:49.722 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:49.371+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:49.722 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:49.371+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:49.722 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:49.371+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:49.722 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:49.372+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:49.996 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:49.735+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:49.996 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:49.735+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:49.997 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:49.735+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:49.997 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:49.736+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:50.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:50.010+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:50.011001+0000) 2024-02-19T20:37:50.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:50.227+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:50.228748+0000) 2024-02-19T20:37:50.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:50.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:50.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:50.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:50.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:50.171+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:50.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:50.171+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:50.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:50.422+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:50.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:50.422+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:50.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:50.422+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:50.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:50.423+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:50.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:50.311+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:50.312026+0000) 2024-02-19T20:37:51.005 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:51.012+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:51.013175+0000) 2024-02-19T20:37:51.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:50.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:51.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:50.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:51.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:50.767+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:51.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:50.768+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:51.376 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:51.231+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:51.232297+0000) 2024-02-19T20:37:51.376 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:51.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:51.376 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:51.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:51.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:51.135+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:51.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:51.136+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:51.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:51.314+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:51.315639+0000) 2024-02-19T20:37:51.721 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:37:51.417+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66539 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:37:51.721 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:51.413+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:51.414515+0000) 2024-02-19T20:37:51.722 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:51.413+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:51.414590+0000) 2024-02-19T20:37:51.722 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:51.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:51.722 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:51.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:51.722 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:51.414+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:51.415008+0000) 2024-02-19T20:37:51.722 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:51.444+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:51.722 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:51.444+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:51.723 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:51.444+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:51.723 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:51.445+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:51.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:51.734+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:51.999 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:51.735+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:52.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:52.231+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:52.232903+0000) 2024-02-19T20:37:52.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:52.012+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:52.013919+0000) 2024-02-19T20:37:52.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:52.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:52.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:52.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:52.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:52.122+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:52.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:52.122+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:52.736 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:52.315+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:52.316139+0000) 2024-02-19T20:37:52.736 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:52.494+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:52.737 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:52.494+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:52.737 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:52.494+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:52.737 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:52.494+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:53.001 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:52.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:53.002 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:52.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:53.002 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:52.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:53.002 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:52.750+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:53.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:53.015+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:53.016350+0000) 2024-02-19T20:37:53.255 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:53.234+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:53.235299+0000) 2024-02-19T20:37:53.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:53.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:53.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:53.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:53.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:53.163+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:53.256 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:53.164+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:53.567 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:53.317+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:53.318186+0000) 2024-02-19T20:37:53.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:53.541+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:53.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:53.541+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:53.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:53.541+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:53.568 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:53.541+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:53.864 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:53.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:53.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:53.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:53.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:53.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:53.865 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:53.742+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:54.185 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:54.017+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:54.018894+0000) 2024-02-19T20:37:54.453 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:54.236+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:54.237797+0000) 2024-02-19T20:37:54.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:54.212+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:54.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:54.212+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:54.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:54.212+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:54.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:54.213+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:54.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:54.320+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:54.321134+0000) 2024-02-19T20:37:54.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:54.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:54.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:54.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:54.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:54.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:54.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:54.560+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:55.186 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:55.020+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:55.021091+0000) 2024-02-19T20:37:55.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:54.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:55.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:54.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:55.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:54.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:55.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:54.785+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:55.453 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:55.239+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:55.240327+0000) 2024-02-19T20:37:55.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:55.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:55.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:55.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:55.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:55.235+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:55.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:55.236+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:55.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:55.322+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:55.323185+0000) 2024-02-19T20:37:55.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:55.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:55.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:55.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:55.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:55.602+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:55.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:55.603+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:56.186 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:56.022+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:56.023188+0000) 2024-02-19T20:37:56.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:55.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:56.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:55.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:56.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:55.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:56.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:55.806+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:56.454 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:37:56.426+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66579 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:37:56.454 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:56.422+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:56.423550+0000) 2024-02-19T20:37:56.455 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:56.241+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:56.242955+0000) 2024-02-19T20:37:56.455 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:56.421+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:56.422382+0000) 2024-02-19T20:37:56.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:56.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:56.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:56.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:56.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:56.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:56.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:56.210+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:56.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:56.324+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:56.325612+0000) 2024-02-19T20:37:56.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:56.422+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:56.423294+0000) 2024-02-19T20:37:56.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:56.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:56.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:56.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:56.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:56.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:56.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:56.568+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:57.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:57.022+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:57.023688+0000) 2024-02-19T20:37:57.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:56.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:57.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:56.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:57.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:56.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:57.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:56.799+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:57.454 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:57.242+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:57.243702+0000) 2024-02-19T20:37:57.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:57.178+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:57.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:57.178+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:57.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:57.178+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:57.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:57.178+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:57.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:57.325+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:57.326362+0000) 2024-02-19T20:37:57.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:57.595+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:57.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:57.595+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:57.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:57.595+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:57.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:57.596+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:58.186 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:58.024+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:58.025774+0000) 2024-02-19T20:37:58.186 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:57.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:58.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:57.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:58.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:57.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:58.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:57.785+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:58.454 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:58.245+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:58.246163+0000) 2024-02-19T20:37:58.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:58.221+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:58.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:58.221+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:58.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:58.221+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:58.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:58.222+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:58.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:58.327+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:58.328752+0000) 2024-02-19T20:37:58.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:58.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:58.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:58.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:58.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:58.570+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:58.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:58.571+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:59.183 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:59.028+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:59.029296+0000) 2024-02-19T20:37:59.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:58.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:59.183 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:58.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:59.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:58.814+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:37:59.184 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:58.815+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:37:59.454 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:59.247+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:59.248171+0000) 2024-02-19T20:37:59.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:59.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:59.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:59.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:59.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:59.196+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:37:59.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:59.197+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:37:59.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:37:59.329+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:59.330881+0000) 2024-02-19T20:37:59.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:59.534+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:59.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:59.534+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:59.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:59.534+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:37:59.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:37:59.534+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:00.137 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:00.030+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:00.031942+0000) 2024-02-19T20:38:00.138 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:59.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:00.138 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:59.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:00.138 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:59.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:00.138 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:37:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:37:59.798+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:00.454 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:00.249+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:00.250689+0000) 2024-02-19T20:38:00.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:00.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:00.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:00.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:00.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:00.151+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:00.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:00.151+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:00.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:00.333+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:00.334510+0000) 2024-02-19T20:38:00.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:00.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:00.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:00.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:00.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:00.567+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:00.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:00.568+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:01.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:01.032+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:01.033956+0000) 2024-02-19T20:38:01.147 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:00.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:01.147 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:00.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:01.148 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:00.798+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:01.148 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:00.799+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:01.414 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:01.252+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:01.253258+0000) 2024-02-19T20:38:01.414 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:01.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:01.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:01.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:01.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:01.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:01.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:01.161+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:01.415 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:01.335+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:01.336631+0000) 2024-02-19T20:38:01.755 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:38:01.434+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66620 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:38:01.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:01.430+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:01.431678+0000) 2024-02-19T20:38:01.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:01.430+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:01.431316+0000) 2024-02-19T20:38:01.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:01.430+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:01.431125+0000) 2024-02-19T20:38:01.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:01.520+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:01.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:01.520+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:01.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:01.520+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:01.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:01.520+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:02.105 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:02.033+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:02.034743+0000) 2024-02-19T20:38:02.106 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:01.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:02.106 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:01.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:02.106 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:01.839+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:02.106 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:01.839+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:02.377 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:02.253+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:02.254046+0000) 2024-02-19T20:38:02.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:02.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:02.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:02.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:02.378 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:02.119+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:02.378 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:02.119+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:02.378 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:02.336+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:02.337150+0000) 2024-02-19T20:38:02.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:02.549+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:02.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:02.549+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:02.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:02.549+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:02.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:02.549+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:03.143 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:03.036+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:03.037947+0000) 2024-02-19T20:38:03.143 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:02.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:03.143 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:02.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:03.143 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:02.863+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:03.144 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:02.863+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:03.454 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:03.255+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:03.256130+0000) 2024-02-19T20:38:03.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:03.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:03.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:03.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:03.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:03.156+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:03.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:03.157+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:03.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:03.338+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:03.339307+0000) 2024-02-19T20:38:03.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:03.556+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:03.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:03.556+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:03.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:03.556+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:03.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:03.556+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:04.162 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:04.039+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:04.040009+0000) 2024-02-19T20:38:04.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:03.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:04.162 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:03.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:04.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:03.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:04.163 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:03.910+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:04.454 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:04.257+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:04.258808+0000) 2024-02-19T20:38:04.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:04.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:04.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:04.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:04.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:04.175+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:04.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:04.176+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:04.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:04.340+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:04.341808+0000) 2024-02-19T20:38:04.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:04.533+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:04.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:04.533+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:04.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:04.533+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:04.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:04.534+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:05.164 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:05.041+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:05.042076+0000) 2024-02-19T20:38:05.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:04.911+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:05.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:04.911+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:05.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:04.911+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:05.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:04.911+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:05.454 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:05.261+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:05.262378+0000) 2024-02-19T20:38:05.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:05.177+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:05.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:05.177+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:05.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:05.177+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:05.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:05.178+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:05.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:05.343+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:05.344254+0000) 2024-02-19T20:38:05.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:05.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:05.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:05.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:05.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:05.505+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:05.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:05.506+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:06.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:06.043+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:06.044513+0000) 2024-02-19T20:38:06.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:05.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:06.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:05.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:06.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:05.938+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:06.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:05.939+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:06.445 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:38:06.442+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66660 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:38:06.445 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:06.438+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:06.439768+0000) 2024-02-19T20:38:06.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:06.263+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:06.264843+0000) 2024-02-19T20:38:06.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:06.438+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:06.439476+0000) 2024-02-19T20:38:06.446 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:06.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:06.446 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:06.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:06.446 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:06.211+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:06.447 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:06.212+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:06.447 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:06.345+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:06.346708+0000) 2024-02-19T20:38:06.447 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:06.438+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:06.439695+0000) 2024-02-19T20:38:06.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:06.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:06.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:06.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:06.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:06.518+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:06.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:06.519+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:07.163 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:07.044+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:07.045169+0000) 2024-02-19T20:38:07.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:07.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:07.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:07.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:07.164 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:07.176+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:07.164 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:06.926+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:07.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:06.926+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:07.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:06.926+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:07.165 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:06.927+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:07.455 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:07.264+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:07.265613+0000) 2024-02-19T20:38:07.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:07.177+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:07.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:07.346+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:07.347487+0000) 2024-02-19T20:38:07.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:07.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:07.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:07.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:07.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:07.558+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:07.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:07.559+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:08.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:08.047+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:08.048189+0000) 2024-02-19T20:38:08.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:07.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:08.187 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:07.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:08.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:07.920+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:08.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:07.920+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:08.455 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:08.267+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:08.268158+0000) 2024-02-19T20:38:08.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:08.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:08.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:08.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:08.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:08.209+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:08.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:08.209+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:08.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:08.348+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:08.349461+0000) 2024-02-19T20:38:08.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:08.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:08.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:08.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:08.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:08.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:08.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:08.556+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:09.235 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:09.049+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:09.050118+0000) 2024-02-19T20:38:09.235 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:08.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:09.236 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:08.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:09.236 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:08.957+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:09.236 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:08.958+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:09.499 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:09.270+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:09.271709+0000) 2024-02-19T20:38:09.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:09.248+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:09.499 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:09.248+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:09.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:09.248+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:09.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:09.249+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:09.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:09.350+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:09.351496+0000) 2024-02-19T20:38:09.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:09.512+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:09.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:09.512+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:09.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:09.512+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:09.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:09.513+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:10.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:10.051+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:10.052263+0000) 2024-02-19T20:38:10.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:09.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:10.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:09.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:10.189 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:09.912+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:10.189 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:09.913+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:10.455 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:10.274+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:10.275271+0000) 2024-02-19T20:38:10.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:10.255+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:10.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:10.255+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:10.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:10.255+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:10.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:10.256+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:10.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:10.352+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:10.353653+0000) 2024-02-19T20:38:10.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:10.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:10.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:10.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:10.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:10.555+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:10.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:10.556+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:11.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:11.053+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:11.054468+0000) 2024-02-19T20:38:11.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:10.921+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:11.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:10.921+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:11.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:10.921+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:11.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:10.923+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:11.444 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:38:11.450+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66701 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:38:11.445 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:11.446+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:11.447924+0000) 2024-02-19T20:38:11.445 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:11.277+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:11.278816+0000) 2024-02-19T20:38:11.446 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:11.446+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:11.447607+0000) 2024-02-19T20:38:11.446 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:11.243+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:11.446 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:11.243+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:11.446 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:11.243+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:11.446 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:11.243+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:11.446 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:11.354+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:11.355622+0000) 2024-02-19T20:38:11.446 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:11.446+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:11.447925+0000) 2024-02-19T20:38:11.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:11.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:11.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:11.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:11.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:11.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:11.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:11.536+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:12.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:12.054+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:12.054975+0000) 2024-02-19T20:38:12.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:11.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:12.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:11.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:12.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:11.933+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:12.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:11.934+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:12.455 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:12.278+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:12.279506+0000) 2024-02-19T20:38:12.455 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:12.254+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:12.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:12.254+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:12.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:12.254+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:12.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:12.255+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:12.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:12.355+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:12.356311+0000) 2024-02-19T20:38:12.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:12.534+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:12.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:12.534+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:12.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:12.534+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:12.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:12.535+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:13.187 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:13.056+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:13.057377+0000) 2024-02-19T20:38:13.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:12.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:13.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:12.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:13.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:12.910+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:13.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:12.911+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:13.455 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:13.281+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:13.282781+0000) 2024-02-19T20:38:13.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:13.213+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:13.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:13.214+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:13.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:13.214+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:13.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:13.214+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:13.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:13.358+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:13.359718+0000) 2024-02-19T20:38:13.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:13.516+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:13.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:13.516+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:13.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:13.516+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:13.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:13.517+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:14.177 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:14.059+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:14.060870+0000) 2024-02-19T20:38:14.177 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:13.876+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:14.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:13.876+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:14.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:13.876+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:14.178 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:13.877+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:14.455 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:14.285+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:14.286214+0000) 2024-02-19T20:38:14.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:14.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:14.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:14.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:14.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:14.191+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:14.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:14.191+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:14.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:14.362+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:14.363177+0000) 2024-02-19T20:38:14.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:14.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:14.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:14.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:14.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:14.559+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:14.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:14.560+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:15.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:15.063+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:15.064388+0000) 2024-02-19T20:38:15.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:14.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:15.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:14.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:15.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:14.896+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:15.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:14.896+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:15.455 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:15.287+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:15.288319+0000) 2024-02-19T20:38:15.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:15.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:15.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:15.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:15.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:15.193+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:15.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:15.193+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:15.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:15.365+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:15.366609+0000) 2024-02-19T20:38:15.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:15.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:15.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:15.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:15.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:15.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:15.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:15.526+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:16.188 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:16.066+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:16.067818+0000) 2024-02-19T20:38:16.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:15.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:16.188 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:15.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:16.189 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:15.854+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:16.189 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:15.855+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:16.439 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:16.289+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:16.290368+0000) 2024-02-19T20:38:16.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:16.232+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:16.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:16.232+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:16.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:16.232+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:16.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:16.232+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:16.440 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:16.368+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:16.369157+0000) 2024-02-19T20:38:16.755 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:38:16.458+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66741 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:38:16.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:16.454+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:16.455019+0000) 2024-02-19T20:38:16.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:16.454+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:16.455457+0000) 2024-02-19T20:38:16.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:16.454+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:16.455706+0000) 2024-02-19T20:38:16.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:16.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:16.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:16.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:16.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:16.491+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:16.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:16.491+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:17.053 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:16.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:17.054 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:16.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:17.054 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:16.806+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:17.055 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:16.807+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:17.355 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:17.067+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:17.068361+0000) 2024-02-19T20:38:17.355 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:17.290+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:17.291195+0000) 2024-02-19T20:38:17.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:17.226+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:17.355 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:17.226+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:17.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:17.226+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:17.356 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:17.227+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:17.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:17.368+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:17.369714+0000) 2024-02-19T20:38:17.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:17.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:17.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:17.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:17.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:17.535+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:17.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:17.536+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:18.056 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:17.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:18.056 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:17.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:18.056 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:17.796+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:18.057 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:17.797+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:18.357 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:18.069+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:18.070966+0000) 2024-02-19T20:38:18.358 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:18.292+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:18.293214+0000) 2024-02-19T20:38:18.358 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:18.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:18.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:18.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:18.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:18.216+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:18.359 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:18.216+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:18.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:18.370+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:18.371968+0000) 2024-02-19T20:38:18.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:18.552+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:18.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:18.552+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:18.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:18.552+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:18.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:18.553+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:19.058 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:18.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:19.059 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:18.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:19.059 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:18.795+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:19.059 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:18.796+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:19.359 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:19.072+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:19.073469+0000) 2024-02-19T20:38:19.359 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:19.294+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:19.295222+0000) 2024-02-19T20:38:19.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:19.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:19.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:19.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:19.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:19.172+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:19.360 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:19.172+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:19.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:19.373+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:19.374122+0000) 2024-02-19T20:38:19.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:19.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:19.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:19.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:19.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:19.589+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:19.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:19.589+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:20.179 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:20.074+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:20.075709+0000) 2024-02-19T20:38:20.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:19.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:20.179 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:19.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:20.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:19.838+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:20.180 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:19.839+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:20.453 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:20.297+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:20.298186+0000) 2024-02-19T20:38:20.453 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:20.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:20.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:20.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:20.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:20.192+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:20.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:20.193+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:20.454 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:20.375+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:20.376189+0000) 2024-02-19T20:38:20.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:20.603+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:20.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:20.603+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:20.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:20.603+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:20.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:20.604+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:21.141 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:21.077+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:21.078148+0000) 2024-02-19T20:38:21.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:20.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:21.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:20.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:21.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:20.859+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:21.142 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:20.860+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:21.448 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:21.300+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:21.300862+0000) 2024-02-19T20:38:21.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:21.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:21.449 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:21.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:21.449 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:21.155+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:21.449 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:21.155+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:21.449 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:21.378+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:21.378732+0000) 2024-02-19T20:38:21.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:21.463+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:21.464252+0000) 2024-02-19T20:38:21.755 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:38:21.467+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66782 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:38:21.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:21.463+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:21.464351+0000) 2024-02-19T20:38:21.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:21.464+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:21.464995+0000) 2024-02-19T20:38:21.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:21.597+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:21.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:21.597+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:21.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:21.597+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:21.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:21.597+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:22.148 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:22.078+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:22.078720+0000) 2024-02-19T20:38:22.148 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:21.856+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:22.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:21.856+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:22.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:21.856+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:22.149 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:21.858+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:22.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:22.300+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:22.301610+0000) 2024-02-19T20:38:22.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:22.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:22.456 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:22.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:22.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:22.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:22.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:22.161+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:22.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:22.377+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:22.379436+0000) 2024-02-19T20:38:22.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:22.592+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:22.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:22.592+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:22.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:22.592+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:22.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:22.593+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:23.147 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:23.080+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:23.081036+0000) 2024-02-19T20:38:23.147 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:22.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:23.147 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:22.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:23.148 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:22.883+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:23.148 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:22.883+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:23.148 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:23.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:23.148 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:23.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:23.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:23.302+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:23.303640+0000) 2024-02-19T20:38:23.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:23.160+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:23.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:23.161+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:23.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:23.381+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:23.382091+0000) 2024-02-19T20:38:23.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:23.565+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:23.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:23.565+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:23.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:23.565+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:23.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:23.566+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:24.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:23.845+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:24.139 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:23.845+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:24.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:23.845+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:24.140 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:23.846+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:24.140 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:24.082+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:24.083452+0000) 2024-02-19T20:38:24.456 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:24.305+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:24.306103+0000) 2024-02-19T20:38:24.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:24.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:24.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:24.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:24.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:24.153+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:24.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:24.153+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:24.457 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:24.383+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:24.384827+0000) 2024-02-19T20:38:24.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:24.612+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:24.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:24.612+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:24.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:24.612+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:24.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:24.613+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:25.071 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:24.802+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:25.071 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:24.802+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:25.071 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:24.802+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:25.072 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:24.802+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:25.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:25.084+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:25.085897+0000) 2024-02-19T20:38:25.372 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:25.308+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:25.309289+0000) 2024-02-19T20:38:25.372 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:25.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:25.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:25.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:25.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:25.141+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:25.373 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:25.141+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:25.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:25.386+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:25.387249+0000) 2024-02-19T20:38:25.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:25.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:25.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:25.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:25.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:25.631+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:25.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:25.632+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:26.074 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:25.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:26.074 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:25.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:26.074 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:25.769+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:26.074 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:25.770+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:26.376 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:26.310+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:26.311377+0000) 2024-02-19T20:38:26.376 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:26.087+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:26.088693+0000) 2024-02-19T20:38:26.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:26.092+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:26.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:26.092+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:26.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:26.092+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:26.377 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:26.092+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:26.735 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:38:26.475+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66822 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:38:26.735 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:26.471+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:26.472801+0000) 2024-02-19T20:38:26.736 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:26.389+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:26.390915+0000) 2024-02-19T20:38:26.736 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:26.472+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:26.473197+0000) 2024-02-19T20:38:26.736 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:26.471+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:26.472700+0000) 2024-02-19T20:38:26.736 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:26.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:26.736 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:26.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:26.736 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:26.594+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:26.737 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:26.594+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:27.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:26.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:27.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:26.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:27.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:26.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:27.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:26.749+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:27.297 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:27.088+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:27.089303+0000) 2024-02-19T20:38:27.297 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:27.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:27.297 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:27.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:27.298 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:27.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:27.298 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:27.059+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:27.617 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:27.310+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:27.311929+0000) 2024-02-19T20:38:27.617 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:27.390+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:27.391618+0000) 2024-02-19T20:38:27.969 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:27.771+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:27.969 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:27.771+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:27.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:27.771+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:27.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:27.772+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:27.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:27.630+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:27.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:27.630+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:27.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:27.630+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:27.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:27.631+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:28.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:28.091+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:28.092812+0000) 2024-02-19T20:38:28.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:28.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:28.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:28.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:28.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:28.038+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:28.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:28.038+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:28.596 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:28.313+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:28.314275+0000) 2024-02-19T20:38:28.596 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:28.393+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:28.394199+0000) 2024-02-19T20:38:28.969 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:28.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:28.969 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:28.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:28.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:28.778+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:28.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:28.779+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:28.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:28.609+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:28.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:28.610+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:28.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:28.610+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:28.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:28.611+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:29.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:29.093+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:29.094931+0000) 2024-02-19T20:38:29.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:29.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:29.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:29.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:29.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:29.083+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:29.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:29.083+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:29.563 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:29.315+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:29.316310+0000) 2024-02-19T20:38:29.563 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:29.396+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:29.397734+0000) 2024-02-19T20:38:29.969 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:29.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:29.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:29.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:29.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:29.577+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:29.970 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:29.578+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:29.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:29.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:29.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:29.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:29.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:29.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:29.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:29.765+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:30.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:30.095+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:30.096944+0000) 2024-02-19T20:38:30.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:30.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:30.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:30.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:30.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:30.064+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:30.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:30.064+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:30.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:30.318+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:30.319779+0000) 2024-02-19T20:38:30.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:30.400+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:30.401124+0000) 2024-02-19T20:38:30.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:30.554+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:30.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:30.554+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:30.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:30.554+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:30.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:30.555+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:31.045 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:30.772+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:31.045 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:30.772+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:31.045 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:30.772+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:31.046 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:30.773+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:31.308 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:31.098+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:31.099058+0000) 2024-02-19T20:38:31.308 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:31.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:31.308 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:31.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:31.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:31.059+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:31.309 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:31.059+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:31.586 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:38:31.484+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66863 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:38:31.587 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:31.322+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:31.323099+0000) 2024-02-19T20:38:31.587 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:31.480+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:31.481919+0000) 2024-02-19T20:38:31.587 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:31.403+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:31.404535+0000) 2024-02-19T20:38:31.588 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:31.479+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:31.480917+0000) 2024-02-19T20:38:31.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:31.480+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:31.481268+0000) 2024-02-19T20:38:31.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:31.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:31.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:31.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:31.588 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:31.569+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:31.589 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:31.570+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:31.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:31.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:31.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:31.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:31.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:31.753+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:31.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:31.753+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:32.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:32.098+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:32.099930+0000) 2024-02-19T20:38:32.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:32.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:32.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:32.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:32.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:32.015+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:32.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:32.015+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:32.530 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:32.322+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:32.323950+0000) 2024-02-19T20:38:32.530 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:32.404+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:32.405458+0000) 2024-02-19T20:38:32.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:32.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:32.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:32.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:32.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:32.779+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:32.956 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:32.780+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:32.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:32.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:32.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:32.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:32.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:32.544+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:32.957 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:32.545+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:33.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:33.101+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:33.102028+0000) 2024-02-19T20:38:33.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:32.969+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:33.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:32.969+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:33.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:32.969+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:33.255 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:32.970+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:33.564 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:33.324+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:33.325954+0000) 2024-02-19T20:38:33.564 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:33.408+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:33.409047+0000) 2024-02-19T20:38:33.866 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:33.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:33.867 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:33.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:33.867 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:33.578+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:33.867 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:33.579+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:33.867 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:33.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:33.867 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:33.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:33.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:33.732+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:33.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:33.733+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:34.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:34.103+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:34.104147+0000) 2024-02-19T20:38:34.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:33.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:34.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:33.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:34.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:33.928+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:34.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:33.929+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:34.457 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:34.327+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:34.328011+0000) 2024-02-19T20:38:34.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:34.410+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:34.411212+0000) 2024-02-19T20:38:34.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:34.580+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:34.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:34.580+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:34.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:34.580+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:34.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:34.581+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:34.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:34.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:34.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:34.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:34.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:34.719+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:34.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:34.720+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:35.190 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:35.105+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:35.106201+0000) 2024-02-19T20:38:35.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:34.944+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:35.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:34.944+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:35.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:34.944+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:35.191 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:34.944+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:35.457 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:35.328+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:35.329959+0000) 2024-02-19T20:38:35.458 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:35.412+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:35.413808+0000) 2024-02-19T20:38:35.737 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:35.588+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:35.737 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:35.588+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:35.738 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:35.588+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:35.738 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:35.588+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:36.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:35.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:36.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:35.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:36.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:35.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:36.008 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:35.752+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:36.008 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:35.932+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:36.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:35.932+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:36.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:35.932+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:36.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:35.933+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:36.380 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:36.331+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:36.331998+0000) 2024-02-19T20:38:36.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:36.108+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:36.109517+0000) 2024-02-19T20:38:36.734 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:38:36.491+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66903 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:38:36.734 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:36.488+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:36.488994+0000) 2024-02-19T20:38:36.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:36.487+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:36.488950+0000) 2024-02-19T20:38:36.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:36.556+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:36.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:36.556+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:36.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:36.556+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:36.735 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:36.557+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:36.736 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:36.415+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:36.416281+0000) 2024-02-19T20:38:36.736 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:36.488+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:36.489364+0000) 2024-02-19T20:38:37.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:36.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:37.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:36.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:37.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:36.915+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:37.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:36.915+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:37.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:36.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:37.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:36.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:37.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:36.748+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:37.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:36.748+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:37.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:37.109+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:37.110271+0000) 2024-02-19T20:38:37.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:37.331+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:37.332623+0000) 2024-02-19T20:38:37.467 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-19T20:38:37.480 INFO:tasks.cephadm:Teardown begin 2024-02-19T20:38:37.481 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-19T20:38:37.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-19T20:38:37.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:38:37.521 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:38:37.717 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:37.415+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:37.416871+0000) 2024-02-19T20:38:37.717 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:37.730+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:37.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:37.730+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:37.718 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:37.730+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:37.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:37.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:37.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:37.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:37.718 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:37.576+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:37.719 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:37.576+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:37.970 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:37.731+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:37.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:37.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:37.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:37.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:37.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:37.866+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:37.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:37.866+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:38.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:38.111+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:38.112537+0000) 2024-02-19T20:38:38.723 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:38.335+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:38.336097+0000) 2024-02-19T20:38:38.723 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:38.418+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:38.418996+0000) 2024-02-19T20:38:38.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:38.572+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:38.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:38.572+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:38.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:38.572+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:38.724 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:38.573+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:39.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:38.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:39.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:38.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:39.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:38.822+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:39.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:38.823+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:39.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:38.736+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:39.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:38.736+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:39.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:38.736+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:39.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:38.737+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:39.380 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:39.113+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:39.114744+0000) 2024-02-19T20:38:39.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:39.337+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:39.338168+0000) 2024-02-19T20:38:39.730 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:39.420+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:39.421013+0000) 2024-02-19T20:38:39.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:39.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:39.730 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:39.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:39.731 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:39.523+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:39.731 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:39.524+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:40.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:39.805+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:40.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:39.805+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:40.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:39.805+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:40.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:39.805+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:40.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:39.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:40.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:39.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:40.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:39.744+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:40.006 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:39.745+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:40.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:40.115+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:40.116931+0000) 2024-02-19T20:38:40.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:40.339+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:40.340209+0000) 2024-02-19T20:38:40.572 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:38:40.732 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:40.422+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:40.423046+0000) 2024-02-19T20:38:40.732 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:40.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:40.732 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:40.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:40.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:40.524+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:40.733 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:40.525+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:41.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:40.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:41.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:40.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:41.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:40.746+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:41.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:40.747+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:41.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:40.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:41.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:40.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:41.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:40.832+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:41.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:40.833+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:41.381 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:41.117+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:41.118759+0000) 2024-02-19T20:38:41.381 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:41.341+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:41.342200+0000) 2024-02-19T20:38:41.686 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:38:41.500+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66944 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:38:41.687 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:41.496+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:41.497862+0000) 2024-02-19T20:38:41.687 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:41.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:41.687 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:41.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:41.687 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:41.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:41.688 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:41.424+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:41.425104+0000) 2024-02-19T20:38:41.688 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:41.496+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:41.497933+0000) 2024-02-19T20:38:41.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:41.494+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:41.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:41.494+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:41.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:41.494+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:41.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:41.496+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:41.497120+0000) 2024-02-19T20:38:41.689 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:41.496+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:41.957 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:41.701+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:41.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:41.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:41.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:41.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:41.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:41.803+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:41.957 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:41.803+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:42.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:42.118+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:42.119479+0000) 2024-02-19T20:38:42.710 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:42.424+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:42.425819+0000) 2024-02-19T20:38:42.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:42.465+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:42.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:42.465+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:42.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:42.465+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:42.711 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:42.466+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:42.711 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:42.341+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:42.342932+0000) 2024-02-19T20:38:42.712 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:42.724+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:42.712 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:42.724+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:42.713 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:42.724+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:42.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:42.725+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:42.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:42.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:42.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:42.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:42.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:42.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:42.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:42.764+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:43.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:43.120+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:43.121585+0000) 2024-02-19T20:38:43.664 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:43.343+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:43.344907+0000) 2024-02-19T20:38:43.664 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:43.427+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:43.428040+0000) 2024-02-19T20:38:43.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:43.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:43.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:43.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:43.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:43.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:43.665 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:43.452+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:43.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:43.739+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:43.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:43.739+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:43.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:43.739+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:43.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:43.740+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:43.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:43.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:43.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:43.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:43.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:43.678+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:43.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:43.679+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:44.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:44.122+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:44.123682+0000) 2024-02-19T20:38:44.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:44.429+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:44.430251+0000) 2024-02-19T20:38:44.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:44.345+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:44.346872+0000) 2024-02-19T20:38:44.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:44.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:44.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:44.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:44.696 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:44.709+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:44.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:44.415+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:44.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:44.415+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:44.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:44.415+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:44.697 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:44.415+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:44.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:44.710+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:44.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:44.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:44.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:44.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:44.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:44.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:44.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:44.762+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:45.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:45.126+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:45.127201+0000) 2024-02-19T20:38:45.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:45.432+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:45.433878+0000) 2024-02-19T20:38:45.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:45.348+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:45.349020+0000) 2024-02-19T20:38:45.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:45.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:45.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:45.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:45.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:45.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:45.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:45.406+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:45.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:45.406+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:45.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:45.406+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:45.675 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:45.407+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:45.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:45.688+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:45.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:45.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:45.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:45.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:45.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:45.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:45.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:45.751+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:46.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:46.128+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:46.129522+0000) 2024-02-19T20:38:46.613 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:38:46.508+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 66986 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:38:46.614 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:46.350+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:46.351413+0000) 2024-02-19T20:38:46.614 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:46.503+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:46.504915+0000) 2024-02-19T20:38:46.614 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:46.435+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:46.436041+0000) 2024-02-19T20:38:46.614 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:46.504+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:46.505571+0000) 2024-02-19T20:38:46.615 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:46.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:46.615 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:46.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:46.615 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:46.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:46.615 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:46.421+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:46.615 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:46.504+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:46.505482+0000) 2024-02-19T20:38:46.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:46.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:46.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:46.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:46.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:46.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:46.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:46.721+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:46.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:46.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:46.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:46.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:46.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:46.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:46.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:46.640+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:47.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:47.129+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:47.130283+0000) 2024-02-19T20:38:47.633 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:47.435+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:47.436655+0000) 2024-02-19T20:38:47.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:47.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:47.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:47.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:47.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:47.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:47.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:47.373+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:47.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:47.351+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:47.352040+0000) 2024-02-19T20:38:47.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:47.647+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:47.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:47.647+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:47.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:47.647+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:47.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:47.648+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:47.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:47.692+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:47.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:47.692+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:47.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:47.692+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:47.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:47.692+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:48.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:48.131+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:48.132711+0000) 2024-02-19T20:38:48.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:48.354+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:48.355531+0000) 2024-02-19T20:38:48.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:48.438+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:48.439084+0000) 2024-02-19T20:38:48.629 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:48.369+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:48.629 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:48.369+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:48.629 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:48.369+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:48.629 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:48.370+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:48.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:48.642+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:48.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:48.642+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:48.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:48.642+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:48.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:48.643+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:48.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:48.651+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:48.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:48.651+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:48.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:48.651+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:48.975 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:48.651+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:49.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:49.134+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:49.135074+0000) 2024-02-19T20:38:49.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:38:49.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:38:49.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:49.357+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:49.358021+0000) 2024-02-19T20:38:49.610 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:49.440+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:49.441026+0000) 2024-02-19T20:38:49.610 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:49.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:49.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:49.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:49.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:49.347+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:49.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:49.347+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:49.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:49.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:49.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:49.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:49.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:49.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:49.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:49.623+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:49.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:49.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:49.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:49.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:49.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:49.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:49.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:49.624+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:50.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:50.136+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:50.137819+0000) 2024-02-19T20:38:50.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:50.359+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:50.360424+0000) 2024-02-19T20:38:50.566 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:50.442+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:50.443010+0000) 2024-02-19T20:38:50.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:50.315+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:50.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:50.315+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:50.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:50.315+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:50.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:50.316+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:50.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:50.655+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:50.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:50.655+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:50.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:50.655+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:50.976 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:50.655+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:50.976 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:50.614+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:50.976 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:50.614+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:50.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:50.614+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:50.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:50.614+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:51.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:51.138+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:51.139808+0000) 2024-02-19T20:38:51.556 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:38:51.516+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67027 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:38:51.557 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:51.362+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:51.363162+0000) 2024-02-19T20:38:51.557 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:51.513+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:51.514122+0000) 2024-02-19T20:38:51.557 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:51.292+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:51.557 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:51.292+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:51.558 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:51.292+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:51.558 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:51.293+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:51.558 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:51.513+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:51.514018+0000) 2024-02-19T20:38:51.558 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:51.445+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:51.446607+0000) 2024-02-19T20:38:51.558 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:51.512+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:51.513486+0000) 2024-02-19T20:38:51.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:51.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:51.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:51.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:51.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:51.679+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:51.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:51.680+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:51.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:51.565+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:51.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:51.565+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:51.976 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:51.565+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:51.976 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:51.565+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:52.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:52.139+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:52.140492+0000) 2024-02-19T20:38:52.619 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:52.363+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:52.364088+0000) 2024-02-19T20:38:52.619 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:52.332+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:52.619 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:52.332+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:52.620 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:52.332+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:52.620 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:52.333+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:52.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:52.446+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:52.447357+0000) 2024-02-19T20:38:52.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:52.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:52.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:52.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:52.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:52.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:52.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:52.548+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:52.640 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:38:52.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:52.633+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:52.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:52.633+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:52.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:52.633+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:52.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:52.633+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:53.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:53.142+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:53.143074+0000) 2024-02-19T20:38:53.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:53.343+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:53.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:53.343+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:53.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:53.343+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:53.640 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:53.343+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:53.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:53.365+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:53.366478+0000) 2024-02-19T20:38:53.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:53.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:53.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:53.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:53.641 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:53.654+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:53.641 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:53.449+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:53.450064+0000) 2024-02-19T20:38:53.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:53.563+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:53.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:53.563+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:53.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:53.563+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:53.642 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:53.564+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:53.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:53.654+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:54.244 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:54.144+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:54.145674+0000) 2024-02-19T20:38:54.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:54.368+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:54.369079+0000) 2024-02-19T20:38:54.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:54.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:54.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:54.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:54.506 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:54.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:54.506 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:54.374+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:54.506 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:54.451+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:54.452568+0000) 2024-02-19T20:38:54.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:54.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:54.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:54.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:54.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:54.643+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:54.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:54.644+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:54.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:54.535+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:54.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:54.535+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:54.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:54.535+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:54.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:54.536+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:55.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:55.146+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:55.147864+0000) 2024-02-19T20:38:55.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:55.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:55.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:55.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:55.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:55.353+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:55.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:55.354+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:55.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:55.370+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:55.371194+0000) 2024-02-19T20:38:55.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:55.597+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:55.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:55.597+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:55.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:55.597+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:55.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:55.598+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:55.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:55.455+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:55.456073+0000) 2024-02-19T20:38:55.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:55.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:55.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:55.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:55.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:55.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:55.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:55.494+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:56.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:56.149+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:56.150099+0000) 2024-02-19T20:38:56.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:56.373+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:56.374501+0000) 2024-02-19T20:38:56.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:56.457+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:56.458146+0000) 2024-02-19T20:38:56.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:56.521+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:56.522041+0000) 2024-02-19T20:38:56.507 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:56.387+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:56.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:56.387+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:56.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:56.387+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:56.508 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:56.387+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:56.972 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:38:56.525+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67067 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:38:56.973 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:56.521+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:56.522554+0000) 2024-02-19T20:38:56.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:56.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:56.973 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:56.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:56.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:56.529+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:56.974 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:56.530+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:56.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:56.522+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:56.523410+0000) 2024-02-19T20:38:56.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:56.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:56.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:56.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:56.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:56.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:56.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:56.626+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:57.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:57.149+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:57.150828+0000) 2024-02-19T20:38:57.601 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:57.374+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:57.375443+0000) 2024-02-19T20:38:57.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:57.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:57.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:57.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:57.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:57.364+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:57.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:57.365+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:57.602 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:57.458+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:57.459093+0000) 2024-02-19T20:38:57.603 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:57.515+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:57.603 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:57.515+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:57.603 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:57.515+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:57.603 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:57.515+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:57.972 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:57.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:57.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:57.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:57.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:57.615+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:57.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:57.616+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:58.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:58.152+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:58.153438+0000) 2024-02-19T20:38:58.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:58.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:58.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:58.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:58.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:58.344+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:58.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:58.345+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:58.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:58.460+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:58.461743+0000) 2024-02-19T20:38:58.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:58.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:58.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:58.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:58.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:58.490+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:58.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:58.491+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:58.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:58.376+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:58.377476+0000) 2024-02-19T20:38:58.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:58.567+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:58.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:58.567+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:58.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:58.567+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:58.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:58.568+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:38:59.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:59.154+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:59.155490+0000) 2024-02-19T20:38:59.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:59.464+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:59.465270+0000) 2024-02-19T20:38:59.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:59.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:59.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:59.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:59.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:59.473+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:38:59.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:38:59.473+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:59.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:59.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:59.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:59.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:59.756 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:59.359+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:38:59.757 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:38:59.360+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:38:59.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:59.378+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:59.379897+0000) 2024-02-19T20:38:59.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:59.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:59.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:59.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:59.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:59.538+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:38:59.758 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:38:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:38:59.538+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:00.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:00.158+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:00.159022+0000) 2024-02-19T20:39:00.754 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:00.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:00.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:00.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:00.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:00.373+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:00.755 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:00.374+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:00.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:00.381+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:00.382637+0000) 2024-02-19T20:39:00.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:00.561+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:00.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:00.561+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:00.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:00.561+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:00.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:00.562+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:00.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:00.466+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:00.467247+0000) 2024-02-19T20:39:00.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:00.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:00.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:00.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:00.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:00.522+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:00.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:00.522+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:01.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:01.160+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:01.161574+0000) 2024-02-19T20:39:01.574 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:39:01.533+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67108 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:39:01.575 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:01.383+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:01.384755+0000) 2024-02-19T20:39:01.575 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:01.528+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:01.529700+0000) 2024-02-19T20:39:01.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:01.327+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:01.576 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:01.327+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:01.576 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:01.327+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:01.576 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:01.328+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:01.576 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:01.529+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:01.530800+0000) 2024-02-19T20:39:01.576 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:01.468+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:01.469227+0000) 2024-02-19T20:39:01.576 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:01.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:01.576 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:01.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:01.577 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:01.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:01.577 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:01.499+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:01.577 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:01.529+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:01.530111+0000) 2024-02-19T20:39:01.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:01.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:01.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:01.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:01.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:01.593+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:01.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:01.594+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:02.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:02.161+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:02.162230+0000) 2024-02-19T20:39:02.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:02.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:02.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:02.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:02.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:02.355+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:02.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:02.356+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:02.605 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:02.384+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:02.385550+0000) 2024-02-19T20:39:02.605 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:02.594+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:02.606 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:02.594+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:02.606 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:02.594+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:02.606 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:02.595+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:02.606 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:02.469+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:02.470358+0000) 2024-02-19T20:39:02.606 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:02.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:02.606 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:02.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:02.607 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:02.548+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:02.607 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:02.549+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:03.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:03.164+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:03.165493+0000) 2024-02-19T20:39:03.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:03.388+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:03.389063+0000) 2024-02-19T20:39:03.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:03.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:03.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:03.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:03.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:03.310+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:03.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:03.310+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:03.567 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:03.472+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:03.473771+0000) 2024-02-19T20:39:03.567 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:03.526+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:03.567 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:03.526+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:03.567 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:03.526+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:03.568 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:03.527+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:03.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:03.614+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:03.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:03.614+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:03.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:03.614+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:03.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:03.614+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:04.460 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:04.390+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:04.391158+0000) 2024-02-19T20:39:04.461 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:04.166+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:04.167508+0000) 2024-02-19T20:39:04.461 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:04.322+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:04.461 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:04.322+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:04.461 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:04.322+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:04.461 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:04.323+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:04.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:39:04.642 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:39:04.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:04.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:04.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:04.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:04.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:04.569+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:04.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:04.570+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:04.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:04.475+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:04.476575+0000) 2024-02-19T20:39:04.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:04.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:04.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:04.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:04.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:04.530+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:04.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:04.530+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:05.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:05.168+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:05.169750+0000) 2024-02-19T20:39:05.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:05.334+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:05.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:05.334+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:05.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:05.334+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:05.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:05.335+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:05.580 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:05.392+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:05.393323+0000) 2024-02-19T20:39:05.580 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:05.548+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:05.580 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:05.548+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:05.580 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:05.548+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:05.580 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:05.549+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:05.580 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:05.477+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:05.478807+0000) 2024-02-19T20:39:05.581 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:05.537+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:05.581 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:05.537+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:05.581 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:05.537+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:05.581 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:05.537+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:06.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:06.170+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:06.171752+0000) 2024-02-19T20:39:06.574 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:39:06.541+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67148 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:39:06.574 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:06.395+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:06.396424+0000) 2024-02-19T20:39:06.574 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:06.537+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:06.538779+0000) 2024-02-19T20:39:06.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:06.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:06.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:06.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:06.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:06.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:06.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:06.332+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:06.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:06.538+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:06.539451+0000) 2024-02-19T20:39:06.576 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:06.480+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:06.481361+0000) 2024-02-19T20:39:06.576 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:06.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:06.576 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:06.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:06.576 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:06.519+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:06.576 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:06.520+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:06.576 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:06.537+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:06.538502+0000) 2024-02-19T20:39:06.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:06.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:06.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:06.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:06.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:06.588+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:06.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:06.589+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:07.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:07.171+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:07.172444+0000) 2024-02-19T20:39:07.581 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:07.396+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:07.397072+0000) 2024-02-19T20:39:07.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:07.299+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:07.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:07.299+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:07.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:07.299+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:07.583 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:07.300+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:07.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:07.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:07.583 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:07.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:07.584 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:07.472+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:07.584 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:07.472+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:07.584 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:07.481+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:07.482232+0000) 2024-02-19T20:39:07.708 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:39:07.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:07.595+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:07.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:07.595+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:07.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:07.595+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:07.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:07.596+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:08.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:08.174+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:08.175964+0000) 2024-02-19T20:39:08.584 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:08.398+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:08.399224+0000) 2024-02-19T20:39:08.584 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:08.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:08.584 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:08.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:08.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:08.474+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:08.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:08.475+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:08.585 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:08.483+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:08.484193+0000) 2024-02-19T20:39:08.585 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:08.318+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:08.586 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:08.318+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:08.586 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:08.318+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:08.586 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:08.319+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:08.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:08.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:08.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:08.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:08.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:08.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:08.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:08.636+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:09.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:09.177+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:09.178386+0000) 2024-02-19T20:39:09.632 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:09.401+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:09.401264+0000) 2024-02-19T20:39:09.632 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:09.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:09.632 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:09.352+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:09.632 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:09.352+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:09.633 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:09.352+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:09.633 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:09.352+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:09.633 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:09.481+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:09.633 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:09.481+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:09.633 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:09.481+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:09.633 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:09.482+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:09.633 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:09.485+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:09.486220+0000) 2024-02-19T20:39:09.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:09.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:09.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:09.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:09.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:09.646+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:10.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:10.179+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:10.181233+0000) 2024-02-19T20:39:10.672 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:10.402+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:10.404019+0000) 2024-02-19T20:39:10.672 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:10.362+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:10.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:10.362+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:10.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:10.362+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:10.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:10.363+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:10.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:10.487+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:10.488430+0000) 2024-02-19T20:39:10.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:10.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:10.673 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:10.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:10.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:10.502+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:10.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:10.502+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:10.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:39:10.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:39:10.780 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:39:10.780 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:39:10.781 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-19T20:39:10.781 DEBUG:teuthology.orchestra.run.smithi114:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-19T20:39:10.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:39:10.797 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:39:10.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:10.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:10.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:10.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:10.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:10.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:10.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:10.686+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:11.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:11.182+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:11.183366+0000) 2024-02-19T20:39:11.626 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:39:11.550+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67189 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:39:11.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:11.405+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:11.406473+0000) 2024-02-19T20:39:11.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:11.546+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:11.547154+0000) 2024-02-19T20:39:11.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:11.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:11.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:11.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:11.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:11.356+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:11.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:11.357+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:11.628 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:11.546+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:11.547893+0000) 2024-02-19T20:39:11.628 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:11.489+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:11.490754+0000) 2024-02-19T20:39:11.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:11.517+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:11.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:11.517+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:11.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:11.517+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:11.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:11.517+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:11.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:11.546+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:11.547862+0000) 2024-02-19T20:39:11.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:11.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:11.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:11.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:11.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:11.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:11.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:11.642+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:12.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:12.183+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:12.184451+0000) 2024-02-19T20:39:12.586 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:12.406+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:12.407553+0000) 2024-02-19T20:39:12.587 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:12.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:12.587 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:12.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:12.587 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:12.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:12.587 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:12.331+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:12.588 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:12.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:12.588 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:12.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:12.588 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:12.468+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:12.588 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:12.468+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:12.588 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:12.490+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:12.491433+0000) 2024-02-19T20:39:12.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:12.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:12.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:12.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:12.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:12.624+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:12.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:12.625+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:13.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:13.185+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:13.186942+0000) 2024-02-19T20:39:13.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:13.409+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:13.410941+0000) 2024-02-19T20:39:13.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:13.288+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:13.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:13.288+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:13.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:13.288+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:13.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:13.289+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:13.567 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:13.492+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:13.493439+0000) 2024-02-19T20:39:13.567 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:13.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:13.567 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:13.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:13.568 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:13.511+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:13.568 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:13.512+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:13.860 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:39:13.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:13.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:13.869 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:13.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:13.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:13.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:13.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:13.618+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:14.461 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:14.412+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:14.413002+0000) 2024-02-19T20:39:14.462 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:14.187+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:14.188953+0000) 2024-02-19T20:39:14.462 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:14.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:14.462 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:14.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:14.462 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:14.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:14.462 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:14.256+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:14.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:14.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:14.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:14.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:14.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:14.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:14.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:14.641+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:14.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:14.494+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:14.495463+0000) 2024-02-19T20:39:14.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:14.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:14.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:14.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:14.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:14.494+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:14.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:14.495+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:15.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:15.190+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:15.191462+0000) 2024-02-19T20:39:15.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:15.212+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:15.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:15.212+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:15.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:15.212+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:15.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:15.212+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:15.647 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:15.414+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:15.415502+0000) 2024-02-19T20:39:15.648 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:15.461+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:15.648 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:15.461+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:15.648 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:15.461+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:15.648 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:15.461+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:15.648 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:15.496+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:15.497963+0000) 2024-02-19T20:39:15.974 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:15.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:15.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:15.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:15.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:15.662+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:15.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:15.662+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:16.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:16.164+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:16.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:16.164+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:16.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:16.164+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:16.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:16.164+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:16.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:16.192+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:16.193908+0000) 2024-02-19T20:39:16.663 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:39:16.558+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67229 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:39:16.664 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:16.555+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:16.556376+0000) 2024-02-19T20:39:16.664 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:16.416+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:16.417654+0000) 2024-02-19T20:39:16.665 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:16.554+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:16.555606+0000) 2024-02-19T20:39:16.665 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:16.613+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:16.665 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:16.613+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:16.665 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:16.613+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:16.665 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:16.614+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:16.665 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:16.417+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:16.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:16.417+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:16.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:16.417+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:16.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:16.418+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:16.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:16.499+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:16.500570+0000) 2024-02-19T20:39:16.666 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:16.554+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:16.555949+0000) 2024-02-19T20:39:17.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:17.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:17.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:17.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:17.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:17.136+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:17.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:17.137+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:17.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:17.193+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:17.194617+0000) 2024-02-19T20:39:17.650 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:17.417+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:17.418476+0000) 2024-02-19T20:39:17.650 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:17.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:17.650 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:17.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:17.650 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:17.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:17.651 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:17.409+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:17.651 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:17.409+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:17.651 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:17.409+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:17.651 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:17.409+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:17.651 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:17.500+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:17.501403+0000) 2024-02-19T20:39:17.975 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:17.665+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:18.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:18.186+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:18.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:18.186+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:18.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:18.186+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:18.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:18.187+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:18.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:18.195+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:18.196646+0000) 2024-02-19T20:39:18.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:18.420+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:18.421018+0000) 2024-02-19T20:39:18.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:18.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:18.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:18.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:18.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:18.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:18.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:18.640+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:18.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:18.451+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:18.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:18.451+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:18.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:18.451+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:18.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:18.452+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:18.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:18.502+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:18.503387+0000) 2024-02-19T20:39:19.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:19.198+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:19.199179+0000) 2024-02-19T20:39:19.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:19.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:19.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:19.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:19.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:19.219+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:19.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:19.220+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:19.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:19.422+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:19.422997+0000) 2024-02-19T20:39:19.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:19.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:19.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:19.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:19.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:19.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:19.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:19.617+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:19.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:19.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:19.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:19.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:19.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:19.484+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:19.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:19.485+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:19.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:19.504+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:19.505617+0000) 2024-02-19T20:39:20.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:20.201+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:20.202738+0000) 2024-02-19T20:39:20.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:20.231+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:20.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:20.231+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:20.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:20.231+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:20.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:20.232+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:20.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:20.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:20.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:20.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:20.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:20.498+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:20.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:20.499+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:20.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:20.506+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:20.507829+0000) 2024-02-19T20:39:20.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:20.424+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:20.425073+0000) 2024-02-19T20:39:20.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:20.642+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:20.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:20.642+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:20.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:20.642+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:20.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:20.643+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:21.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:21.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:21.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:21.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:21.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:21.205+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:21.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:21.205+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:21.206265+0000) 2024-02-19T20:39:21.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:21.206+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:21.674 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:39:21.566+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67270 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:39:21.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:21.563+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:21.564723+0000) 2024-02-19T20:39:21.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:21.426+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:21.427114+0000) 2024-02-19T20:39:21.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:21.562+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:21.563431+0000) 2024-02-19T20:39:21.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:21.596+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:21.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:21.596+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:21.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:21.596+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:21.675 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:21.597+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:21.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:21.488+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:21.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:21.488+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:21.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:21.488+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:21.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:21.489+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:21.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:21.509+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:21.510312+0000) 2024-02-19T20:39:21.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:21.562+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:21.563510+0000) 2024-02-19T20:39:22.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:22.206+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:22.207226+0000) 2024-02-19T20:39:22.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:22.218+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:22.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:22.218+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:22.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:22.218+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:22.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:22.220+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:22.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:22.426+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:22.427789+0000) 2024-02-19T20:39:22.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:22.589+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:22.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:22.589+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:22.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:22.589+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:22.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:22.589+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:22.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:22.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:22.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:22.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:22.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:22.454+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:22.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:22.455+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:22.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:22.510+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:22.511234+0000) 2024-02-19T20:39:22.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:39:22.861 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:39:23.462 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:23.428+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:23.429743+0000) 2024-02-19T20:39:23.462 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:23.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:23.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:23.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:23.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:23.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:23.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:23.185+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:23.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:23.208+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:23.209758+0000) 2024-02-19T20:39:23.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:23.547+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:23.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:23.547+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:23.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:23.547+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:23.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:23.548+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:23.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:23.501+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:23.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:23.501+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:23.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:23.501+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:23.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:23.502+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:23.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:23.513+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:23.514552+0000) 2024-02-19T20:39:24.438 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:24.430+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:24.431798+0000) 2024-02-19T20:39:24.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:24.195+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:24.439 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:24.195+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:24.439 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:24.195+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:24.439 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:24.196+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:24.439 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:24.210+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:24.211765+0000) 2024-02-19T20:39:24.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:24.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:24.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:24.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:24.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:24.573+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:24.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:24.574+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:24.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:24.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:24.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:24.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:24.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:24.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:24.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:24.453+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:24.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:24.515+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:24.516571+0000) 2024-02-19T20:39:25.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:25.189+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:25.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:25.189+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:25.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:25.189+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:25.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:25.190+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:25.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:25.214+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:25.215333+0000) 2024-02-19T20:39:25.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:25.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:25.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:25.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:25.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:25.452+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:25.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:25.453+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:25.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:25.519+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:25.520096+0000) 2024-02-19T20:39:25.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:25.433+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:25.434004+0000) 2024-02-19T20:39:25.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:25.576+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:25.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:25.576+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:25.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:25.576+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:25.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:25.577+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:25.920 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:39:26.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:26.214+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:26.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:26.214+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:26.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:26.214+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:26.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:26.215+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:26.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:26.217+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:26.218747+0000) 2024-02-19T20:39:26.507 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:26.435+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:26.436021+0000) 2024-02-19T20:39:26.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:26.450+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:26.507 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:26.450+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:26.508 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:26.450+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:26.508 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:26.450+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:26.976 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:39:26.574+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67310 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:39:26.976 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:26.571+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:26.572287+0000) 2024-02-19T20:39:26.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:26.521+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:26.522488+0000) 2024-02-19T20:39:26.977 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:26.570+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:26.571592+0000) 2024-02-19T20:39:26.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:26.571+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:26.572333+0000) 2024-02-19T20:39:26.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:26.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:26.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:26.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:26.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:26.600+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:26.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:26.601+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:27.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:27.218+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:27.219756+0000) 2024-02-19T20:39:27.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:27.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:27.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:27.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:27.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:27.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:27.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:27.226+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:27.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:27.435+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:27.436740+0000) 2024-02-19T20:39:27.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:27.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:27.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:27.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:27.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:27.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:27.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:27.649+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:27.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:27.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:27.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:27.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:27.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:27.445+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:27.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:27.446+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:27.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:27.522+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:27.523389+0000) 2024-02-19T20:39:28.462 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:28.439+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:28.440237+0000) 2024-02-19T20:39:28.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:28.220+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:28.221912+0000) 2024-02-19T20:39:28.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:28.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:28.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:28.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:28.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:28.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:28.463 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:28.259+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:28.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:28.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:28.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:28.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:28.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:28.631+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:28.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:28.632+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:28.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:28.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:28.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:28.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:28.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:28.489+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:28.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:28.490+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:28.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:28.526+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:28.527117+0000) 2024-02-19T20:39:29.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:29.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:29.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:29.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:29.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:29.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:29.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:29.214+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:29.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:29.223+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:29.224293+0000) 2024-02-19T20:39:29.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:29.441+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:29.442334+0000) 2024-02-19T20:39:29.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:29.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:29.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:29.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:29.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:29.635+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:29.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:29.635+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:29.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:29.529+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:29.530519+0000) 2024-02-19T20:39:29.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:29.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:29.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:29.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:29.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:29.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:29.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:29.537+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:30.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:30.226+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:30.227709+0000) 2024-02-19T20:39:30.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:30.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:30.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:30.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:30.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:30.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:30.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:30.240+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:30.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:30.444+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:30.445874+0000) 2024-02-19T20:39:30.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:30.612+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:30.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:30.612+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:30.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:30.612+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:30.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:30.613+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:30.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:30.532+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:30.533914+0000) 2024-02-19T20:39:30.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:30.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:30.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:30.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:30.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:30.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:30.757 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:30.559+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:31.226 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:31.228+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:31.229889+0000) 2024-02-19T20:39:31.227 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:31.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:31.227 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:31.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:31.227 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:31.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:31.227 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:31.240+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:31.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:31.447+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:31.447975+0000) 2024-02-19T20:39:31.976 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:39:31.582+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67351 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:39:31.977 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:31.578+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:31.579745+0000) 2024-02-19T20:39:31.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:31.578+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:31.579821+0000) 2024-02-19T20:39:31.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:31.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:31.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:31.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:31.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:31.648+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:31.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:31.649+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:31.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:31.536+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:31.537361+0000) 2024-02-19T20:39:31.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:31.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:31.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:31.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:31.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:31.546+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:31.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:31.546+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:31.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:31.578+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:31.579605+0000) 2024-02-19T20:39:32.240 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:32.229+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:32.230549+0000) 2024-02-19T20:39:32.241 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:32.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:32.241 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:32.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:32.241 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:32.255+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:32.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:32.255+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:32.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:32.447+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:32.448641+0000) 2024-02-19T20:39:32.946 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:32.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:32.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:32.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:32.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:32.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:32.947 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:32.607+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:32.947 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:32.523+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:32.947 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:32.523+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:32.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:32.523+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:32.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:32.524+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:32.948 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:32.537+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:32.538501+0000) 2024-02-19T20:39:33.463 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:33.450+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:33.451928+0000) 2024-02-19T20:39:33.464 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:33.231+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:33.232631+0000) 2024-02-19T20:39:33.464 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:33.261+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:33.464 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:33.261+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:33.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:33.261+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:33.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:33.263+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:33.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:33.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:33.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:33.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:33.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:33.528+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:33.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:33.529+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:33.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:33.540+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:33.541901+0000) 2024-02-19T20:39:33.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:33.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:33.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:33.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:33.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:33.623+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:33.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:33.624+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:34.438 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:34.234+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:34.235149+0000) 2024-02-19T20:39:34.439 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:34.286+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:34.439 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:34.286+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:34.439 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:34.286+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:34.440 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:34.288+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:34.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:34.452+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:34.453936+0000) 2024-02-19T20:39:34.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:34.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:34.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:34.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:34.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:34.636+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:34.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:34.637+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:34.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:34.543+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:34.544635+0000) 2024-02-19T20:39:34.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:34.579+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:34.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:34.579+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:34.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:34.579+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:34.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:34.579+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:35.231 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:35.236+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:35.237592+0000) 2024-02-19T20:39:35.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:35.246+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:35.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:35.246+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:35.232 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:35.246+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:35.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:35.247+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:35.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:35.456+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:35.457384+0000) 2024-02-19T20:39:35.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:35.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:35.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:35.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:35.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:35.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:35.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:35.640+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:35.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:35.546+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:35.547174+0000) 2024-02-19T20:39:35.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:35.617+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:35.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:35.617+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:35.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:35.617+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:35.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:35.617+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:36.236 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:36.238+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:36.239640+0000) 2024-02-19T20:39:36.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:36.458+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:36.459454+0000) 2024-02-19T20:39:36.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:36.250+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:36.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:36.250+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:36.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:36.250+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:36.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:36.251+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:36.977 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:39:36.590+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67391 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:39:36.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:36.587+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:36.588285+0000) 2024-02-19T20:39:36.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:36.586+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:36.587047+0000) 2024-02-19T20:39:36.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:36.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:36.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:36.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:36.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:36.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:36.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:36.617+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:36.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:36.548+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:36.549179+0000) 2024-02-19T20:39:36.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:36.586+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:36.587824+0000) 2024-02-19T20:39:36.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:36.612+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:36.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:36.612+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:36.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:36.612+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:36.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:36.612+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:37.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:37.239+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:37.240464+0000) 2024-02-19T20:39:37.534 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:37.459+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:37.460268+0000) 2024-02-19T20:39:37.535 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:37.280+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:37.535 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:37.280+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:37.535 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:37.280+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:37.535 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:37.281+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:37.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:39:37.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:39:37.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:37.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:37.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:37.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:37.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:37.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:37.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:37.646+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:37.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:37.549+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:37.550025+0000) 2024-02-19T20:39:37.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:37.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:37.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:37.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:37.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:37.575+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:37.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:37.575+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:38.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:38.461+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:38.462630+0000) 2024-02-19T20:39:38.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:38.243+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:38.244048+0000) 2024-02-19T20:39:38.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:38.303+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:38.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:38.303+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:38.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:38.303+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:38.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:38.304+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:38.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:38.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:38.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:38.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:38.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:38.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:38.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:38.671+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:38.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:38.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:38.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:38.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:38.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:38.540+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:38.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:38.541+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:38.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:38.551+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:38.552156+0000) 2024-02-19T20:39:39.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:39.463+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:39.464603+0000) 2024-02-19T20:39:39.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:39.245+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:39.246156+0000) 2024-02-19T20:39:39.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:39.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:39.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:39.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:39.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:39.290+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:39.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:39.290+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:39.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:39.642+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:39.977 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:39.642+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:39.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:39.642+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:39.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:39.643+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:39.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:39.553+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:39.554647+0000) 2024-02-19T20:39:39.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:39.561+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:39.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:39.561+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:39.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:39.561+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:39.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:39.561+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:40.500 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:40.465+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:40.466708+0000) 2024-02-19T20:39:40.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:40.247+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:40.248307+0000) 2024-02-19T20:39:40.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:40.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:40.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:40.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:40.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:40.270+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:40.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:40.271+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:40.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:40.620+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:40.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:40.621+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:40.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:40.621+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:40.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:40.621+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:40.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:40.514+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:40.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:40.514+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:40.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:40.514+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:40.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:40.515+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:40.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:40.555+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:40.556707+0000) 2024-02-19T20:39:40.992 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:39:41.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:41.467+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:41.468927+0000) 2024-02-19T20:39:41.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:41.249+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:41.250720+0000) 2024-02-19T20:39:41.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:41.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:41.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:41.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:41.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:41.259+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:41.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:41.260+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:41.977 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:39:41.598+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67433 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:39:41.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:41.594+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:41.595421+0000) 2024-02-19T20:39:41.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:41.594+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:41.595780+0000) 2024-02-19T20:39:41.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:41.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:41.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:41.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:41.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:41.645+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:41.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:41.646+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:41.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:41.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:41.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:41.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:41.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:41.536+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:41.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:41.536+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:41.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:41.557+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:41.558893+0000) 2024-02-19T20:39:41.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:41.594+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:41.595321+0000) 2024-02-19T20:39:42.236 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:42.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:42.236 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:42.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:42.236 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:42.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:42.237 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:42.222+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:42.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:42.250+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:42.251658+0000) 2024-02-19T20:39:42.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:42.469+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:42.470027+0000) 2024-02-19T20:39:42.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:42.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:42.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:42.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:42.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:42.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:42.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:42.661+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:42.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:42.558+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:42.559595+0000) 2024-02-19T20:39:42.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:42.563+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:42.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:42.563+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:42.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:42.563+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:42.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:42.563+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:43.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:43.471+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:43.472565+0000) 2024-02-19T20:39:43.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:43.253+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:43.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:43.253+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:43.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:43.253+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:43.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:43.253+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:43.254898+0000) 2024-02-19T20:39:43.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:43.254+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:43.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:43.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:43.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:43.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:43.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:43.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:43.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:43.665+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:43.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:43.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:43.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:43.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:43.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:43.559+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:43.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:43.560+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:43.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:43.561+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:43.562118+0000) 2024-02-19T20:39:43.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:39:43.994 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:39:44.060 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:39:44.060 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:39:44.061 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-19T20:39:44.063 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566627/remote/smithi114/crash 2024-02-19T20:39:44.064 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/crash -- . 2024-02-19T20:39:44.091 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566627/remote/smithi174/crash 2024-02-19T20:39:44.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:39:44.092 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:39:44.459 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:44.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:44.460 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:44.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:44.460 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:44.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:44.460 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:44.238+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:44.460 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:44.257+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:44.258080+0000) 2024-02-19T20:39:44.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:44.474+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:44.475193+0000) 2024-02-19T20:39:44.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:44.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:44.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:44.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:44.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:44.649+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:44.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:44.649+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:44.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:44.564+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:44.565712+0000) 2024-02-19T20:39:44.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:44.605+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:44.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:44.605+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:44.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:44.605+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:44.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:44.606+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:45.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:45.476+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:45.477346+0000) 2024-02-19T20:39:45.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:45.243+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:45.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:45.243+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:45.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:45.243+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:45.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:45.244+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:45.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:45.259+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:45.260201+0000) 2024-02-19T20:39:45.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:45.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:45.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:45.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:45.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:45.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:45.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:45.672+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:45.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:45.568+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:45.569326+0000) 2024-02-19T20:39:45.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:45.654+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:45.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:45.654+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:45.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:45.654+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:45.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:45.654+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:46.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:46.478+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:46.479422+0000) 2024-02-19T20:39:46.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:46.261+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:46.262205+0000) 2024-02-19T20:39:46.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:46.263+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:46.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:46.263+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:46.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:46.263+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:46.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:46.263+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:46.978 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:39:46.607+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67473 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:39:46.978 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:46.603+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:46.604674+0000) 2024-02-19T20:39:46.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:46.603+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:46.604370+0000) 2024-02-19T20:39:46.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:46.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:46.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:46.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:46.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:46.688+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:46.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:46.689+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:46.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:46.570+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:46.571519+0000) 2024-02-19T20:39:46.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:46.602+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:46.603696+0000) 2024-02-19T20:39:46.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:46.611+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:46.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:46.611+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:46.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:46.611+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:46.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:46.612+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:47.164 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:39:47.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:47.479+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:47.480352+0000) 2024-02-19T20:39:47.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:47.262+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:47.262999+0000) 2024-02-19T20:39:47.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:47.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:47.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:47.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:47.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:47.264+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:47.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:47.265+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:47.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:47.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:47.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:47.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:47.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:47.720+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:47.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:47.721+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:47.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:47.571+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:47.572391+0000) 2024-02-19T20:39:47.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:47.655+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:47.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:47.655+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:47.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:47.655+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:47.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:47.655+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:48.250 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:48.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:48.250 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:48.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:48.250 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:48.241+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:48.250 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:48.242+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:48.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:48.264+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:48.265484+0000) 2024-02-19T20:39:48.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:48.482+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:48.483873+0000) 2024-02-19T20:39:48.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:48.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:48.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:48.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:48.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:48.685+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:48.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:48.685+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:48.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:48.573+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:48.574485+0000) 2024-02-19T20:39:48.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:48.650+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:48.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:48.650+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:48.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:48.650+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:48.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:48.650+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:49.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:49.485+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:49.486442+0000) 2024-02-19T20:39:49.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:49.261+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:49.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:49.261+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:49.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:49.261+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:49.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:49.262+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:49.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:49.266+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:49.267858+0000) 2024-02-19T20:39:49.978 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:49.707+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:49.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:49.707+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:49.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:49.707+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:49.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:49.707+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:49.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:49.575+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:49.576440+0000) 2024-02-19T20:39:49.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:49.632+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:49.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:49.632+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:49.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:49.632+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:49.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:49.632+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:50.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:50.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:50.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:50.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:50.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:50.237+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:50.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:50.238+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:50.563 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:50.270+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:50.271315+0000) 2024-02-19T20:39:50.563 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:50.487+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:50.488650+0000) 2024-02-19T20:39:50.978 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:50.577+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:50.578446+0000) 2024-02-19T20:39:50.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:50.601+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:50.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:50.601+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:50.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:50.601+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:50.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:50.601+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:50.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:50.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:50.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:50.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:50.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:50.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:50.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:50.726+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:51.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:51.218+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:51.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:51.218+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:51.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:51.218+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:51.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:51.219+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:51.565 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:51.272+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:51.273939+0000) 2024-02-19T20:39:51.565 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:51.489+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:51.490816+0000) 2024-02-19T20:39:51.978 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:39:51.615+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67514 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:39:51.979 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:51.611+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:51.612632+0000) 2024-02-19T20:39:51.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:51.611+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:51.612376+0000) 2024-02-19T20:39:51.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:51.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:51.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:51.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:51.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:51.712+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:51.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:51.713+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:51.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:51.579+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:51.580714+0000) 2024-02-19T20:39:51.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:51.611+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:51.612238+0000) 2024-02-19T20:39:51.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:51.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:51.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:51.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:51.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:51.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:51.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:51.623+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:52.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:52.235+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:52.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:52.235+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:52.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:52.235+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:52.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:52.236+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:52.562 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:52.273+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:52.274741+0000) 2024-02-19T20:39:52.563 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:52.490+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:52.491634+0000) 2024-02-19T20:39:52.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:52.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:52.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:52.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:52.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:52.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:52.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:52.699+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:52.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:52.576+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:52.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:52.577+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:52.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:52.577+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:52.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:52.577+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:52.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:52.580+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:52.581540+0000) 2024-02-19T20:39:53.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:53.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:53.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:53.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:53.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:53.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:53.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:53.204+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:53.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:53.493+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:53.494894+0000) 2024-02-19T20:39:53.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:53.275+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:53.276844+0000) 2024-02-19T20:39:53.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:53.659+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:53.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:53.659+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:53.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:53.659+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:53.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:53.660+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:53.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:53.584+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:53.585129+0000) 2024-02-19T20:39:53.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:53.609+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:53.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:53.609+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:53.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:53.609+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:53.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:53.610+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:54.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:54.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:54.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:54.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:54.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:54.203+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:54.465 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:54.204+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:54.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:54.277+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:54.278861+0000) 2024-02-19T20:39:54.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:54.497+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:54.498432+0000) 2024-02-19T20:39:54.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:54.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:54.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:54.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:54.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:54.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:54.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:54.642+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:54.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:54.586+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:54.587252+0000) 2024-02-19T20:39:54.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:54.648+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:54.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:54.648+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:54.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:54.648+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:54.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:54.649+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:55.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:55.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:55.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:55.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:55.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:55.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:55.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:55.211+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:55.575 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:55.280+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:55.281015+0000) 2024-02-19T20:39:55.575 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:55.500+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:55.501022+0000) 2024-02-19T20:39:55.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:55.589+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:55.590842+0000) 2024-02-19T20:39:55.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:55.648+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:55.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:55.648+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:55.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:55.648+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:55.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:55.649+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:55.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:55.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:55.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:55.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:55.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:55.640+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:55.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:55.641+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:56.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:39:56.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:39:56.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:56.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:56.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:56.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:56.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:56.200+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:56.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:56.201+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:56.578 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:56.282+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:56.283033+0000) 2024-02-19T20:39:56.579 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:56.502+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:56.503026+0000) 2024-02-19T20:39:56.979 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:39:56.623+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67554 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:39:56.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:56.620+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:56.621109+0000) 2024-02-19T20:39:56.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:56.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:56.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:56.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:56.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:56.617+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:56.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:56.618+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:56.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:56.620+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:56.621365+0000) 2024-02-19T20:39:56.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:56.593+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:56.594184+0000) 2024-02-19T20:39:56.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:56.619+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:56.620243+0000) 2024-02-19T20:39:56.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:56.693+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:56.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:56.693+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:56.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:56.693+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:56.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:56.694+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:57.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:57.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:57.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:57.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:57.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:57.239+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:57.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:57.240+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:57.579 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:57.283+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:57.284119+0000) 2024-02-19T20:39:57.580 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:57.502+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:57.503755+0000) 2024-02-19T20:39:57.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:57.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:57.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:57.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:57.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:57.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:57.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:57.634+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:57.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:57.594+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:57.595100+0000) 2024-02-19T20:39:57.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:57.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:57.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:57.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:57.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:57.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:57.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:57.666+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:58.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:58.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:58.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:58.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:58.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:58.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:58.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:58.227+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:58.582 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:58.286+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:58.287488+0000) 2024-02-19T20:39:58.582 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:58.505+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:58.507023+0000) 2024-02-19T20:39:58.979 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:58.677+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:58.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:58.677+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:58.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:58.677+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:58.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:58.677+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:58.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:58.595+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:58.597790+0000) 2024-02-19T20:39:58.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:58.652+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:58.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:58.652+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:58.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:58.652+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:58.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:58.652+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:59.228 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:39:59.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:59.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:59.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:59.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:59.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:59.226+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:39:59.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:59.227+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:39:59.584 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:39:59.288+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:59.289785+0000) 2024-02-19T20:39:59.585 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:59.508+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:59.509102+0000) 2024-02-19T20:39:59.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:59.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:59.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:59.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:59.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:59.726+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:39:59.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:39:59.727+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:39:59.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:59.599+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:59.600219+0000) 2024-02-19T20:39:59.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:59.701+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:59.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:59.701+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:59.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:59.701+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:39:59.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:39:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:39:59.702+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:00.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:00.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:00.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:00.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:00.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:00.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:00.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:00.217+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:00.587 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:00.291+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:00.292306+0000) 2024-02-19T20:40:00.587 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:00.510+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:00.511721+0000) 2024-02-19T20:40:00.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:00.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:00.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:00.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:00.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:00.693+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:00.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:00.694+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:00.980 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:00.601+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:00.602731+0000) 2024-02-19T20:40:00.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:00.740+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:00.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:00.740+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:00.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:00.740+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:00.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:00.741+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:01.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:01.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:01.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:01.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:01.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:01.197+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:01.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:01.198+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:01.589 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:01.294+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:01.295906+0000) 2024-02-19T20:40:01.590 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:01.513+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:01.514240+0000) 2024-02-19T20:40:01.980 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:40:01.632+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67595 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:40:01.980 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:01.627+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:01.628917+0000) 2024-02-19T20:40:01.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:01.627+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:01.628570+0000) 2024-02-19T20:40:01.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:01.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:01.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:01.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:01.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:01.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:01.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:01.700+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:01.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:01.604+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:01.605496+0000) 2024-02-19T20:40:01.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:01.628+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:01.629313+0000) 2024-02-19T20:40:01.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:01.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:01.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:01.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:01.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:01.761+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:01.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:01.762+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:02.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:02.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:02.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:02.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:02.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:02.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:02.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:02.228+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:02.590 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:02.295+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:02.296689+0000) 2024-02-19T20:40:02.591 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:02.514+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:02.515000+0000) 2024-02-19T20:40:02.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:02.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:02.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:02.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:02.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:02.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:02.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:02.702+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:02.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:02.605+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:02.606410+0000) 2024-02-19T20:40:02.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:02.740+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:02.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:02.740+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:02.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:02.740+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:02.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:02.740+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:03.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:03.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:03.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:03.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:03.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:03.184+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:03.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:03.185+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:03.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:03.299+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:03.300171+0000) 2024-02-19T20:40:03.566 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:03.516+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:03.517078+0000) 2024-02-19T20:40:03.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:03.738+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:03.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:03.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:03.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:03.739+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:03.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:03.739+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:03.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:03.608+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:03.609853+0000) 2024-02-19T20:40:03.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:03.745+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:03.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:03.745+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:03.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:03.745+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:03.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:03.745+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:04.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:04.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:04.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:04.211+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:04.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:04.302+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:04.303138+0000) 2024-02-19T20:40:04.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:04.611+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:04.612150+0000) 2024-02-19T20:40:04.757 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:04.518+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:04.519469+0000) 2024-02-19T20:40:05.200 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:04.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:05.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:04.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:05.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:04.768+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:05.201 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:04.769+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:05.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:04.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:05.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:04.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:05.201 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:04.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:05.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:04.783+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:05.466 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:05.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:05.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:05.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:05.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:05.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:05.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:05.225+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:05.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:05.304+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:05.305310+0000) 2024-02-19T20:40:05.726 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:05.520+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:05.521824+0000) 2024-02-19T20:40:05.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:05.613+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:05.614469+0000) 2024-02-19T20:40:05.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:05.740+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:05.726 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:05.740+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:05.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:05.740+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:05.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:05.741+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:05.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:05.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:05.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:05.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:05.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:05.749+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:05.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:05.750+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:06.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:06.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:06.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:06.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:06.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:06.236+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:06.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:06.237+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:06.601 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:06.306+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:06.307790+0000) 2024-02-19T20:40:06.601 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:06.523+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:06.524389+0000) 2024-02-19T20:40:06.980 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:40:06.642+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67635 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:40:06.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:06.637+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:06.638672+0000) 2024-02-19T20:40:06.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:06.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:06.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:06.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:06.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:06.700+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:06.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:06.701+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:06.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:06.615+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:06.616701+0000) 2024-02-19T20:40:06.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:06.637+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:06.638301+0000) 2024-02-19T20:40:06.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:06.768+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:06.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:06.768+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:06.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:06.768+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:06.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:06.769+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:06.982 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:06.637+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:06.638038+0000) 2024-02-19T20:40:07.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:07.194+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:07.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:07.194+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:07.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:07.194+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:07.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:07.195+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:07.602 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:07.307+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:07.308559+0000) 2024-02-19T20:40:07.602 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:07.524+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:07.525709+0000) 2024-02-19T20:40:07.980 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:07.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:07.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:07.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:07.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:07.701+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:07.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:07.702+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:07.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:07.616+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:07.617676+0000) 2024-02-19T20:40:07.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:07.779+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:07.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:07.779+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:07.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:07.779+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:07.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:07.779+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:08.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:08.244+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:08.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:08.244+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:08.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:08.244+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:08.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:08.245+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:08.604 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:08.526+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:08.527813+0000) 2024-02-19T20:40:08.605 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:08.309+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:08.310532+0000) 2024-02-19T20:40:08.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:08.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:08.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:08.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:08.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:08.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:08.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:08.696+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:08.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:08.619+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:08.619970+0000) 2024-02-19T20:40:08.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:08.749+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:08.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:08.749+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:08.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:08.749+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:08.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:08.749+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:09.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:09.267+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:09.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:09.267+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:09.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:09.267+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:09.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:09.268+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:09.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:09.311+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:09.312731+0000) 2024-02-19T20:40:09.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:09.621+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:09.622516+0000) 2024-02-19T20:40:09.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:09.748+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:09.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:09.748+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:09.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:09.748+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:09.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:09.748+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:09.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:09.530+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:09.531075+0000) 2024-02-19T20:40:09.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:09.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:09.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:09.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:09.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:09.733+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:09.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:09.734+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:10.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:10.240+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:10.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:10.240+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:10.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:10.240+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:10.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:10.241+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:10.609 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:10.314+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:10.315257+0000) 2024-02-19T20:40:10.609 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:10.533+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:10.534448+0000) 2024-02-19T20:40:10.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:10.623+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:10.624530+0000) 2024-02-19T20:40:10.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:10.753+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:10.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:10.753+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:10.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:10.753+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:10.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:10.753+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:10.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:10.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:10.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:10.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:10.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:10.784+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:10.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:10.785+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:11.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:40:11.230 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:40:11.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:11.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:11.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:11.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:11.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:11.222+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:11.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:11.224+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:11.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:11.316+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:11.317927+0000) 2024-02-19T20:40:11.611 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:11.536+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:11.537014+0000) 2024-02-19T20:40:11.981 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:40:11.650+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67676 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:40:11.981 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:11.646+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:11.647170+0000) 2024-02-19T20:40:11.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:11.625+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:11.626619+0000) 2024-02-19T20:40:11.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:11.646+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:11.647550+0000) 2024-02-19T20:40:11.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:11.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:11.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:11.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:11.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:11.751+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:11.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:11.751+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:11.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:11.645+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:11.646946+0000) 2024-02-19T20:40:11.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:11.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:11.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:11.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:11.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:11.809+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:11.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:11.809+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:12.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:12.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:12.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:12.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:12.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:12.216+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:12.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:12.217+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:12.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:12.317+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:12.318689+0000) 2024-02-19T20:40:12.612 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:12.537+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:12.538204+0000) 2024-02-19T20:40:12.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:12.626+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:12.627834+0000) 2024-02-19T20:40:12.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:12.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:12.981 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:12.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:12.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:12.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:12.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:12.789+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:12.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:12.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:12.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:12.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:12.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:12.763+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:12.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:12.764+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:13.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:13.188+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:13.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:13.188+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:13.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:13.188+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:13.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:13.188+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:13.567 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:13.320+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:13.321925+0000) 2024-02-19T20:40:13.567 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:13.539+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:13.540226+0000) 2024-02-19T20:40:13.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:13.788+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:13.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:13.788+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:13.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:13.788+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:13.873 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:13.788+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:13.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:13.629+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:13.630029+0000) 2024-02-19T20:40:13.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:13.741+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:13.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:13.741+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:13.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:13.741+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:13.873 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:13.741+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:14.300 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:40:14.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:14.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:14.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:14.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:14.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:14.193+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:14.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:14.193+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:14.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:14.323+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:14.324217+0000) 2024-02-19T20:40:14.735 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:14.541+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:14.542366+0000) 2024-02-19T20:40:14.736 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:14.631+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:14.632006+0000) 2024-02-19T20:40:15.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:14.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:15.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:14.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:15.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:14.750+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:15.009 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:14.751+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:15.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:14.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:15.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:14.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:15.009 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:14.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:15.010 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:14.763+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:15.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:15.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:15.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:15.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:15.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:15.156+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:15.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:15.157+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:15.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:15.325+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:15.326739+0000) 2024-02-19T20:40:15.740 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:15.544+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:15.545704+0000) 2024-02-19T20:40:15.786 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:15.633+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:15.634389+0000) 2024-02-19T20:40:16.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:15.755+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:16.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:15.755+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:16.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:15.755+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:16.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:15.756+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:16.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:15.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:16.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:15.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:16.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:15.763+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:16.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:15.763+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:16.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:16.196+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:16.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:16.196+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:16.467 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:16.196+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:16.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:16.197+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:16.468 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:16.329+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:16.330028+0000) 2024-02-19T20:40:16.726 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:40:16.658+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67716 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:40:16.727 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:16.546+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:16.547811+0000) 2024-02-19T20:40:16.727 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:16.653+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:16.654844+0000) 2024-02-19T20:40:16.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:16.635+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:16.636538+0000) 2024-02-19T20:40:16.727 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:16.655+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:16.656339+0000) 2024-02-19T20:40:16.728 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:16.654+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:16.655872+0000) 2024-02-19T20:40:16.981 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:16.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:16.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:16.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:16.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:16.741+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:16.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:16.742+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:16.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:16.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:16.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:16.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:16.982 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:16.782+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:16.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:16.783+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:17.256 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:17.187+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:17.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:17.187+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:17.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:17.187+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:17.257 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:17.187+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:17.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:40:17.302 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:40:17.376 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:40:17.376 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:40:17.376 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-19T20:40:17.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-19T20:40:17.410 INFO:tasks.cephadm:Compressing logs... 2024-02-19T20:40:17.410 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-19T20:40:17.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:40:17.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:40:17.476 INFO:teuthology.orchestra.run.smithi114.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-19T20:40:17.477 INFO:teuthology.orchestra.run.smithi114.stderr:: No such file or directory 2024-02-19T20:40:17.477 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi114.vpoeop.log 2024-02-19T20:40:17.477 INFO:teuthology.orchestra.run.smithi114.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi114.tcwpod.log 2024-02-19T20:40:17.479 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph.audit.log 2024-02-19T20:40:17.479 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi114.vpoeop.log: /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi114.tcwpod.log: gzip -5 --verbose -- /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-osd.1.log 2024-02-19T20:40:17.479 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-client.ceph-exporter.smithi114.log 2024-02-19T20:40:17.479 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-mon.smithi114.log 2024-02-19T20:40:17.480 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-client.ceph-exporter.smithi114.log: gzip -5 --verbose -- /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-osd.0.log 2024-02-19T20:40:17.481 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-mon.smithi114.log: 91.3% -- replaced with /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph.audit.log.gz 2024-02-19T20:40:17.487 INFO:teuthology.orchestra.run.smithi114.stderr: 94.7%gzip -5 --verbose -- /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph.cephadm.log 2024-02-19T20:40:17.490 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-osd.0.log: -- replaced with /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-client.ceph-exporter.smithi114.log.gz 2024-02-19T20:40:17.490 INFO:teuthology.orchestra.run.smithi114.stderr: 89.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-19T20:40:17.491 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph.log 2024-02-19T20:40:17.492 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph.cephadm.log: 85.6% -- replaced with /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph.cephadm.log.gz 2024-02-19T20:40:17.492 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-osd.2.log 2024-02-19T20:40:17.493 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-19T20:40:17.495 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-osd.2.log: 87.6% -- replaced with /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph.log.gz 2024-02-19T20:40:17.496 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-mgr.smithi114.pcvtrk.log 2024-02-19T20:40:17.509 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-volume.log: /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-mgr.smithi114.pcvtrk.log: 90.8% -- replaced with /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-19T20:40:17.612 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi114.vpoeop.log: file size changed while zipping 2024-02-19T20:40:17.612 INFO:teuthology.orchestra.run.smithi114.stderr: 94.9% -- replaced with /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi114.vpoeop.log.gz 2024-02-19T20:40:17.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:17.329+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:17.330893+0000) 2024-02-19T20:40:17.627 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:17.551+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:17.552004+0000) 2024-02-19T20:40:17.627 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:17.636+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:17.637499+0000) 2024-02-19T20:40:17.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:17.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:17.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:17.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:17.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:17.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:17.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:17.765+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:17.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:17.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:17.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:17.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:17.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:17.759+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:17.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:17.760+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:18.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:18.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:18.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:18.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:18.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:18.227+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:18.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:18.228+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:18.627 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:18.331+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:18.332949+0000) 2024-02-19T20:40:18.628 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:18.554+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:18.555002+0000) 2024-02-19T20:40:18.629 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:18.640+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:18.641785+0000) 2024-02-19T20:40:18.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:18.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:18.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:18.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:18.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:18.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:18.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:18.718+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:18.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:18.764+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:18.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:18.764+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:18.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:18.764+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:18.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:18.765+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:19.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:19.234+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:19.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:19.234+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:19.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:19.234+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:19.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:19.235+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:19.454 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-mgr.smithi114.pcvtrk.log: file size changed while zipping 2024-02-19T20:40:19.454 INFO:teuthology.orchestra.run.smithi114.stderr: 89.8% -- replaced with /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-mgr.smithi114.pcvtrk.log.gz 2024-02-19T20:40:19.629 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:19.556+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:19.557464+0000) 2024-02-19T20:40:19.630 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:19.340+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:19.341612+0000) 2024-02-19T20:40:19.630 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:19.643+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:19.644635+0000) 2024-02-19T20:40:19.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:19.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:19.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:19.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:19.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:19.728+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:19.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:19.728+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:19.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:19.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:19.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:19.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:19.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:19.775+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:19.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:19.776+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:20.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:20.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:20.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:20.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:20.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:20.204+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:20.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:20.205+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:20.508 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:40:20.544 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:20.344+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:20.345625+0000) 2024-02-19T20:40:20.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:20.558+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:20.559960+0000) 2024-02-19T20:40:20.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:20.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:20.982 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:20.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:20.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:20.751+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:20.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:20.752+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:20.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:20.647+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:20.648004+0000) 2024-02-19T20:40:20.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:20.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:20.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:20.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:20.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:20.772+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:20.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:20.773+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:21.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:21.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:21.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:21.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:21.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:21.211+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:21.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:21.213+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:21.635 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:21.347+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:21.348834+0000) 2024-02-19T20:40:21.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:21.562+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:21.563642+0000) 2024-02-19T20:40:21.982 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:40:21.669+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67757 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:40:21.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:21.662+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:21.663774+0000) 2024-02-19T20:40:21.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:21.668+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:21.669027+0000) 2024-02-19T20:40:21.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:21.722+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:21.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:21.722+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:21.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:21.722+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:21.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:21.723+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:21.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:21.649+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:21.650526+0000) 2024-02-19T20:40:21.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:21.668+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:21.669140+0000) 2024-02-19T20:40:21.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:21.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:21.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:21.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:21.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:21.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:21.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:21.761+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:22.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:22.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:22.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:22.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:22.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:22.213+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:22.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:22.214+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:22.636 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:22.349+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:22.350049+0000) 2024-02-19T20:40:22.637 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:22.563+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:22.564646+0000) 2024-02-19T20:40:22.972 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-02-19T20:40:22.972 INFO:teuthology.orchestra.run.smithi114.stderr: 90.2% -- replaced with /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-osd.2.log.gz 2024-02-19T20:40:22.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:22.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:22.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:22.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:22.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:22.713+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:22.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:22.714+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:22.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:22.650+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:22.651517+0000) 2024-02-19T20:40:22.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:22.779+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:22.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:22.779+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:22.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:22.779+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:22.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:22.779+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:23.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:23.253+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:23.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:23.253+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:23.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:23.253+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:23.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:23.255+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:23.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:23.351+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:23.352399+0000) 2024-02-19T20:40:23.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:23.567+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:23.568048+0000) 2024-02-19T20:40:23.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:23.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:23.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:23.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:23.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:23.671+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:23.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:23.672+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:23.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:23.653+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:23.654200+0000) 2024-02-19T20:40:24.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:23.774+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:24.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:23.774+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:24.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:23.774+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:24.202 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:23.775+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:24.470 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:24.286+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:24.470 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:24.286+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:24.470 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:24.286+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:24.470 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:24.287+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:24.470 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:24.353+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:24.354610+0000) 2024-02-19T20:40:24.746 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:24.569+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:24.570795+0000) 2024-02-19T20:40:24.746 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:24.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:24.746 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:24.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:24.746 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:24.717+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:24.747 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:24.717+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:24.747 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:24.657+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:24.658073+0000) 2024-02-19T20:40:24.747 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:24.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:24.747 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:24.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:24.747 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:24.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:25.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:24.761+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:25.163 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-02-19T20:40:25.163 INFO:teuthology.orchestra.run.smithi114.stderr: 90.5% -- replaced with /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-osd.1.log.gz 2024-02-19T20:40:25.469 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:25.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:25.470 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:25.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:25.470 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:25.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:25.470 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:25.302+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:25.470 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:25.356+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:25.357163+0000) 2024-02-19T20:40:25.751 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:25.572+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:25.573500+0000) 2024-02-19T20:40:25.751 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:25.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:25.751 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:25.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:25.751 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:25.676+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:25.751 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:25.677+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:25.752 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:25.660+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:25.661188+0000) 2024-02-19T20:40:25.752 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:25.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:26.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:25.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:26.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:25.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:26.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:25 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:25.765+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:26.560 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:26.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:26.561 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:26.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:26.561 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:26.305+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:26.561 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:26.307+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:26.561 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:26.359+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:26.360350+0000) 2024-02-19T20:40:26.983 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:40:26.678+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67797 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:40:26.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:26.677+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:26.678793+0000) 2024-02-19T20:40:26.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:26.575+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:26.576063+0000) 2024-02-19T20:40:26.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:26.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:26.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:26.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:26.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:26.670+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:26.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:26.671+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:26.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:26.675+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:26.676777+0000) 2024-02-19T20:40:26.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:26.663+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:26.664747+0000) 2024-02-19T20:40:26.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:26.674+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:26.675748+0000) 2024-02-19T20:40:26.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:26.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:26.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:26.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:26.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:26.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:26.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:26 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:26.735+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:27.620 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:27.576+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:27.577807+0000) 2024-02-19T20:40:27.621 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:27.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:27.621 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:27.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:27.621 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:27.634+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:27.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:27.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:27.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:27.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:27.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:27.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:27.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:27.333+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:27.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:27.360+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:27.361328+0000) 2024-02-19T20:40:27.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:27.664+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:27.665784+0000) 2024-02-19T20:40:27.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:27.716+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:27.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:27.716+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:27.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:27.716+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:27.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:27.716+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:27.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:27 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:27.635+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:28.559 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:28.302+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:28.559 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:28.302+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:28.559 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:28.302+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:28.559 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:28.303+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:28.559 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:28.363+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:28.364345+0000) 2024-02-19T20:40:28.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:28.579+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:28.580775+0000) 2024-02-19T20:40:28.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:28.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:28.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:28.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:28.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:28.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:28.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:28.647+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:28.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:28.667+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:28.668542+0000) 2024-02-19T20:40:28.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:28.671+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:28.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:28.671+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:28.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:28.671+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:28.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:28 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:28.671+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:29.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:29.266+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:29.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:29.266+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:29.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:29.266+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:29.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:29.267+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:29.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:29.366+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:29.367500+0000) 2024-02-19T20:40:29.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:40:29.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:40:29.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:29.582+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:29.583805+0000) 2024-02-19T20:40:29.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:29.632+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:29.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:29.632+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:29.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:29.632+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:29.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:29.632+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:29.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:29.667+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:29.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:29.667+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:29.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:29.667+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:29.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:29.668+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:29.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:29 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:29.670+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:29.671784+0000) 2024-02-19T20:40:30.571 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:30.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:30.572 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:30.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:30.572 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:30.296+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:30.572 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:30.297+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:30.572 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:30.369+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:30.370491+0000) 2024-02-19T20:40:30.698 INFO:teuthology.orchestra.run.smithi114.stderr: 92.4% -- replaced with /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi114.tcwpod.log.gz 2024-02-19T20:40:30.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:30.630+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:30.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:30.630+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:30.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:30.630+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:30.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:30.630+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:30.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:30.674+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:30.675160+0000) 2024-02-19T20:40:30.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:30.585+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:30.586866+0000) 2024-02-19T20:40:30.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:30.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:30.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:30.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:30.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:30.664+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:30.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:30 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:30.665+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:31.216 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-02-19T20:40:31.216 INFO:teuthology.orchestra.run.smithi114.stderr: 95.0% -- replaced with /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-osd.0.log.gz 2024-02-19T20:40:31.574 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:31.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:31.574 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:31.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:31.574 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:31.301+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:31.574 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:31.302+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:31.574 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:31.373+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:31.374061+0000) 2024-02-19T20:40:31.983 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:40:31.686+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67839 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:40:31.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:31.682+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:31.683923+0000) 2024-02-19T20:40:31.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:31.588+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:31.589808+0000) 2024-02-19T20:40:31.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:31.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:31.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:31.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:31.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:31.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:31.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:31.639+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:31.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:31.682+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:31.683264+0000) 2024-02-19T20:40:31.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:31.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:31.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:31.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:31.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:31.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:31.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:31.659+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:31.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:31.677+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:31.678751+0000) 2024-02-19T20:40:31.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:31 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:31.682+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:31.683292+0000) 2024-02-19T20:40:32.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:32.253+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:32.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:32.253+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:32.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:32.253+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:32.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:32.255+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:32.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:32.374+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:32.374995+0000) 2024-02-19T20:40:32.576 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:40:32.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:32.590+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:32.591129+0000) 2024-02-19T20:40:32.983 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:32.614+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:32.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:32.614+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:32.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:32.614+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:32.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:32.615+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:32.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:32.664+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:32.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:32.664+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:32.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:32.664+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:32.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:32.665+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:32.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:32 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:32.678+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:32.679939+0000) 2024-02-19T20:40:33.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:33.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:33.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:33.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:33.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:33.209+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:33.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:33.210+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:33.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:33.377+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:33.378482+0000) 2024-02-19T20:40:33.634 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:33.593+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:33.594443+0000) 2024-02-19T20:40:33.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:33.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:33.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:33.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:33.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:33.606+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:33.635 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:33.607+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:33.635 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:33.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:33.635 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:33.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:33.635 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:33.623+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:33.636 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:33.623+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:33.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:33 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:33.682+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:33.683313+0000) 2024-02-19T20:40:34.176 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-mon.smithi114.log: file size changed while zipping 2024-02-19T20:40:34.176 INFO:teuthology.orchestra.run.smithi114.stderr: 98.0% -- replaced with /var/log/ceph/be02a9a6-cf57-11ee-95bb-87774f69a715/ceph-mon.smithi114.log.gz 2024-02-19T20:40:34.179 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-02-19T20:40:34.179 INFO:teuthology.orchestra.run.smithi114.stderr:real 0m16.715s 2024-02-19T20:40:34.180 INFO:teuthology.orchestra.run.smithi114.stderr:user 0m57.460s 2024-02-19T20:40:34.180 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m1.244s 2024-02-19T20:40:34.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:34.234+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:34.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:34.234+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:34.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:34.234+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:34.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:34.235+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:34.649 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:34.380+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:34.381069+0000) 2024-02-19T20:40:34.649 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:34.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:34.649 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:34.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:34.650 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:34.590+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:34.650 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:34.590+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:34.650 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:34.596+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:34.597156+0000) 2024-02-19T20:40:34.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:34.664+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:34.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:34.664+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:34.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:34.664+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:34.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:34.664+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:34.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:34 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:34.685+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:34.686494+0000) 2024-02-19T20:40:35.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:35.220+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:35.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:35.220+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:35.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:35.220+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:35.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:35.222+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:35.619 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:35.382+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:35.383224+0000) 2024-02-19T20:40:35.620 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:35.598+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:35.599248+0000) 2024-02-19T20:40:35.620 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:35.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:35.620 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:35.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:35.620 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:35.604+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:35.620 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:35.605+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:35.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:35.627+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:35.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:35.627+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:35.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:35.627+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:35.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:35.627+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:35.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:35 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:35.687+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:35.688808+0000) 2024-02-19T20:40:36.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:36.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:36.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:36.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:36.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:36.206+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:36.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:36.207+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:36.673 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:36.384+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:36.385300+0000) 2024-02-19T20:40:36.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:36.571+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:36.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:36.571+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:36.673 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:36.571+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:36.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:36.572+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:36.674 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:36.601+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:36.602619+0000) 2024-02-19T20:40:36.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:36.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:36.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:36.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:36.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:36.587+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:36.674 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:36.588+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:36.983 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:40:36.695+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67879 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:40:36.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:36.690+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:36.691676+0000) 2024-02-19T20:40:36.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:36.690+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:36.691904+0000) 2024-02-19T20:40:36.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:36.690+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:36.691770+0000) 2024-02-19T20:40:36.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:36 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:36.691+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:36.692394+0000) 2024-02-19T20:40:37.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:37.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:37.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:37.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:37.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:37.225+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:37.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:37.226+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:37.679 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:37.385+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:37.386085+0000) 2024-02-19T20:40:37.680 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:37.602+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:37.603887+0000) 2024-02-19T20:40:37.680 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:37.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:37.680 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:37.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:37.680 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:37.616+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:37.681 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:37.617+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:37.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:37.615+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:37.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:37.615+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:37.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:37.615+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:37.681 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:37.615+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:37.983 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:37 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:37.694+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:37.695704+0000) 2024-02-19T20:40:38.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:38.229+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:38.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:38.229+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:38.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:38.229+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:38.255 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:38.231+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:38.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:38.387+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:38.388027+0000) 2024-02-19T20:40:38.632 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:38.592+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:38.632 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:38.592+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:38.632 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:38.592+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:38.632 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:38.593+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:38.633 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:38.605+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:38.606173+0000) 2024-02-19T20:40:38.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:38.646+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:38.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:38.646+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:38.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:38.646+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:38.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:38.647+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:38.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:38 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:38.697+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:38.697998+0000) 2024-02-19T20:40:39.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:39.231+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:39.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:39.231+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:39.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:39.231+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:39.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:39.232+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:39.624 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:39.389+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:39.390052+0000) 2024-02-19T20:40:39.625 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:39.608+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:39.609724+0000) 2024-02-19T20:40:39.625 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:39.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:39.625 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:39.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:39.625 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:39.639+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:39.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:39.639+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:39.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:39.644+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:39.984 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:39.644+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:39.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:39.644+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:39.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:39.645+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:39.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:39 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:39.699+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:39.700591+0000) 2024-02-19T20:40:40.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:40.269+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:40.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:40.269+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:40.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:40.269+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:40.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:40.270+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:40.597 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:40.392+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:40.393477+0000) 2024-02-19T20:40:40.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:40.610+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:40.611922+0000) 2024-02-19T20:40:40.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:40.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:40.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:40.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:40.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:40.626+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:40.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:40.627+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:40.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:40.677+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:40.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:40.677+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:40.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:40.677+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:40.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:40.678+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:40.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:40 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:40.703+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:40.704189+0000) 2024-02-19T20:40:41.503 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:41.251+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:41.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:41.251+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:41.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:41.251+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:41.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:41.252+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:41.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:41.396+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:41.396964+0000) 2024-02-19T20:40:41.984 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:40:41.703+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67920 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:40:41.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:41.699+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:41.700068+0000) 2024-02-19T20:40:41.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:41.657+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:41.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:41.657+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:41.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:41.657+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:41.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:41.657+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:41.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:41.699+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:41.700371+0000) 2024-02-19T20:40:41.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:41.704+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:41.705018+0000) 2024-02-19T20:40:41.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:41.613+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:41.614145+0000) 2024-02-19T20:40:41.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:41.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:41.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:41.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:41.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:41.641+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:41.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:41.641+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:41.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:41 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:41.699+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:41.700458+0000) 2024-02-19T20:40:42.599 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:42.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:42.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:42.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:42.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:42.291+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:42.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:42.292+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:42.600 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:42.396+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:42.397831+0000) 2024-02-19T20:40:42.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:42.614+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:42.615444+0000) 2024-02-19T20:40:42.984 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:42.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:42.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:42.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:42.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:42.646+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:42.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:42.647+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:42.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:42.636+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:42.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:42.636+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:42.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:42.636+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:42.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:42.636+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:42.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:42 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:42.707+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:42.708022+0000) 2024-02-19T20:40:43.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:43.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:43.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:43.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:43.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:43.289+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:43.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:43.290+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:43.566 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:43.399+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:43.400376+0000) 2024-02-19T20:40:43.874 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:43.616+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:43.617803+0000) 2024-02-19T20:40:43.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:43.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:43.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:43.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:43.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:43.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:43.875 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:43.661+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:43.875 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:43.662+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:43.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:43.662+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:43.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:43.662+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:43.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:43.662+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:43.876 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:43 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:43.709+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:43.710171+0000) 2024-02-19T20:40:44.471 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:44.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:44.471 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:44.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:44.472 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:44.335+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:44.472 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:44.336+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:44.472 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:44.402+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:44.403848+0000) 2024-02-19T20:40:44.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:40:44.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:40:44.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:44.620+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:44.621350+0000) 2024-02-19T20:40:44.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:44.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:44.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:44.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:44.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:44.657+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:44.756 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:44.658+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:44.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:44.701+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:44.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:44.701+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:44.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:44.701+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:44.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:44.701+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:44.756 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:44 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:44.711+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:44.712275+0000) 2024-02-19T20:40:45.471 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:45.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:45.471 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:45.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:45.472 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:45.330+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:45.472 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:45.331+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:45.472 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:45.406+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:45.407105+0000) 2024-02-19T20:40:45.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:45.623+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:45.623781+0000) 2024-02-19T20:40:45.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:45.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:45.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:45.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:45.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:45.661+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:45.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:45.662+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:45.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:45.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:45.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:45.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:45.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:45.694+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:45.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:45.694+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:45.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:45 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:45.714+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:45.714507+0000) 2024-02-19T20:40:46.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:46.352+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:46.611 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:46.352+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:46.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:46.352+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:46.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:46.352+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:46.612 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:46.407+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:46.409422+0000) 2024-02-19T20:40:46.985 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:40:46.711+0000 7fe5f9967700 -1 mon.smithi114@0(probing) e3 get_health_metrics reporting 67960 slow ops, oldest is log(1 entries from seq 628 at 2024-02-19T19:08:36.556872+0000) 2024-02-19T20:40:46.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:46.706+0000 7f76588d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:46.708711+0000) 2024-02-19T20:40:46.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:46.626+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:46.627310+0000) 2024-02-19T20:40:46.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:46.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:46.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:46.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:46.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:46.644+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:46.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:46.644+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:46.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:46.706+0000 7f4efc2ae700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:46.708371+0000) 2024-02-19T20:40:46.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:46.678+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:46.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:46.678+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:46.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:46.678+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:46.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:46.679+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:46.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:46.707+0000 7f0343441700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:46.709739+0000) 2024-02-19T20:40:46.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:46 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:46.713+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:46.715375+0000) 2024-02-19T20:40:47.648 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:40:47.667 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:47.627+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:47.628226+0000) 2024-02-19T20:40:47.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:47.383+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:47.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:47.383+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:47.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:47.383+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:47.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:47.384+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:47.668 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:47.409+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:47.410651+0000) 2024-02-19T20:40:47.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:47.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:47.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:47.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:47.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:47.695+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:47.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:47.695+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:47.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:47.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:47.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:47.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:47.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:47.682+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:47.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:47.683+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:47.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:47 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:47.716+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:47.717972+0000) 2024-02-19T20:40:48.704 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:48.412+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:48.413144+0000) 2024-02-19T20:40:48.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:48.426+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:48.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:48.426+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:48.705 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:48.426+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:48.706 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:48.427+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:48.706 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:48.629+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:48.630695+0000) 2024-02-19T20:40:48.706 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:48.653+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:48.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:48.653+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:48.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:48.653+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:48.707 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:48.654+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:48.985 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:48.719+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:48.720182+0000) 2024-02-19T20:40:48.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:48.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:48.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:48.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:48.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:48.726+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:48.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:48 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:48.727+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:49.648 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:49.631+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:49.632767+0000) 2024-02-19T20:40:49.648 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:49.408+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:49.648 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:49.408+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:49.648 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:49.408+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:49.649 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:49.409+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:49.649 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:49.415+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:49.415973+0000) 2024-02-19T20:40:49.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:49.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:49.985 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:49.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:49.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:49.660+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:49.986 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:49.661+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:49.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:49.721+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:49.722722+0000) 2024-02-19T20:40:49.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:49.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:49.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:49.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:49.986 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:49.722+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:49.987 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:49 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:49.722+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:50.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:40:50.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:40:50.653 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:50.633+0000 7f4eeaec7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:50.634928+0000) 2024-02-19T20:40:50.654 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:50.667+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:50.654 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:50.667+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:50.654 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:50.667+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:50.654 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:50.390+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:50.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:50.390+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:50.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:50.390+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:50.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:50.391+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:50.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:50.417+0000 7f7647cef700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:50.418076+0000) 2024-02-19T20:40:50.716 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:40:50.716 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:40:50.717 INFO:tasks.cephadm:Removing cluster... 2024-02-19T20:40:50.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 --force 2024-02-19T20:40:50.867 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: be02a9a6-cf57-11ee-95bb-87774f69a715 2024-02-19T20:40:50.959 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:50.668+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:50.959 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:50.723+0000 7f0336062700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:50.724922+0000) 2024-02-19T20:40:50.960 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:50.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:50.960 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:50.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:50.960 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:50.765+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:50.960 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:50 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:50.765+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:51.234 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:50 smithi114 systemd[1]: Stopping Ceph mon.smithi114 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T20:40:51.504 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:40:51.248+0000 7fe5fc96d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi114 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T20:40:51.504 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114[92979]: 2024-02-19T20:40:51.248+0000 7fe5fc96d700 -1 mon.smithi114@0(probing) e3 *** Got Signal Terminated *** 2024-02-19T20:40:51.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:51.426+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:51.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:51.426+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:51.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:51.426+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:51.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:51.427+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:52.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:51.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:52.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:51.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:52.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:51.699+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:52.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:51.701+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:52.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:51.767+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:52.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:51.767+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:52.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:51.767+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:52.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:51 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:51.768+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:52.586 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:52 smithi114 podman[133150]: 2024-02-19 20:40:52.306285474 +0000 UTC m=+1.173320559 container died e51f650f9be9c107f0a3e8b1342c8856ab70c7dc1a090e8598d81e6574275a33 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T20:40:52.587 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:52.405+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:52.587 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:52.405+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:52.587 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:52.405+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:52.587 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:52.406+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:53.004 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:52 smithi114 podman[133150]: 2024-02-19 20:40:52.768000066 +0000 UTC m=+1.635035150 container cleanup e51f650f9be9c107f0a3e8b1342c8856ab70c7dc1a090e8598d81e6574275a33 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T20:40:53.004 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:52 smithi114 bash[133150]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114 2024-02-19T20:40:53.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:52.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:53.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:52.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:53.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:52.788+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:53.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:52.789+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:53.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:52.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:53.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:52.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:53.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:52.698+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:53.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:52 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:52.699+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:53.436 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:53 smithi114 podman[133164]: 2024-02-19 20:40:53.06151102 +0000 UTC m=+0.747477425 container remove e51f650f9be9c107f0a3e8b1342c8856ab70c7dc1a090e8598d81e6574275a33 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-mon-smithi114, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-02-19T20:40:53.436 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:53 smithi114 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@mon.smithi114.service: Deactivated successfully. 2024-02-19T20:40:53.436 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:53 smithi114 systemd[1]: Stopped Ceph mon.smithi114 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T20:40:53.436 INFO:journalctl@ceph.mon.smithi114.smithi114.stdout:Feb 19 20:40:53 smithi114 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@mon.smithi114.service: Consumed 1h 14min 1.549s CPU time. 2024-02-19T20:40:53.720 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:53.450+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:53.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:53.450+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:53.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:53.450+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:53.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:53.450+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:54.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:53.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:54.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:53.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:54.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:53.823+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:54.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:53.824+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:54.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:53.735+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:54.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:53.735+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:54.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:53.735+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:54.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:53 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:53.736+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:54.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:54.483+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:54.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:54.483+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:54.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:54.483+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:54.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:54.484+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:55.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:54.716+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:55.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:54.716+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:55.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:54.716+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:55.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:54.717+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:55.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:54.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:55.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:54.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:55.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:54.813+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:55.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:54 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:54.814+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:55.699 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:55.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:55.699 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:55.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:55.699 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:55.451+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:55.699 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:55.451+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:56.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:55.804+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:56.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:55.804+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:56.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:55.804+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:56.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:55.805+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:56.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:55.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:56.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:55.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:56.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:55.765+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:56.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:55 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:55.766+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:56.603 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:56.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:56.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:56.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:56.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:56.468+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:56.604 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:56.469+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:56.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:56.730+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:56.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:56.730+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:56.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:56.730+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:56.971 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:56.731+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:56.971 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:56.789+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:56.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:56.789+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:56.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:56.789+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:56.972 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:56 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:56.789+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:57.593 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:57.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:57.593 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:57.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:57.593 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:57.421+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:57.593 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:57.422+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:57.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:57.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:57.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:57.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:57.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:57.764+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:57.868 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:57.764+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:57.868 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:57.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:57.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:57.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:57.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:57.760+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:57.869 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:57 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:57.761+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:58.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:58.386+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:58.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:58.386+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:58.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:58.386+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:58.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:58.387+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:58.921 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:58.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:58.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:58.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:58.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:58.757+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:40:58.922 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:58.757+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:40:58.922 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:58.730+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:58.923 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:58.730+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:58.923 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:58.730+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:40:58.923 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:58 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:58.731+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:40:59.631 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:59.341+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:59.632 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:59.341+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:59.632 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:59.341+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:40:59.632 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:40:59.342+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:00.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:59.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:00.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:59.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:00.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:59.759+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:00.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:40:59.760+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:41:00.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:59.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:00.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:59.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:00.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:59.721+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:00.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:40:59 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:40:59.721+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:00.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:00.377+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:00.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:00.377+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:00.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:00.377+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:00.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:00.378+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:01.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:00.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:01.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:00.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:01.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:00.735+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:01.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:00.736+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:01.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:00.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:01.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:00.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:01.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:00.797+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:01.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:00 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:00.798+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:41:01.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:01.377+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:01.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:01.377+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:01.364 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:01.377+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:01.614 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:01.378+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:02.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:01.816+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:02.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:01.816+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:02.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:01.816+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:02.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:01.817+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:41:02.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:01.691+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:02.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:01.691+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:02.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:01.691+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:02.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:01 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:01.691+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:02.721 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:02.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:02.722 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:02.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:02.722 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:02.396+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:02.722 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:02.397+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:03.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:02.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:03.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:02.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:03.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:02.861+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:03.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:02.862+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:41:03.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:02.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:03.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:02.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:03.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:02.736+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:03.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:02 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:02.736+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:03.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:03 smithi114 systemd[1]: Stopping Ceph osd.0 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T20:41:03.389 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:03.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:03.389 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:03.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:03.389 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:03.358+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:03.389 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:03.359+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:03.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:03.403+0000 7f76570d2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T20:41:03.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:03.403+0000 7f76570d2700 -1 osd.0 85 *** Got signal Terminated *** 2024-02-19T20:41:03.701 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:03.403+0000 7f76570d2700 -1 osd.0 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T20:41:04.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:03.716+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:04.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:03.716+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:04.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:03.716+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:04.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:03.716+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:04.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:03.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:04.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:03.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:04.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:03.877+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:04.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:03 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:03.878+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:41:04.702 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:04.398+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:04.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:04.398+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:04.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:04.398+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:04.703 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:04.399+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:05.003 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:04.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:05.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:04.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:05.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:04.904+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:05.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:04.905+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:41:05.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:04.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:05.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:04.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:05.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:04.717+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:05.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:04 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:04.718+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:05.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:05.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:05.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:05.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:05.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:05.363+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:05.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:05.364+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:06.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:05.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:06.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:05.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:06.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:05.907+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:06.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:05.907+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:41:06.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:05.709+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:06.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:05.710+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:06.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:05.710+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:06.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:05 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:05.710+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:06.651 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:06.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:06.651 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:06.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:06.651 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:06.331+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:06.651 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:06.332+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:06.919 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:06.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:06.929 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:06.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:06.929 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:06.665+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:06.929 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:06.666+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:07.253 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:06.934+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:07.253 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:06.934+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:07.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:06.934+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:07.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:06 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:06.934+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:41:07.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:07.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:07.655 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:07.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:07.656 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:07.372+0000 7f764febf700 -1 osd.0 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:20.228227+0000 (oldest deadline 2024-02-19T20:33:40.228227+0000) 2024-02-19T20:41:07.656 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0[101864]: 2024-02-19T20:41:07.373+0000 7f764febf700 -1 osd.0 85 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24311.0:3 3.12 3:4def72d4:::10000000003.00000000:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:07.969 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:07.670+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:07.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:07.670+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:07.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:07.670+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:07.970 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:07.671+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:08.253 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:07.984+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:08.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:07.984+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:08.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:07.984+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:08.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:07 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:07.985+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:41:08.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:08.654+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:08.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:08.654+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:08.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:08.654+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:08.939 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:08.655+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:09.253 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:09 smithi114 podman[134125]: 2024-02-19 20:41:09.11663318 +0000 UTC m=+5.833061213 container died 764769439e0ec1441476971554e8740debb987b61e0be6f8185311fc190106b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-02-19T20:41:09.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:08.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:09.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:08.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:09.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:08.953+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:09.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:08 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:08.954+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:41:09.663 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:09 smithi114 podman[134125]: 2024-02-19 20:41:09.459339454 +0000 UTC m=+6.175767485 container cleanup 764769439e0ec1441476971554e8740debb987b61e0be6f8185311fc190106b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-19T20:41:09.663 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:09 smithi114 bash[134125]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0 2024-02-19T20:41:09.967 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:09 smithi114 podman[134144]: 2024-02-19 20:41:09.691662521 +0000 UTC m=+0.577650505 container remove 764769439e0ec1441476971554e8740debb987b61e0be6f8185311fc190106b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-02-19T20:41:09.967 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:09.677+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:09.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:09.677+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:09.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:09.677+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:09.968 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:09.678+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:10.239 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:10 smithi114 podman[134235]: 2024-02-19 20:41:10.026130519 +0000 UTC m=+0.114008025 container create ef05f3b3cb40657cb137e595eee8787beef41b6eaea9be634e7053a69bb327d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T20:41:10.240 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:10 smithi114 podman[134235]: 2024-02-19 20:41:09.929443645 +0000 UTC m=+0.017321124 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T20:41:10.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:09.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:10.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:09.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:10.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:09.982+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:10.240 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:09 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:09.982+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:41:10.503 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:10 smithi114 podman[134235]: 2024-02-19 20:41:10.253701124 +0000 UTC m=+0.341578588 container init ef05f3b3cb40657cb137e595eee8787beef41b6eaea9be634e7053a69bb327d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-19T20:41:10.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:10 smithi114 podman[134235]: 2024-02-19 20:41:10.258685539 +0000 UTC m=+0.346563007 container start ef05f3b3cb40657cb137e595eee8787beef41b6eaea9be634e7053a69bb327d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) 2024-02-19T20:41:10.504 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:10 smithi114 podman[134235]: 2024-02-19 20:41:10.361854468 +0000 UTC m=+0.449731941 container attach ef05f3b3cb40657cb137e595eee8787beef41b6eaea9be634e7053a69bb327d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True) 2024-02-19T20:41:10.832 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:10 smithi114 podman[134235]: 2024-02-19 20:41:10.557699599 +0000 UTC m=+0.645577062 container died ef05f3b3cb40657cb137e595eee8787beef41b6eaea9be634e7053a69bb327d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-19T20:41:10.832 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:10.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:10.832 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:10.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:10.832 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:10.723+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:10.833 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:10.723+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:11.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:10.950+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:11.083 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:10.950+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:11.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:10.950+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:11.084 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:10 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:10.950+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:41:11.387 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:11 smithi114 systemd[1]: Stopping Ceph osd.1 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T20:41:11.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:11 smithi114 podman[134261]: 2024-02-19 20:41:11.098927932 +0000 UTC m=+0.534749596 container remove ef05f3b3cb40657cb137e595eee8787beef41b6eaea9be634e7053a69bb327d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.license=GPLv2) 2024-02-19T20:41:11.388 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:11 smithi114 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-19T20:41:11.389 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:11 smithi114 systemd[1]: Stopped Ceph osd.0 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T20:41:11.389 INFO:journalctl@ceph.osd.0.smithi114.stdout:Feb 19 20:41:11 smithi114 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.0.service: Consumed 44min 33.007s CPU time. 2024-02-19T20:41:11.670 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:11.601+0000 7f4efa2aa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T20:41:11.670 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:11.601+0000 7f4efa2aa700 -1 osd.1 85 *** Got signal Terminated *** 2024-02-19T20:41:11.670 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:11.601+0000 7f4efa2aa700 -1 osd.1 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T20:41:11.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:11.685+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:11.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:11.685+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:11.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:11.685+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:11.965 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:11.685+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:12.253 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:11.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:12.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:11.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:12.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:11.980+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:12.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:11 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:11.981+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:41:12.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:12.706+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:12.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:12.706+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:12.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:12.706+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:12.953 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:12.707+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:13.253 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:12.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:13.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:12.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:13.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:12.968+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:13.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:12 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:12.969+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:41:14.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:13.947+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:14.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:13.947+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:14.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:13.947+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:14.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:13.947+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:41:14.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:13.730+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:14.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:13.730+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:14.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:13.730+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:14.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:13 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:13.730+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:15.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:14.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:15.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:14.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:15.004 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:14.908+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:15.005 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:14.909+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:41:15.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:14.702+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:15.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:14.702+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:15.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:14.702+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:15.006 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:14 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:14.702+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:15.916 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:15.668+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:15.916 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:15.668+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:15.916 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:15.668+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:15.916 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:15.668+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:16.253 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:15.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:16.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:15.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:16.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:15.931+0000 7f4ef3097700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:13.101865+0000 (oldest deadline 2024-02-19T20:33:33.101865+0000) 2024-02-19T20:41:16.254 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:15 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1[107278]: 2024-02-19T20:41:15.931+0000 7f4ef3097700 -1 osd.1 85 get_health_metrics reporting 218 slow ops, oldest is osd_op(mds.0.13:252 2.13 2:ceffbd45:::200.0000000a:head [write 0~518488 [fadvise_dontneed] in=518488b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-02-19T20:41:17.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:16.674+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:17.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:16.674+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:17.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:16.674+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:17.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:16 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:16.674+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:17.454 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:17 smithi114 podman[134341]: 2024-02-19 20:41:17.214091364 +0000 UTC m=+5.716845771 container died c1c32dbdf57335eabf09d22bbc653009e14ba4000e3d1fa3daa4510894ae380f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T20:41:17.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:17 smithi114 podman[134341]: 2024-02-19 20:41:17.56908462 +0000 UTC m=+6.071839027 container cleanup c1c32dbdf57335eabf09d22bbc653009e14ba4000e3d1fa3daa4510894ae380f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T20:41:17.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:17 smithi114 bash[134341]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1 2024-02-19T20:41:17.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:17.651+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:17.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:17.651+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:17.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:17.651+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:17.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:17 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:17.652+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:18.159 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:17 smithi114 podman[134369]: 2024-02-19 20:41:17.837217742 +0000 UTC m=+0.617573935 container remove c1c32dbdf57335eabf09d22bbc653009e14ba4000e3d1fa3daa4510894ae380f (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=) 2024-02-19T20:41:18.475 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:18 smithi114 podman[134457]: 2024-02-19 20:41:18.078173793 +0000 UTC m=+0.017946222 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T20:41:18.475 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:18 smithi114 podman[134457]: 2024-02-19 20:41:18.196337786 +0000 UTC m=+0.136110206 container create e0aafa0d69f2d6cceb2cde1db2b7c4320290be4b0c1deca16f73c798e74d82a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux ) 2024-02-19T20:41:18.476 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:18 smithi114 podman[134457]: 2024-02-19 20:41:18.39920977 +0000 UTC m=+0.338982191 container init e0aafa0d69f2d6cceb2cde1db2b7c4320290be4b0c1deca16f73c798e74d82a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, ceph=True) 2024-02-19T20:41:18.476 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:18 smithi114 podman[134457]: 2024-02-19 20:41:18.402281571 +0000 UTC m=+0.342053997 container start e0aafa0d69f2d6cceb2cde1db2b7c4320290be4b0c1deca16f73c798e74d82a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-02-19T20:41:18.753 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:18 smithi114 podman[134457]: 2024-02-19 20:41:18.489648654 +0000 UTC m=+0.429421087 container attach e0aafa0d69f2d6cceb2cde1db2b7c4320290be4b0c1deca16f73c798e74d82a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T20:41:18.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:18 smithi114 podman[134457]: 2024-02-19 20:41:18.691155692 +0000 UTC m=+0.630928116 container died e0aafa0d69f2d6cceb2cde1db2b7c4320290be4b0c1deca16f73c798e74d82a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-19T20:41:18.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:18.649+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:18.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:18.649+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:18.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:18.649+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:18.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:18 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:18.650+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:19.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:19 smithi114 systemd[1]: Stopping Ceph osd.2 for be02a9a6-cf57-11ee-95bb-87774f69a715... 2024-02-19T20:41:19.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:19 smithi114 podman[134483]: 2024-02-19 20:41:19.244086679 +0000 UTC m=+0.543024278 container remove e0aafa0d69f2d6cceb2cde1db2b7c4320290be4b0c1deca16f73c798e74d82a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-19T20:41:19.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:19 smithi114 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-19T20:41:19.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:19 smithi114 systemd[1]: Stopped Ceph osd.1 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T20:41:19.505 INFO:journalctl@ceph.osd.1.smithi114.stdout:Feb 19 20:41:19 smithi114 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.1.service: Consumed 51.054s CPU time. 2024-02-19T20:41:20.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:19.649+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:20.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:19.649+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:20.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:19.649+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:20.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:19.649+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:20.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:19.704+0000 7f034143d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T20:41:20.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:19.704+0000 7f034143d700 -1 osd.2 85 *** Got signal Terminated *** 2024-02-19T20:41:20.005 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:19 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:19.704+0000 7f034143d700 -1 osd.2 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T20:41:21.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:20.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:21.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:20.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:21.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:20.658+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:21.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:20 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:20.658+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:22.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:21.635+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:22.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:21.635+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:22.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:21.635+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:22.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:21 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:21.635+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:22.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:22.622+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:22.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:22.622+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:22.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:22.622+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:22.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:22 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:22.622+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:24.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:23.629+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:24.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:23.629+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:24.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:23.629+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:24.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:23 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:23.629+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:25.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:24.590+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:25.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:24.590+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:25.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:24.590+0000 7f033e232700 -1 osd.2 85 heartbeat_check: no reply from 172.21.15.174:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:33:08.268894+0000 (oldest deadline 2024-02-19T20:33:28.268894+0000) 2024-02-19T20:41:25.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:24 smithi114 ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2[113150]: 2024-02-19T20:41:24.590+0000 7f033e232700 -1 osd.2 85 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.24311.0:26 3.1e 3:78cea7f0:::10000000017.00000000:head [write 0~3719 in=3719b] snapc 1=[] ondisk+write+known_if_redirected e85) 2024-02-19T20:41:25.381 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:25 smithi114 podman[134564]: 2024-02-19 20:41:25.108884874 +0000 UTC m=+5.524180358 container died 11e64d20e378e5796713cb722c7d426ecb96ab922eda5782300cb59c884e2d6e (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-02-19T20:41:25.381 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:25 smithi114 podman[134564]: 2024-02-19 20:41:25.397613161 +0000 UTC m=+5.812908642 container cleanup 11e64d20e378e5796713cb722c7d426ecb96ab922eda5782300cb59c884e2d6e (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2, GIT_CLEAN=True, org.label-schema.build-date=20240214, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T20:41:25.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:25 smithi114 bash[134564]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2 2024-02-19T20:41:25.753 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:25 smithi114 podman[134581]: 2024-02-19 20:41:25.615555585 +0000 UTC m=+0.501207705 container remove 11e64d20e378e5796713cb722c7d426ecb96ab922eda5782300cb59c884e2d6e (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T20:41:26.178 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:25 smithi114 podman[134672]: 2024-02-19 20:41:25.836217836 +0000 UTC m=+0.017875371 image pull f081f847070acba44285717fbebd438c29e1171bb26f61956e59e7adb866ff0a quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 2024-02-19T20:41:26.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:25 smithi114 podman[134672]: 2024-02-19 20:41:25.95792133 +0000 UTC m=+0.139578864 container create d23f1d0da29e02f4cfa3859daebfe9047584feb5c657334cdb5239e1838e2cd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-19T20:41:26.179 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:26 smithi114 podman[134672]: 2024-02-19 20:41:26.193828725 +0000 UTC m=+0.375486261 container init d23f1d0da29e02f4cfa3859daebfe9047584feb5c657334cdb5239e1838e2cd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-02-19T20:41:26.448 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:26 smithi114 podman[134672]: 2024-02-19 20:41:26.197614941 +0000 UTC m=+0.379272484 container start d23f1d0da29e02f4cfa3859daebfe9047584feb5c657334cdb5239e1838e2cd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240215.160715-f78a58c) 2024-02-19T20:41:26.449 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:26 smithi114 podman[134672]: 2024-02-19 20:41:26.285339108 +0000 UTC m=+0.466996663 container attach d23f1d0da29e02f4cfa3859daebfe9047584feb5c657334cdb5239e1838e2cd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T20:41:26.748 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:26 smithi114 podman[134672]: 2024-02-19 20:41:26.493874991 +0000 UTC m=+0.675532521 container died d23f1d0da29e02f4cfa3859daebfe9047584feb5c657334cdb5239e1838e2cd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-02-19T20:41:27.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:26 smithi114 podman[134699]: 2024-02-19 20:41:26.988644514 +0000 UTC m=+0.489672371 container remove d23f1d0da29e02f4cfa3859daebfe9047584feb5c657334cdb5239e1838e2cd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, name=ceph-be02a9a6-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240215.160715-f78a58c, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-19T20:41:27.003 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:27 smithi114 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-19T20:41:27.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:27 smithi114 systemd[1]: Stopped Ceph osd.2 for be02a9a6-cf57-11ee-95bb-87774f69a715. 2024-02-19T20:41:27.004 INFO:journalctl@ceph.osd.2.smithi114.stdout:Feb 19 20:41:27 smithi114 systemd[1]: ceph-be02a9a6-cf57-11ee-95bb-87774f69a715@osd.2.service: Consumed 44.919s CPU time. 2024-02-19T20:41:31.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:41:31.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:41:34.140 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:41:43.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:41:43.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:41:46.204 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:41:58.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:41:58.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:42:01.276 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:42:04.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:42:04.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:42:04.348 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:42:04.348 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:42:04.349 INFO:tasks.cephadm:Teardown complete 2024-02-19T20:42:04.349 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1553, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1810, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1686, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 142, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 310, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi174 2024-02-19T20:42:04.373 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-19T20:42:04.385 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-19T20:42:04.402 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-19T20:42:04.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T20:42:04.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:42:04.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:42:07.487 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:42:16.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:42:16.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:42:19.552 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:42:31.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:42:31.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:42:34.620 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:42:37.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:42:37.622 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:42:37.696 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:42:37.696 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:42:37.747 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi174 2024-02-19T20:42:37.768 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-19T20:42:37.781 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-19T20:42:37.781 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T20:42:37.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:42:37.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:42:37.804 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-02-19T20:42:37.810 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T20:42:37.810 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-02-19T20:42:37.810 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 +492us[ +463us] +/- 88ms 2024-02-19T20:42:37.810 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv02.front.sepia.ceph.com 3 8 377 23 -738us[ -766us] +/- 70ms 2024-02-19T20:42:37.811 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 27 +789us[ +761us] +/- 94ms 2024-02-19T20:42:37.811 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T20:42:40.860 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:42:49.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:42:49.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:42:52.924 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:43:04.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:43:04.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:43:08.000 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:43:11.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:43:11.002 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:43:11.068 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:43:11.068 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:43:11.069 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi174 2024-02-19T20:43:11.086 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-19T20:43:11.105 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-19T20:43:11.106 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-19T20:43:11.118 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T20:43:11.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:43:11.216 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:43:14.272 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:43:23.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:43:23.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:43:26.332 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:43:38.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:43:38.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:43:41.404 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:43:44.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:43:44.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:43:44.476 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:43:44.476 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:43:44.477 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi174 2024-02-19T20:43:44.485 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-19T20:43:44.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-19T20:43:44.507 INFO:teuthology.task.internal:Duration was 7306.234586 seconds 2024-02-19T20:43:44.508 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-19T20:43:44.519 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-19T20:43:44.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T20:43:44.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:43:44.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:43:44.574 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T20:43:47.589 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:43:56.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:43:56.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:43:59.648 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:44:11.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:44:11.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:14.716 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:44:17.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:44:17.718 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:17.792 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:44:17.792 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:44:17.793 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi174 2024-02-19T20:44:17.811 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-19T20:44:17.827 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-19T20:44:17.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T20:44:17.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:44:17.866 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:20.924 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:44:29.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:44:29.925 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:32.988 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:44:44.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:44:44.990 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:48.060 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:44:51.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:44:51.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:51.136 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:44:51.136 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:44:51.137 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi174 2024-02-19T20:44:51.152 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-19T20:44:51.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T20:44:51.172 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:44:51.172 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:51.200 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-02-19T20:44:54.236 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:45:03.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:45:03.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:45:06.304 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:45:18.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:45:18.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:45:21.372 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:45:24.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:45:24.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:45:24.444 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:45:24.444 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:45:24.445 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi174 2024-02-19T20:45:24.462 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-19T20:45:24.475 INFO:teuthology.task.internal:Transferring archived files... 2024-02-19T20:45:24.477 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566627/remote/smithi114 2024-02-19T20:45:24.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T20:45:24.523 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566627/remote/smithi174 2024-02-19T20:45:24.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:45:24.524 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:45:27.584 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:45:36.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:45:36.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:45:39.648 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:45:51.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:45:51.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:45:54.720 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:45:57.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:45:57.722 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:45:57.788 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:45:57.788 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:45:57.789 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi174 2024-02-19T20:45:57.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-19T20:45:57.819 INFO:teuthology.task.internal:Not uploading archives. 2024-02-19T20:45:57.820 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-19T20:45:57.888 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-19T20:45:57.889 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T20:45:57.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:45:57.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:45:57.911 INFO:teuthology.orchestra.run.smithi114.stdout: 265809 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 19 20:42 /home/ubuntu/cephtest 2024-02-19T20:45:57.911 INFO:teuthology.orchestra.run.smithi114.stdout: 265810 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Feb 19 20:44 /home/ubuntu/cephtest/archive 2024-02-19T20:45:57.911 INFO:teuthology.orchestra.run.smithi114.stdout: 265825 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 18:41 /home/ubuntu/cephtest/archive/syslog 2024-02-19T20:45:57.911 INFO:teuthology.orchestra.run.smithi114.stdout: 265839 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 19 18:41 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T20:45:57.911 INFO:teuthology.orchestra.run.smithi114.stdout: 265829 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 19 18:41 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T20:45:57.911 INFO:teuthology.orchestra.run.smithi114.stdout: 402380 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 20:43 /home/ubuntu/cephtest/archive/audit 2024-02-19T20:45:57.912 INFO:teuthology.orchestra.run.smithi114.stdout: 426286 208 -rw------- 1 ubuntu ubuntu 211292 Feb 19 20:43 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-02-19T20:45:57.912 INFO:teuthology.orchestra.run.smithi114.stdout: 265374 212 -rwxr-xr-x 1 ubuntu ubuntu 214738 Feb 19 18:50 /home/ubuntu/cephtest/cephadm 2024-02-19T20:45:57.913 INFO:teuthology.orchestra.run.smithi114.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-19T20:46:00.960 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:46:09.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:46:09.961 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:46:13.020 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:46:25.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:46:25.022 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:46:28.092 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:46:31.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-02-19T20:46:31.094 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:46:31.164 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-02-19T20:46:31.164 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:46:31.165 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi174 2024-02-19T20:46:31.180 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-19T20:46:31.193 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-19T20:46:31.206 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-02-19T20:46:31.206 INFO:teuthology.nuke:Checking targets against current locks 2024-02-19T20:46:31.239 DEBUG:teuthology.nuke:shortname: smithi114 2024-02-19T20:46:31.239 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T20:46:31.263 DEBUG:teuthology.nuke:shortname: smithi174 2024-02-19T20:46:31.263 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T20:46:31.267 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566627', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 18:29:52.269874', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPl8kXpGXbVfdtRkjLq6EX56iauEUR6aqba3MpgdRhe6RRcJUzXa4M6z8LW9rOsHFyxNvbNWlz330XuCkDwraMw='} 2024-02-19T20:46:31.295 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566627', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 18:29:52.271432', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN3HqTMVQLehp+LPhy0ilDziUjDNwqBIKb3kelgvF3ztYyHGtT/3snZ/CGkU1mJg/z7RZyvJQQhT073dhTF6rKQ='} 2024-02-19T20:46:31.324 INFO:teuthology.orchestra.console.smithi114:Power off 2024-02-19T20:46:31.324 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T20:46:31.348 DEBUG:teuthology.orchestra.console.smithi114:power off output: Chassis Power Control: Down/Off 2024-02-19T20:46:31.349 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:46:31.366 INFO:teuthology.orchestra.console.smithi174:Power off 2024-02-19T20:46:31.367 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T20:46:31.377 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-02-19T20:46:31.395 DEBUG:teuthology.orchestra.console.smithi174:power off output: Chassis Power Control: Down/Off 2024-02-19T20:46:31.396 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:46:31.418 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-02-19T20:46:35.378 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:46:35.420 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:46:35.489 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-02-19T20:46:35.529 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-02-19T20:46:39.492 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:46:39.530 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:46:39.603 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-02-19T20:46:39.641 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-02-19T20:46:43.606 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:46:43.642 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:46:43.718 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is off 2024-02-19T20:46:43.752 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off 2024-02-19T20:46:43.819 INFO:teuthology.orchestra.console.smithi114:Power off completed 2024-02-19T20:46:43.854 INFO:teuthology.orchestra.console.smithi174:Power off completed 2024-02-19T20:46:43.959 INFO:teuthology.run:Summary data: description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 0a 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 7306.234585523605 failure_reason: 'Command failed on smithi114 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid be02a9a6-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c ''ceph orch ps''"' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4681efabf223408f9f0bbe4aaae5d81d status: fail success: false 2024-02-19T20:46:43.960 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T20:46:44.037 INFO:teuthology.run:FAIL